(cassandra) branch cassandra-4.0 updated: Ensure that repair doesn't exceed repair_session_space by running limiting validation parallelism
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch cassandra-4.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/cassandra-4.0 by this push: new 505f5af645 Ensure that repair doesn't exceed repair_session_space by running limiting validation parallelism 505f5af645 is described below commit 505f5af645c1712c3da42d98d005276396ff2667 Author: Andrés de la Peña AuthorDate: Tue Feb 6 22:17:14 2024 + Ensure that repair doesn't exceed repair_session_space by running limiting validation parallelism patch by Andrés de la Peña; reviewed by David Capwell for CASSANDRA-19336 Co-authored-by: Andrés de la Peña Co-authored-by: David Capwell --- CHANGES.txt| 1 + conf/cassandra.yaml| 36 +- src/java/org/apache/cassandra/config/Config.java | 2 + .../cassandra/config/DatabaseDescriptor.java | 10 ++ .../org/apache/cassandra/repair/RepairJob.java | 95 +++--- .../apache/cassandra/repair/RepairRunnable.java| 10 +- .../org/apache/cassandra/repair/RepairSession.java | 3 + .../org/apache/cassandra/repair/Scheduler.java | 142 + .../cassandra/service/ActiveRepairService.java | 5 +- .../test/OptimiseStreamsRepairTest.java| 18 +-- .../repair/ConcurrentValidationRequestsTest.java | 136 .../org/apache/cassandra/repair/RepairJobTest.java | 3 +- .../apache/cassandra/repair/RepairSessionTest.java | 2 +- 13 files changed, 392 insertions(+), 71 deletions(-) diff --git a/CHANGES.txt b/CHANGES.txt index c02e1ffc7d..baeb9b7f61 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 4.0.13 + * Add new concurrent_merkle_tree_requests config property to prevent OOM during multi-range and/or multi-table repairs (CASSANDRA-19336) Merged from 3.11: Merged from 3.0: * Backport CASSANDRA-16418 to 3.x (CASSANDRA-18824) diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml index 3e4da52f5f..7f162749d2 100644 --- a/conf/cassandra.yaml +++ b/conf/cassandra.yaml @@ -553,17 +553,41 @@ concurrent_materialized_view_writes: 32 #off heap objects memtable_allocation_type: heap_buffers -# Limit memory usage for Merkle tree calculations during repairs. The default -# is 1/16th of the available heap. The main tradeoff is that smaller trees -# have less resolution, which can lead to over-streaming data. If you see heap -# pressure during repairs, consider lowering this, but you cannot go below -# one megabyte. If you see lots of over-streaming, consider raising -# this or using subrange repair. +# Limit memory usage for Merkle tree calculations during repairs of a certain +# table and common token range. Repair commands targetting multiple tables or +# virtual nodes can exceed this limit depending on concurrent_merkle_tree_requests. +# +# The default is 1/16th of the available heap. The main tradeoff is that +# smaller trees have less resolution, which can lead to over-streaming data. +# If you see heap pressure during repairs, consider lowering this, but you +# cannot go below one mebibyte. If you see lots of over-streaming, consider +# raising this or using subrange repair. # # For more details see https://issues.apache.org/jira/browse/CASSANDRA-14096. # # repair_session_space_in_mb: +# The number of simultaneous Merkle tree requests during repairs that can +# be performed by a repair command. The size of each validation request is +# limited by the repair_session_space property, so setting this to 1 will make +# sure that a repair command doesn't exceed that limit, even if the repair +# command is repairing multiple tables or multiple virtual nodes. +# +# There isn't a limit by default for backwards compatibility, but this can +# produce OOM for commands repairing multiple tables or multiple virtual nodes. +# A limit of just 1 simultaneous Merkle tree request is generally recommended +# with no virtual nodes so repair_session_space, and thereof the Merkle tree +# resolution, can be high. For virtual nodes a value of 1 with the default +# repair_session_space value will produce higher resolution Merkle trees +# at the expense of speed. Alternatively, when working with virtual nodes it +# can make sense to reduce the repair_session_space and increase the value of +# concurrent_merkle_tree_requests because each range will contain fewer data. +# +# For more details see https://issues.apache.org/jira/browse/CASSANDRA-19336. +# +# A zero value means no limit. +# concurrent_merkle_tree_requests: 0 + # Total space to use for commit logs on disk. # # If space gets above this value, Cassandra will flush every dirty CF diff --git a/src/java/org/apache/cassandra/config/Config.java b/src/java/org/apache/cassandra/config/Config.java index f8d8d46db8..d7517124df 100644 --- a/src/java/org/apache
(cassandra) branch cassandra-5.0 updated (78dca99973 -> 5b9321eee1)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch cassandra-5.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git from 78dca99973 Merge branch 'cassandra-4.1' into cassandra-5.0 new 505f5af645 Ensure that repair doesn't exceed repair_session_space by running limiting validation parallelism new 89a8155916 Merge branch 'cassandra-4.0' into cassandra-4.1 new 5b9321eee1 Merge branch 'cassandra-4.1' into cassandra-5.0 The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + conf/cassandra.yaml| 36 +- src/java/org/apache/cassandra/config/Config.java | 2 + .../cassandra/config/DatabaseDescriptor.java | 10 ++ .../cassandra/repair/AbstractRepairTask.java | 5 +- .../cassandra/repair/IncrementalRepairTask.java| 4 +- .../apache/cassandra/repair/NormalRepairTask.java | 4 +- .../apache/cassandra/repair/PreviewRepairTask.java | 4 +- .../apache/cassandra/repair/RepairCoordinator.java | 4 +- .../org/apache/cassandra/repair/RepairJob.java | 129 ++- .../org/apache/cassandra/repair/RepairSession.java | 3 + .../org/apache/cassandra/repair/RepairTask.java| 6 +- .../org/apache/cassandra/repair/Scheduler.java | 118 ++ .../cassandra/service/ActiveRepairService.java | 17 ++- .../service/ActiveRepairServiceMBean.java | 4 + .../test/OptimiseStreamsRepairTest.java| 17 +-- .../repair/ConcurrentValidationRequestsTest.java | 136 + .../org/apache/cassandra/repair/RepairJobTest.java | 5 +- .../apache/cassandra/repair/RepairSessionTest.java | 2 +- 19 files changed, 412 insertions(+), 95 deletions(-) create mode 100644 src/java/org/apache/cassandra/repair/Scheduler.java create mode 100644 test/distributed/org/apache/cassandra/distributed/test/repair/ConcurrentValidationRequestsTest.java - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch cassandra-4.1 updated (d9dbd5e1d0 -> 89a8155916)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch cassandra-4.1 in repository https://gitbox.apache.org/repos/asf/cassandra.git from d9dbd5e1d0 Merge branch 'cassandra-4.0' into cassandra-4.1 new 505f5af645 Ensure that repair doesn't exceed repair_session_space by running limiting validation parallelism new 89a8155916 Merge branch 'cassandra-4.0' into cassandra-4.1 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + conf/cassandra.yaml| 36 +- src/java/org/apache/cassandra/config/Config.java | 2 + .../cassandra/config/DatabaseDescriptor.java | 10 ++ .../cassandra/repair/AbstractRepairTask.java | 6 +- .../cassandra/repair/IncrementalRepairTask.java| 4 +- .../apache/cassandra/repair/NormalRepairTask.java | 4 +- .../apache/cassandra/repair/PreviewRepairTask.java | 4 +- .../org/apache/cassandra/repair/RepairJob.java | 106 .../apache/cassandra/repair/RepairRunnable.java| 4 +- .../org/apache/cassandra/repair/RepairSession.java | 3 + .../org/apache/cassandra/repair/RepairTask.java| 6 +- .../org/apache/cassandra/repair/Scheduler.java | 118 ++ .../cassandra/service/ActiveRepairService.java | 17 ++- .../service/ActiveRepairServiceMBean.java | 4 + .../test/OptimiseStreamsRepairTest.java| 18 +-- .../repair/ConcurrentValidationRequestsTest.java | 136 + .../org/apache/cassandra/repair/RepairJobTest.java | 6 +- .../apache/cassandra/repair/RepairSessionTest.java | 2 +- 19 files changed, 403 insertions(+), 84 deletions(-) create mode 100644 src/java/org/apache/cassandra/repair/Scheduler.java create mode 100644 test/distributed/org/apache/cassandra/distributed/test/repair/ConcurrentValidationRequestsTest.java - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch trunk updated (64e2a4e9a3 -> aa3ee3c7f1)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from 64e2a4e9a3 Refactor structure of caching metrics and expose auth cache metrics via JMX new 505f5af645 Ensure that repair doesn't exceed repair_session_space by running limiting validation parallelism new 89a8155916 Merge branch 'cassandra-4.0' into cassandra-4.1 new 5b9321eee1 Merge branch 'cassandra-4.1' into cassandra-5.0 new aa3ee3c7f1 Merge branch 'cassandra-5.0' into trunk The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + conf/cassandra.yaml| 36 +- src/java/org/apache/cassandra/config/Config.java | 2 + .../cassandra/config/DatabaseDescriptor.java | 10 ++ .../cassandra/repair/AbstractRepairTask.java | 5 +- .../cassandra/repair/IncrementalRepairTask.java| 4 +- .../apache/cassandra/repair/NormalRepairTask.java | 4 +- .../apache/cassandra/repair/PreviewRepairTask.java | 4 +- .../apache/cassandra/repair/RepairCoordinator.java | 4 +- .../org/apache/cassandra/repair/RepairJob.java | 129 ++- .../org/apache/cassandra/repair/RepairSession.java | 3 + .../org/apache/cassandra/repair/RepairTask.java| 6 +- .../org/apache/cassandra/repair/Scheduler.java | 118 ++ .../cassandra/service/ActiveRepairService.java | 17 ++- .../service/ActiveRepairServiceMBean.java | 4 + .../test/OptimiseStreamsRepairTest.java| 17 +-- .../repair/ConcurrentValidationRequestsTest.java | 136 + .../org/apache/cassandra/repair/RepairJobTest.java | 5 +- .../apache/cassandra/repair/RepairSessionTest.java | 2 +- 19 files changed, 412 insertions(+), 95 deletions(-) create mode 100644 src/java/org/apache/cassandra/repair/Scheduler.java create mode 100644 test/distributed/org/apache/cassandra/distributed/test/repair/ConcurrentValidationRequestsTest.java - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit aa3ee3c7f12ab89f8590d1366135afc65c042095
Merge: 64e2a4e9a3 5b9321eee1
Author: Andrés de la Peña
AuthorDate: Thu Feb 8 11:34:27 2024 +
Merge branch 'cassandra-5.0' into trunk
CHANGES.txt| 1 +
conf/cassandra.yaml| 36 +-
src/java/org/apache/cassandra/config/Config.java | 2 +
.../cassandra/config/DatabaseDescriptor.java | 10 ++
.../cassandra/repair/AbstractRepairTask.java | 5 +-
.../cassandra/repair/IncrementalRepairTask.java| 4 +-
.../apache/cassandra/repair/NormalRepairTask.java | 4 +-
.../apache/cassandra/repair/PreviewRepairTask.java | 4 +-
.../apache/cassandra/repair/RepairCoordinator.java | 4 +-
.../org/apache/cassandra/repair/RepairJob.java | 129 ++-
.../org/apache/cassandra/repair/RepairSession.java | 3 +
.../org/apache/cassandra/repair/RepairTask.java| 6 +-
.../org/apache/cassandra/repair/Scheduler.java | 118 ++
.../cassandra/service/ActiveRepairService.java | 17 ++-
.../service/ActiveRepairServiceMBean.java | 4 +
.../test/OptimiseStreamsRepairTest.java| 17 +--
.../repair/ConcurrentValidationRequestsTest.java | 136 +
.../org/apache/cassandra/repair/RepairJobTest.java | 5 +-
.../apache/cassandra/repair/RepairSessionTest.java | 2 +-
19 files changed, 412 insertions(+), 95 deletions(-)
diff --cc CHANGES.txt
index 9b8da81077,4f223e469a..4eb4c9a822
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -29,9 -15,13 +29,10 @@@ Merged from 5.0
* Fix resource cleanup after SAI query timeouts (CASSANDRA-19177)
* Suppress CVE-2023-6481 (CASSANDRA-19184)
Merged from 4.1:
- * Memoize Cassandra verion and add a backoff interval for failed schema
pulls (CASSANDRA-18902)
* Fix StackOverflowError on ALTER after many previous schema changes
(CASSANDRA-19166)
+ * Memoize Cassandra verion (CASSANDRA-18902)
Merged from 4.0:
+ * Add new concurrent_merkle_tree_requests config property to prevent OOM
during multi-range and/or multi-table repairs (CASSANDRA-19336)
- * Skip version check if an endpoint is dead state in
Gossiper#upgradeFromVersionSupplier (CASSANDRA-19187)
- * Fix Gossiper::hasMajorVersion3Nodes to return false during minor upgrade
(CASSANDRA-18999)
- * Revert unnecessary read lock acquisition when reading ring version in
TokenMetadata introduced in CASSANDRA-16286 (CASSANDRA-19107)
Merged from 3.11:
Merged from 3.0:
* Backport CASSANDRA-16418 to 3.x (CASSANDRA-18824)
diff --cc src/java/org/apache/cassandra/repair/RepairJob.java
index e88ef970af,c9966c5a05..caba564061
--- a/src/java/org/apache/cassandra/repair/RepairJob.java
+++ b/src/java/org/apache/cassandra/repair/RepairJob.java
@@@ -127,9 -124,8 +125,8 @@@ public class RepairJob extends AsyncFut
List allEndpoints = new
ArrayList<>(session.state.commonRange.endpoints);
allEndpoints.add(ctx.broadcastAddressAndPort());
- Future> treeResponses;
Future paxosRepair;
-if (paxosRepairEnabled() && ((useV2() && session.repairPaxos) ||
session.paxosOnly))
+if (paxosRepairEnabled() && (((useV2() || isMetadataKeyspace()) &&
session.repairPaxos) || session.paxosOnly))
{
logger.info("{} {}.{} starting paxos repair",
session.previewKind.logPrefix(session.getId()), desc.keyspace,
desc.columnFamily);
TableMetadata metadata =
Schema.instance.getTableMetadata(desc.keyspace, desc.columnFamily);
diff --cc src/java/org/apache/cassandra/service/ActiveRepairService.java
index cc69368248,7324b968a4..6b2e814185
--- a/src/java/org/apache/cassandra/service/ActiveRepairService.java
+++ b/src/java/org/apache/cassandra/service/ActiveRepairService.java
@@@ -82,10 -87,10 +82,11 @@@ import org.apache.cassandra.repair.NoSu
import org.apache.cassandra.repair.RepairJobDesc;
import org.apache.cassandra.repair.RepairParallelism;
import org.apache.cassandra.repair.RepairSession;
++import org.apache.cassandra.repair.Scheduler;
+import org.apache.cassandra.repair.SharedContext;
import org.apache.cassandra.repair.consistent.CoordinatorSessions;
import org.apache.cassandra.repair.consistent.LocalSessions;
+import org.apache.cassandra.repair.consistent.RepairedState;
import org.apache.cassandra.repair.consistent.admin.CleanupSummary;
import org.apache.cassandra.repair.consistent.admin.PendingStats;
import org.apache.cassandra.repair.consistent.admin.RepairStats;
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) 01/01: Merge branch 'cassandra-4.1' into cassandra-5.0
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch cassandra-5.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 5b9321eee1552a519fbefc2e07be2da8a3a9c20a Merge: 78dca99973 89a8155916 Author: Andrés de la Peña AuthorDate: Thu Feb 8 11:33:23 2024 + Merge branch 'cassandra-4.1' into cassandra-5.0 CHANGES.txt| 1 + conf/cassandra.yaml| 36 +- src/java/org/apache/cassandra/config/Config.java | 2 + .../cassandra/config/DatabaseDescriptor.java | 10 ++ .../cassandra/repair/AbstractRepairTask.java | 5 +- .../cassandra/repair/IncrementalRepairTask.java| 4 +- .../apache/cassandra/repair/NormalRepairTask.java | 4 +- .../apache/cassandra/repair/PreviewRepairTask.java | 4 +- .../apache/cassandra/repair/RepairCoordinator.java | 4 +- .../org/apache/cassandra/repair/RepairJob.java | 129 ++- .../org/apache/cassandra/repair/RepairSession.java | 3 + .../org/apache/cassandra/repair/RepairTask.java| 6 +- .../org/apache/cassandra/repair/Scheduler.java | 118 ++ .../cassandra/service/ActiveRepairService.java | 17 ++- .../service/ActiveRepairServiceMBean.java | 4 + .../test/OptimiseStreamsRepairTest.java| 17 +-- .../repair/ConcurrentValidationRequestsTest.java | 136 + .../org/apache/cassandra/repair/RepairJobTest.java | 5 +- .../apache/cassandra/repair/RepairSessionTest.java | 2 +- 19 files changed, 412 insertions(+), 95 deletions(-) diff --cc CHANGES.txt index 1549165f6c,44ca779885..4f223e469a --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,23 -1,14 +1,24 @@@ -4.1.4 +5.0-beta2 + * Fix data corruption in VectorCodec when using heap buffers (CASSANDRA-19167) + * Avoid over-skipping of key iterators from static column indexes during mixed intersections (CASSANDRA-19278) + * Make concurrent_index_builders configurable at runtime (CASSANDRA-19266) + * Fix storage_compatibility_mode for streaming (CASSANDRA-19126) + * Add support of vector type to cqlsh COPY command (CASSANDRA-19118) + * Make CQLSSTableWriter to support building of SAI indexes (CASSANDRA-18714) + * Append additional JVM options when using JDK17+ (CASSANDRA-19001) + * Upgrade Python driver to 3.29.0 (CASSANDRA-19245) + * Creating a SASI index after creating an SAI index does not break secondary index queries (CASSANDRA-18939) + * Optionally fail when a non-partition-restricted query is issued against an index (CASSANDRA-18796) + * Add a startup check to fail startup when using invalid configuration with certain Kernel and FS type (CASSANDRA-19196) + * UCS min_sstable_size should not be lower than target_sstable_size lower bound (CASSANDRA-19112) + * Fix the correspondingMessagingVersion of SSTable format and improve TTL overflow tests coverage (CASSANDRA-19197) + * Fix resource cleanup after SAI query timeouts (CASSANDRA-19177) + * Suppress CVE-2023-6481 (CASSANDRA-19184) +Merged from 4.1: * Memoize Cassandra verion and add a backoff interval for failed schema pulls (CASSANDRA-18902) * Fix StackOverflowError on ALTER after many previous schema changes (CASSANDRA-19166) - * Fixed the inconsistency between distributedKeyspaces and distributedAndLocalKeyspaces (CASSANDRA-18747) - * Internode legacy SSL storage port certificate is not hot reloaded on update (CASSANDRA-18681) - * Nodetool paxos-only repair is no longer incremental (CASSANDRA-18466) - * Waiting indefinitely on ReceivedMessage response in StreamSession#receive() can cause deadlock (CASSANDRA-18733) - * Allow empty keystore_password in encryption_options (CASSANDRA-18778) - * Skip ColumnFamilyStore#topPartitions initialization when client or tool mode (CASSANDRA-18697) Merged from 4.0: + * Add new concurrent_merkle_tree_requests config property to prevent OOM during multi-range and/or multi-table repairs (CASSANDRA-19336) * Skip version check if an endpoint is dead state in Gossiper#upgradeFromVersionSupplier (CASSANDRA-19187) * Fix Gossiper::hasMajorVersion3Nodes to return false during minor upgrade (CASSANDRA-18999) * Revert unnecessary read lock acquisition when reading ring version in TokenMetadata introduced in CASSANDRA-16286 (CASSANDRA-19107) diff --cc conf/cassandra.yaml index 0d2c711608,1986d6fa29..cc3f404d85 --- a/conf/cassandra.yaml +++ b/conf/cassandra.yaml @@@ -750,17 -658,27 +753,38 @@@ memtable_allocation_type: heap_buffer # Min unit: MiB # repair_session_space: + # The number of simultaneous Merkle tree requests during repairs that can + # be performed by a repair command. The size of each validation request is + # limited by the repair_session_space property, so setting this to 1 will make + # sure that a repair command doesn't exceed that limit, even if the repair + # command is repairing multiple tables or multiple
(cassandra) 01/01: Merge branch 'cassandra-4.0' into cassandra-4.1
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch cassandra-4.1 in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 89a8155916ff7f94db9436de3f096aa22e047e35 Merge: d9dbd5e1d0 505f5af645 Author: Andrés de la Peña AuthorDate: Thu Feb 8 11:11:07 2024 + Merge branch 'cassandra-4.0' into cassandra-4.1 CHANGES.txt| 1 + conf/cassandra.yaml| 36 +- src/java/org/apache/cassandra/config/Config.java | 2 + .../cassandra/config/DatabaseDescriptor.java | 10 ++ .../cassandra/repair/AbstractRepairTask.java | 6 +- .../cassandra/repair/IncrementalRepairTask.java| 4 +- .../apache/cassandra/repair/NormalRepairTask.java | 4 +- .../apache/cassandra/repair/PreviewRepairTask.java | 4 +- .../org/apache/cassandra/repair/RepairJob.java | 106 .../apache/cassandra/repair/RepairRunnable.java| 4 +- .../org/apache/cassandra/repair/RepairSession.java | 3 + .../org/apache/cassandra/repair/RepairTask.java| 6 +- .../org/apache/cassandra/repair/Scheduler.java | 118 ++ .../cassandra/service/ActiveRepairService.java | 17 ++- .../service/ActiveRepairServiceMBean.java | 4 + .../test/OptimiseStreamsRepairTest.java| 18 +-- .../repair/ConcurrentValidationRequestsTest.java | 136 + .../org/apache/cassandra/repair/RepairJobTest.java | 6 +- .../apache/cassandra/repair/RepairSessionTest.java | 2 +- 19 files changed, 403 insertions(+), 84 deletions(-) diff --cc CHANGES.txt index 4b8393ebc2,baeb9b7f61..44ca779885 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,13 -1,11 +1,14 @@@ -4.0.13 +4.1.4 + * Memoize Cassandra verion and add a backoff interval for failed schema pulls (CASSANDRA-18902) + * Fix StackOverflowError on ALTER after many previous schema changes (CASSANDRA-19166) + * Fixed the inconsistency between distributedKeyspaces and distributedAndLocalKeyspaces (CASSANDRA-18747) + * Internode legacy SSL storage port certificate is not hot reloaded on update (CASSANDRA-18681) + * Nodetool paxos-only repair is no longer incremental (CASSANDRA-18466) + * Waiting indefinitely on ReceivedMessage response in StreamSession#receive() can cause deadlock (CASSANDRA-18733) + * Allow empty keystore_password in encryption_options (CASSANDRA-18778) + * Skip ColumnFamilyStore#topPartitions initialization when client or tool mode (CASSANDRA-18697) +Merged from 4.0: + * Add new concurrent_merkle_tree_requests config property to prevent OOM during multi-range and/or multi-table repairs (CASSANDRA-19336) -Merged from 3.11: -Merged from 3.0: - * Backport CASSANDRA-16418 to 3.x (CASSANDRA-18824) - - -4.0.12 * Skip version check if an endpoint is dead state in Gossiper#upgradeFromVersionSupplier (CASSANDRA-19187) * Fix Gossiper::hasMajorVersion3Nodes to return false during minor upgrade (CASSANDRA-18999) * Revert unnecessary read lock acquisition when reading ring version in TokenMetadata introduced in CASSANDRA-16286 (CASSANDRA-19107) diff --cc conf/cassandra.yaml index 4b2711cfb7,7f162749d2..1986d6fa29 --- a/conf/cassandra.yaml +++ b/conf/cassandra.yaml @@@ -652,9 -565,29 +655,30 @@@ memtable_allocation_type: heap_buffer # # For more details see https://issues.apache.org/jira/browse/CASSANDRA-14096. # -# repair_session_space_in_mb: +# Min unit: MiB +# repair_session_space: + # The number of simultaneous Merkle tree requests during repairs that can + # be performed by a repair command. The size of each validation request is + # limited by the repair_session_space property, so setting this to 1 will make + # sure that a repair command doesn't exceed that limit, even if the repair + # command is repairing multiple tables or multiple virtual nodes. + # + # There isn't a limit by default for backwards compatibility, but this can + # produce OOM for commands repairing multiple tables or multiple virtual nodes. + # A limit of just 1 simultaneous Merkle tree request is generally recommended + # with no virtual nodes so repair_session_space, and thereof the Merkle tree + # resolution, can be high. For virtual nodes a value of 1 with the default + # repair_session_space value will produce higher resolution Merkle trees + # at the expense of speed. Alternatively, when working with virtual nodes it + # can make sense to reduce the repair_session_space and increase the value of + # concurrent_merkle_tree_requests because each range will contain fewer data. + # + # For more details see https://issues.apache.org/jira/browse/CASSANDRA-19336. + # + # A zero value means no limit. + # concurrent_merkle_tree_requests: 0 + # Total space to use for commit logs on disk. # # If space gets above this value, Cassandra will flush every dirty CF diff --cc src/java/org/apache/cassandra/config/Config.java index 8a59ca2cda
(cassandra) branch trunk updated (46b90364da -> 1e44a0850b)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from 46b90364da Change IP address of the CMS node during transition new c76b32492f Add support of vector type to cqlsh COPY command new 1e44a0850b Merge branch 'cassandra-5.0' into trunk The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + pylib/cqlshlib/copyutil.py | 9 +- .../apache/cassandra/tools/cqlsh/CqlshTest.java| 126 - 3 files changed, 132 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 1e44a0850b8589e5dce3d9750f2c11add713d3f8 Merge: 46b90364da c76b32492f Author: Andrés de la Peña AuthorDate: Mon Jan 22 14:37:38 2024 + Merge branch 'cassandra-5.0' into trunk CHANGES.txt| 1 + pylib/cqlshlib/copyutil.py | 9 +- .../apache/cassandra/tools/cqlsh/CqlshTest.java| 126 - 3 files changed, 132 insertions(+), 4 deletions(-) diff --cc CHANGES.txt index 290185e085,1d71cb52c3..eefc85784f --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,13 -1,5 +1,14 @@@ -5.0-beta2 +5.1 + * Limit cassandra startup to supported JDKs, allow higher JDKs by setting CASSANDRA_JDK_UNSUPPORTED (CASSANDRA-18688) + * Standardize nodetool tablestats formatting of data units (CASSANDRA-19104) + * Make nodetool tablestats use number of significant digits for time and average values consistently (CASSANDRA-19015) + * Upgrade jackson to 2.15.3 and snakeyaml to 2.1 (CASSANDRA-18875) + * Transactional Cluster Metadata [CEP-21] (CASSANDRA-18330) + * Add ELAPSED command to cqlsh (CASSANDRA-18861) + * Add the ability to disable bulk loading of SSTables (CASSANDRA-18781) + * Clean up obsolete functions and simplify cql_version handling in cqlsh (CASSANDRA-18787) +Merged from 5.0: + * Add support of vector type to cqlsh COPY command (CASSANDRA-19118) * Make CQLSSTableWriter to support building of SAI indexes (CASSANDRA-18714) * Append additional JVM options when using JDK17+ (CASSANDRA-19001) * Upgrade Python driver to 3.29.0 (CASSANDRA-19245) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch cassandra-5.0 updated: Add support of vector type to cqlsh COPY command
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cassandra-5.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-5.0 by this push:
new c76b32492f Add support of vector type to cqlsh COPY command
c76b32492f is described below
commit c76b32492f08c4af56846518488ae0b191e077e8
Author: Szymon Miężał
AuthorDate: Thu Nov 30 17:56:48 2023 +0100
Add support of vector type to cqlsh COPY command
This patch adds a converter that allows parsing vector literals
passed via csv files to the COPY command.
patch by Szymon Miezal; reviewed by Andrés de la Peña, Stefan Miklosovic
and Maxwell Guo for CASSANDRA-19118
---
CHANGES.txt| 1 +
pylib/cqlshlib/copyutil.py | 9 +-
.../apache/cassandra/tools/cqlsh/CqlshTest.java| 126 -
3 files changed, 132 insertions(+), 4 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index a7859ee9ec..1d71cb52c3 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
5.0-beta2
+ * Add support of vector type to cqlsh COPY command (CASSANDRA-19118)
* Make CQLSSTableWriter to support building of SAI indexes (CASSANDRA-18714)
* Append additional JVM options when using JDK17+ (CASSANDRA-19001)
* Upgrade Python driver to 3.29.0 (CASSANDRA-19245)
diff --git a/pylib/cqlshlib/copyutil.py b/pylib/cqlshlib/copyutil.py
index 2a8a11d1bf..af35731005 100644
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@ -46,7 +46,7 @@ from queue import Queue
from cassandra import OperationTimedOut
from cassandra.cluster import Cluster, DefaultConnection
-from cassandra.cqltypes import ReversedType, UserType, VarcharType
+from cassandra.cqltypes import ReversedType, UserType, VarcharType, VectorType
from cassandra.metadata import protect_name, protect_names, protect_value
from cassandra.policies import RetryPolicy, WhiteListRoundRobinPolicy,
DCAwareRoundRobinPolicy, FallthroughRetryPolicy
from cassandra.query import BatchStatement, BatchType, SimpleStatement,
tuple_factory
@@ -2074,6 +2074,12 @@ class ImportConversion(object):
return ImmutableDict(frozenset((convert_mandatory(ct.subtypes[0],
v[0]), convert(ct.subtypes[1], v[1]))
for v in [split(split_format_str % vv,
sep=sep) for vv in split(val)]))
+def convert_vector(val, ct=cql_type):
+string_coordinates = split(val)
+if len(string_coordinates) != ct.vector_size:
+raise ParseError("The length of given vector value '%d' is not
equal to the vector size from the type definition '%d'" %
(len(string_coordinates), ct.vector_size))
+return [convert_mandatory(ct.subtype, v) for v in
string_coordinates]
+
def convert_user_type(val, ct=cql_type):
"""
A user type is a dictionary except that we must convert each key
into
@@ -2130,6 +2136,7 @@ class ImportConversion(object):
'map': convert_map,
'tuple': convert_tuple,
'frozen': convert_single_subtype,
+VectorType.typename: convert_vector,
}
return converters.get(cql_type.typename, convert_unknown)
diff --git a/test/unit/org/apache/cassandra/tools/cqlsh/CqlshTest.java
b/test/unit/org/apache/cassandra/tools/cqlsh/CqlshTest.java
index 4e6dd2088b..356769b840 100644
--- a/test/unit/org/apache/cassandra/tools/cqlsh/CqlshTest.java
+++ b/test/unit/org/apache/cassandra/tools/cqlsh/CqlshTest.java
@@ -18,16 +18,24 @@
package org.apache.cassandra.tools.cqlsh;
+import java.io.IOException;
+import java.io.Writer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+
import org.junit.BeforeClass;
import org.junit.Test;
import org.apache.cassandra.cql3.CQLTester;
+import org.apache.cassandra.cql3.UntypedResultSet;
import org.apache.cassandra.tools.ToolRunner;
import org.apache.cassandra.tools.ToolRunner.ToolResult;
-import org.hamcrest.CoreMatchers;
+import static java.lang.String.format;
+import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
public class CqlshTest extends CQLTester
{
@@ -41,7 +49,119 @@ public class CqlshTest extends CQLTester
public void testKeyspaceRequired()
{
ToolResult tool = ToolRunner.invokeCqlsh("SELECT * FROM test");
-assertThat(tool.getCleanedStderr(),
CoreMatchers.containsStringIgnoringCase("No keyspace has been specified"));
+tool.asserts().errorContains("No keyspace has been specified");
assertEquals(2, tool.getExitCode());
}
+
+@Test
+public void testCopyFloatVector() throws IOException
+{
(cassandra) 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 03f0d37cb0120625db25d0aec52bbc6b49c3982c Merge: 89e33a16ea 8fd44ca8fc Author: Andrés de la Peña AuthorDate: Fri Jan 19 17:33:55 2024 + Merge branch 'cassandra-5.0' into trunk .build/build-resolver.xml | 2 - .build/docker/run-tests.sh | 4 +- .build/docker/ubuntu2004_test.docker | 24 +- .build/run-python-dtests.sh| 2 +- .circleci/config.yml | 877 ++-- .circleci/config.yml.FREE | 877 ++-- .circleci/config.yml.PAID | 879 ++--- .circleci/config_template.yml | 170 +--- .circleci/config_template.yml.PAID.patch | 104 +-- CHANGES.txt| 1 + NEWS.txt | 4 +- bin/cqlsh | 6 +- bin/cqlsh.py | 6 +- .../cassandra/pages/installing/installing.adoc | 2 +- lib/cassandra-driver-internal-only-3.28.0.zip | Bin 356837 -> 0 bytes lib/cassandra-driver-internal-only-3.29.0.zip | Bin 0 -> 396496 bytes pylib/Dockerfile.ubuntu.py3| 2 +- ...kerfile.ubuntu.py37 => Dockerfile.ubuntu.py311} | 2 +- pylib/README.asc | 2 +- pylib/cassandra-cqlsh-tests.sh | 2 +- pylib/cqlshlib/test/run_cqlsh.py | 2 +- pylib/requirements.txt | 2 +- 22 files changed, 298 insertions(+), 2672 deletions(-) diff --cc .circleci/config_template.yml.PAID.patch index 422dddc650,a07902b350..8b4da040d0 --- a/.circleci/config_template.yml.PAID.patch +++ b/.circleci/config_template.yml.PAID.patch @@@ -1,5 -1,5 +1,5 @@@ - --- config_template.yml 2023-11-10 16:50:30.517080584 + - +++ config_template.yml.PAID 2023-11-10 16:51:58.40394 + config_template.yml 2024-01-05 00:32:24.148600479 + -+++ config_template.yml.PAID 2024-01-05 00:32:40.861079981 + ++--- config_template.yml 2024-01-05 12:06:28.009687781 + + config_template.yml.PAID 2024-01-05 12:09:25.083221646 + @@ -157,20 +157,20 @@ j11_par_executor: _par_executor executor: diff --cc CHANGES.txt index d1ed33baee,336edfb09f..565b1d80b6 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,13 -1,5 +1,14 @@@ -5.0-beta2 +5.1 + * Limit cassandra startup to supported JDKs, allow higher JDKs by setting CASSANDRA_JDK_UNSUPPORTED (CASSANDRA-18688) + * Standardize nodetool tablestats formatting of data units (CASSANDRA-19104) + * Make nodetool tablestats use number of significant digits for time and average values consistently (CASSANDRA-19015) + * Upgrade jackson to 2.15.3 and snakeyaml to 2.1 (CASSANDRA-18875) + * Transactional Cluster Metadata [CEP-21] (CASSANDRA-18330) + * Add ELAPSED command to cqlsh (CASSANDRA-18861) + * Add the ability to disable bulk loading of SSTables (CASSANDRA-18781) + * Clean up obsolete functions and simplify cql_version handling in cqlsh (CASSANDRA-18787) +Merged from 5.0: + * Upgrade Python driver to 3.29.0 (CASSANDRA-19245) * Creating a SASI index after creating an SAI index does not break secondary index queries (CASSANDRA-18939) * Optionally fail when a non-partition-restricted query is issued against an index (CASSANDRA-18796) * Add a startup check to fail startup when using invalid configuration with certain Kernel and FS type (CASSANDRA-19196) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch trunk updated (89e33a16ea -> 03f0d37cb0)
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
from 89e33a16ea Limit cassandra startup to supported JDKs
new 8fd44ca8fc Upgrade Python driver to 3.29.0
new 03f0d37cb0 Merge branch 'cassandra-5.0' into trunk
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.build/build-resolver.xml | 2 -
.build/docker/run-tests.sh | 4 +-
.build/docker/ubuntu2004_test.docker | 24 +-
.build/run-python-dtests.sh| 2 +-
.circleci/config.yml | 877 ++--
.circleci/config.yml.FREE | 877 ++--
.circleci/config.yml.PAID | 879 ++---
.circleci/config_template.yml | 170 +---
.circleci/config_template.yml.PAID.patch | 104 +--
CHANGES.txt| 1 +
NEWS.txt | 4 +-
bin/cqlsh | 6 +-
bin/cqlsh.py | 6 +-
.../cassandra/pages/installing/installing.adoc | 2 +-
lib/cassandra-driver-internal-only-3.28.0.zip | Bin 356837 -> 0 bytes
lib/cassandra-driver-internal-only-3.29.0.zip | Bin 0 -> 396496 bytes
pylib/Dockerfile.ubuntu.py3| 2 +-
...kerfile.ubuntu.py37 => Dockerfile.ubuntu.py311} | 2 +-
pylib/README.asc | 2 +-
pylib/cassandra-cqlsh-tests.sh | 2 +-
pylib/cqlshlib/test/run_cqlsh.py | 2 +-
pylib/requirements.txt | 2 +-
22 files changed, 298 insertions(+), 2672 deletions(-)
delete mode 100644 lib/cassandra-driver-internal-only-3.28.0.zip
create mode 100644 lib/cassandra-driver-internal-only-3.29.0.zip
rename pylib/{Dockerfile.ubuntu.py37 => Dockerfile.ubuntu.py311} (80%)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch cassandra-5.0 updated: Fix SAI unindexed contexts not considering CONTAINS KEY
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cassandra-5.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-5.0 by this push:
new c26f2a3d69 Fix SAI unindexed contexts not considering CONTAINS KEY
c26f2a3d69 is described below
commit c26f2a3d6936d6ff2f14c6ab19b3727c2f86c941
Author: Andrés de la Peña
AuthorDate: Mon Nov 20 13:42:10 2023 +
Fix SAI unindexed contexts not considering CONTAINS KEY
patch by Andrés de la Peña; reviewed by Maxwell Guo and Michael Semb Wever
for CASSANDRA-19040
---
CHANGES.txt| 1 +
.../cassandra/index/sai/plan/QueryController.java | 4 ++-
.../index/sai/cql/CollectionIndexingTest.java | 40 ++
3 files changed, 38 insertions(+), 7 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index d744f50fa1..19c16fde48 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
5.0-beta1
+ * Fix SAI unindexed contexts not considering CONTAINS KEY (CASSANDRA-19040)
* Ensure that empty SAI column indexes do not fail on validation after
full-SSTable streaming (CASSANDRA-19017)
* SAI in-memory index should check max term size (CASSANDRA-18926)
* Set default disk_access_mode to mmap_index_only (CASSANDRA-19021)
diff --git a/src/java/org/apache/cassandra/index/sai/plan/QueryController.java
b/src/java/org/apache/cassandra/index/sai/plan/QueryController.java
index eee85f016e..b236c61db4 100644
--- a/src/java/org/apache/cassandra/index/sai/plan/QueryController.java
+++ b/src/java/org/apache/cassandra/index/sai/plan/QueryController.java
@@ -146,7 +146,9 @@ public class QueryController
cfs.getPartitioner(),
cfs.getComparator(),
expression.column(),
-IndexTarget.Type.VALUES,
+
expression.operator().isContainsKey()
+?
IndexTarget.Type.KEYS
+:
IndexTarget.Type.VALUES,
null)
: indexes.iterator().next().getIndexContext();
}
diff --git
a/test/unit/org/apache/cassandra/index/sai/cql/CollectionIndexingTest.java
b/test/unit/org/apache/cassandra/index/sai/cql/CollectionIndexingTest.java
index a82f539842..b490dfa946 100644
--- a/test/unit/org/apache/cassandra/index/sai/cql/CollectionIndexingTest.java
+++ b/test/unit/org/apache/cassandra/index/sai/cql/CollectionIndexingTest.java
@@ -42,28 +42,28 @@ public class CollectionIndexingTest extends SAITester
}
@Test
-public void indexMap() throws Throwable
+public void indexMap()
{
createPopulatedMap(createIndexDDL("value"));
assertEquals(2, execute("SELECT * FROM %s WHERE value CONTAINS
'v1'").size());
}
@Test
-public void indexMapKeys() throws Throwable
+public void indexMapKeys()
{
createPopulatedMap(createIndexDDL("KEYS(value)"));
assertEquals(2, execute("SELECT * FROM %s WHERE value CONTAINS KEY
1").size());
}
@Test
-public void indexMapValues() throws Throwable
+public void indexMapValues()
{
createPopulatedMap(createIndexDDL("VALUES(value)"));
assertEquals(2, execute("SELECT * FROM %s WHERE value CONTAINS
'v1'").size());
}
@Test
-public void indexMapEntries() throws Throwable
+public void indexMapEntries()
{
createPopulatedMap(createIndexDDL("ENTRIES(value)"));
assertEquals(2, execute("SELECT * FROM %s WHERE value[1] =
'v1'").size());
@@ -71,7 +71,7 @@ public class CollectionIndexingTest extends SAITester
}
@Test
-public void indexFrozenList() throws Throwable
+public void indexFrozenList()
{
createPopulatedFrozenList(createIndexDDL("FULL(value)"));
assertEquals(2, execute("SELECT * FROM %s WHERE value = ?",
Arrays.asList(1, 2, 3)).size());
@@ -176,6 +176,34 @@ public class CollectionIndexingTest extends SAITester
assertUnsupportedIndexOperator(2, "SELECT * FROM %s WHERE value[1] =
'v1'");
}
+@Test
+public void unindexedContainsExpressions()
+{
+createTable("CREATE TABLE %s (k int PRIMARY KEY, v int, m map)");
+createIndex("CREATE INDEX ON %s(v) USING 'SAI'"); // just to make sure
that SAI is involved
+
+Object[] row = row(0, 1, map(2, 3));
+execute("INSERT INTO %s (k, v, m) VALUES (?, ?, ?)", row);
+execute("INSERT INTO %s (k, v
(cassandra) branch trunk updated (1793c3fd36 -> 06dabb8b3a)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from 1793c3fd36 Merge branch 'cassandra-5.0' into trunk new c26f2a3d69 Fix SAI unindexed contexts not considering CONTAINS KEY new 06dabb8b3a Merge branch 'cassandra-5.0' into trunk The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + .../cassandra/index/sai/plan/QueryController.java | 4 ++- .../index/sai/cql/CollectionIndexingTest.java | 40 ++ 3 files changed, 38 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 06dabb8b3a97ba0d6278d398f4c6dc24d964c8d3 Merge: 1793c3fd36 c26f2a3d69 Author: Andrés de la Peña AuthorDate: Thu Nov 23 11:52:57 2023 + Merge branch 'cassandra-5.0' into trunk CHANGES.txt| 1 + .../cassandra/index/sai/plan/QueryController.java | 4 ++- .../index/sai/cql/CollectionIndexingTest.java | 40 ++ 3 files changed, 38 insertions(+), 7 deletions(-) diff --cc CHANGES.txt index 374db56c60,19c16fde48..9f4038ff7f --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,8 -1,5 +1,9 @@@ -5.0-beta1 +5.1 + * Add ELAPSED command to cqlsh (CASSANDRA-18861) + * Add the ability to disable bulk loading of SSTables (CASSANDRA-18781) + * Clean up obsolete functions and simplify cql_version handling in cqlsh (CASSANDRA-18787) +Merged from 5.0: + * Fix SAI unindexed contexts not considering CONTAINS KEY (CASSANDRA-19040) * Ensure that empty SAI column indexes do not fail on validation after full-SSTable streaming (CASSANDRA-19017) * SAI in-memory index should check max term size (CASSANDRA-18926) * Set default disk_access_mode to mmap_index_only (CASSANDRA-19021) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 1793c3fd36837213d8938c4a78f9911baade3949 Merge: 316a239c7c e8fb4b2f63 Author: Andrés de la Peña AuthorDate: Wed Nov 22 15:15:36 2023 + Merge branch 'cassandra-5.0' into trunk CHANGES.txt| 1 + .../sai/disk/v1/ColumnCompletionMarkerUtil.java| 74 + .../index/sai/disk/v1/MemtableIndexWriter.java | 6 +- .../index/sai/disk/v1/SSTableIndexWriter.java | 4 +- .../index/sai/disk/v1/V1OnDiskFormat.java | 77 +- .../distributed/test/sai/IndexStreamingTest.java | 23 +++ 6 files changed, 140 insertions(+), 45 deletions(-) diff --cc CHANGES.txt index b4b0e8abd3,d744f50fa1..374db56c60 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,8 -1,5 +1,9 @@@ -5.0-beta1 +5.1 + * Add ELAPSED command to cqlsh (CASSANDRA-18861) + * Add the ability to disable bulk loading of SSTables (CASSANDRA-18781) + * Clean up obsolete functions and simplify cql_version handling in cqlsh (CASSANDRA-18787) +Merged from 5.0: + * Ensure that empty SAI column indexes do not fail on validation after full-SSTable streaming (CASSANDRA-19017) * SAI in-memory index should check max term size (CASSANDRA-18926) * Set default disk_access_mode to mmap_index_only (CASSANDRA-19021) * Exclude net.java.dev.jna:jna dependency from dependencies of org.caffinitas.ohc:ohc-core (CASSANDRA-18992) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch trunk updated (316a239c7c -> 1793c3fd36)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from 316a239c7c Merge branch 'cassandra-5.0' into trunk new e8fb4b2f63 Ensure that empty SAI column indexes do not fail on validation after full-SSTable streaming new 1793c3fd36 Merge branch 'cassandra-5.0' into trunk The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + .../sai/disk/v1/ColumnCompletionMarkerUtil.java| 74 + .../index/sai/disk/v1/MemtableIndexWriter.java | 6 +- .../index/sai/disk/v1/SSTableIndexWriter.java | 4 +- .../index/sai/disk/v1/V1OnDiskFormat.java | 77 +- .../distributed/test/sai/IndexStreamingTest.java | 23 +++ 6 files changed, 140 insertions(+), 45 deletions(-) create mode 100644 src/java/org/apache/cassandra/index/sai/disk/v1/ColumnCompletionMarkerUtil.java - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch cassandra-5.0 updated: Ensure that empty SAI column indexes do not fail on validation after full-SSTable streaming
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cassandra-5.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-5.0 by this push:
new e8fb4b2f63 Ensure that empty SAI column indexes do not fail on
validation after full-SSTable streaming
e8fb4b2f63 is described below
commit e8fb4b2f63b32f337447992f9eb57a12e2afc0e4
Author: Andrés de la Peña
AuthorDate: Mon Nov 13 12:46:53 2023 +
Ensure that empty SAI column indexes do not fail on validation after
full-SSTable streaming
patch by Andrés de la Peña; reviewed by Caleb Rackliffe for CASSANDRA-19017
Co-authored-by: Andrés de la Peña
Co-authored-by: Caleb Rackliffe
---
CHANGES.txt| 1 +
.../sai/disk/v1/ColumnCompletionMarkerUtil.java| 74 +
.../index/sai/disk/v1/MemtableIndexWriter.java | 6 +-
.../index/sai/disk/v1/SSTableIndexWriter.java | 4 +-
.../index/sai/disk/v1/V1OnDiskFormat.java | 77 +-
.../distributed/test/sai/IndexStreamingTest.java | 23 +++
6 files changed, 140 insertions(+), 45 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 9a4e2fa287..d744f50fa1 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
5.0-beta1
+ * Ensure that empty SAI column indexes do not fail on validation after
full-SSTable streaming (CASSANDRA-19017)
* SAI in-memory index should check max term size (CASSANDRA-18926)
* Set default disk_access_mode to mmap_index_only (CASSANDRA-19021)
* Exclude net.java.dev.jna:jna dependency from dependencies of
org.caffinitas.ohc:ohc-core (CASSANDRA-18992)
diff --git
a/src/java/org/apache/cassandra/index/sai/disk/v1/ColumnCompletionMarkerUtil.java
b/src/java/org/apache/cassandra/index/sai/disk/v1/ColumnCompletionMarkerUtil.java
new file mode 100644
index 00..760083ad9d
--- /dev/null
+++
b/src/java/org/apache/cassandra/index/sai/disk/v1/ColumnCompletionMarkerUtil.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.index.sai.disk.v1;
+
+import java.io.IOException;
+
+import org.apache.cassandra.index.sai.IndexContext;
+import org.apache.cassandra.index.sai.disk.format.IndexComponent;
+import org.apache.cassandra.index.sai.disk.format.IndexDescriptor;
+import org.apache.cassandra.index.sai.disk.io.IndexOutputWriter;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+
+/**
+ * Utility class for creating and reading the column completion marker, {@link
IndexComponent#COLUMN_COMPLETION_MARKER}.
+ *
+ * The file has a header and a footer, as written by {@link
SAICodecUtils#writeHeader(IndexOutput)} and
+ * {@link SAICodecUtils#writeFooter(IndexOutput)}. The only content of the
file is a single byte indicating whether the
+ * column index is empty or not. If the index is empty the completion marker
will be the only per-index component.
+ */
+public class ColumnCompletionMarkerUtil
+{
+private static final byte EMPTY = (byte) 1;
+private static final byte NOT_EMPTY = (byte) 0;
+
+/**
+ * Creates a column index completion marker for the specified column
index, storing in it whether the index is empty.
+ *
+ * @param descriptor the index descriptor
+ * @param context the column index context
+ * @param isEmpty whether the index is empty
+ */
+public static void create(IndexDescriptor descriptor, IndexContext
context, boolean isEmpty) throws IOException
+{
+try (IndexOutputWriter output =
descriptor.openPerIndexOutput(IndexComponent.COLUMN_COMPLETION_MARKER, context))
+{
+SAICodecUtils.writeHeader(output);
+output.writeByte(isEmpty ? EMPTY : NOT_EMPTY);
+SAICodecUtils.writeFooter(output);
+}
+}
+
+/**
+ * Reads the column index completion marker and returns whether if the
index is empty.
+ *
+ * @param descriptor the index descriptor
+ * @param context the column index context
+ * @return {@code true} if the index is empty, {@c
(cassandra) branch trunk updated (1463fabac3 -> 1b7e895f56)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from 1463fabac3 Merge branch 'cassandra-5.0' into trunk new 91a242fe00 SAI in-memory index should check max term size new 1b7e895f56 Merge branch 'cassandra-5.0' into trunk The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + .../apache/cassandra/index/sai/IndexContext.java | 76 ++ .../cassandra/index/sai/StorageAttachedIndex.java | 6 +- .../index/sai/disk/v1/SSTableIndexWriter.java | 23 +-- .../index/sai/memory/TrieMemoryIndex.java | 2 + .../index/sai/memory/VectorMemoryIndex.java| 2 +- test/unit/org/apache/cassandra/cql3/CQLTester.java | 6 ++ .../index/sai/cql/StorageAttachedIndexDDLTest.java | 47 + 8 files changed, 139 insertions(+), 24 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 1b7e895f56ac4636f04b3646af4f9bf7d45e74d1 Merge: 1463fabac3 91a242fe00 Author: Andrés de la Peña AuthorDate: Mon Nov 20 16:04:40 2023 + Merge branch 'cassandra-5.0' into trunk CHANGES.txt| 1 + .../apache/cassandra/index/sai/IndexContext.java | 76 ++ .../cassandra/index/sai/StorageAttachedIndex.java | 6 +- .../index/sai/disk/v1/SSTableIndexWriter.java | 23 +-- .../index/sai/memory/TrieMemoryIndex.java | 2 + .../index/sai/memory/VectorMemoryIndex.java| 2 +- test/unit/org/apache/cassandra/cql3/CQLTester.java | 6 ++ .../index/sai/cql/StorageAttachedIndexDDLTest.java | 47 + 8 files changed, 139 insertions(+), 24 deletions(-) diff --cc CHANGES.txt index 72ea51472c,9a4e2fa287..b4b0e8abd3 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,8 -1,5 +1,9 @@@ -5.0-beta1 +5.1 + * Add ELAPSED command to cqlsh (CASSANDRA-18861) + * Add the ability to disable bulk loading of SSTables (CASSANDRA-18781) + * Clean up obsolete functions and simplify cql_version handling in cqlsh (CASSANDRA-18787) +Merged from 5.0: + * SAI in-memory index should check max term size (CASSANDRA-18926) * Set default disk_access_mode to mmap_index_only (CASSANDRA-19021) * Exclude net.java.dev.jna:jna dependency from dependencies of org.caffinitas.ohc:ohc-core (CASSANDRA-18992) * Add UCS sstable_growth and min_sstable_size options (CASSANDRA-18945) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch cassandra-5.0 updated: SAI in-memory index should check max term size
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cassandra-5.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-5.0 by this push:
new 91a242fe00 SAI in-memory index should check max term size
91a242fe00 is described below
commit 91a242fe00c9cc740d6d80df61240eebf05f4458
Author: Andrés de la Peña
AuthorDate: Fri Nov 17 15:28:29 2023 +
SAI in-memory index should check max term size
patch by Zhao Yang and Andrés de la Peña; reviewed by Zhao Yang for
CASSANDRA-18926
Co-authored-by: Zhao Yang
Co-authored-by: Andrés de la Peña
---
CHANGES.txt| 1 +
.../apache/cassandra/index/sai/IndexContext.java | 76 ++
.../cassandra/index/sai/StorageAttachedIndex.java | 6 +-
.../index/sai/disk/v1/SSTableIndexWriter.java | 23 +--
.../index/sai/memory/TrieMemoryIndex.java | 2 +
.../index/sai/memory/VectorMemoryIndex.java| 2 +-
test/unit/org/apache/cassandra/cql3/CQLTester.java | 6 ++
.../index/sai/cql/StorageAttachedIndexDDLTest.java | 47 +
8 files changed, 139 insertions(+), 24 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 5102f80bff..9a4e2fa287 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
5.0-beta1
+ * SAI in-memory index should check max term size (CASSANDRA-18926)
* Set default disk_access_mode to mmap_index_only (CASSANDRA-19021)
* Exclude net.java.dev.jna:jna dependency from dependencies of
org.caffinitas.ohc:ohc-core (CASSANDRA-18992)
* Add UCS sstable_growth and min_sstable_size options (CASSANDRA-18945)
diff --git a/src/java/org/apache/cassandra/index/sai/IndexContext.java
b/src/java/org/apache/cassandra/index/sai/IndexContext.java
index 26ef903c40..b88d59b6e6 100644
--- a/src/java/org/apache/cassandra/index/sai/IndexContext.java
+++ b/src/java/org/apache/cassandra/index/sai/IndexContext.java
@@ -25,6 +25,7 @@ import java.util.HashSet;
import java.util.Iterator;
import java.util.Objects;
import java.util.Set;
+import java.util.concurrent.TimeUnit;
import javax.annotation.Nullable;
import com.google.common.base.MoreObjects;
@@ -62,14 +63,28 @@ import org.apache.cassandra.index.sai.view.View;
import org.apache.cassandra.io.sstable.format.SSTableReader;
import org.apache.cassandra.schema.ColumnMetadata;
import org.apache.cassandra.schema.IndexMetadata;
+import org.apache.cassandra.service.ClientWarn;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.NoSpamLogger;
import org.apache.cassandra.utils.Pair;
+import static
org.apache.cassandra.config.CassandraRelevantProperties.SAI_MAX_FROZEN_TERM_SIZE;
+import static
org.apache.cassandra.config.CassandraRelevantProperties.SAI_MAX_STRING_TERM_SIZE;
+import static
org.apache.cassandra.config.CassandraRelevantProperties.SAI_MAX_VECTOR_TERM_SIZE;
+
/**
* Manages metadata for each column index.
*/
public class IndexContext
{
private static final Logger logger =
LoggerFactory.getLogger(IndexContext.class);
+private static final NoSpamLogger noSpamLogger =
NoSpamLogger.getLogger(logger, 1, TimeUnit.MINUTES);
+
+public static final long MAX_STRING_TERM_SIZE =
SAI_MAX_STRING_TERM_SIZE.getSizeInBytes();
+public static final long MAX_FROZEN_TERM_SIZE =
SAI_MAX_FROZEN_TERM_SIZE.getSizeInBytes();
+public static final long MAX_VECTOR_TERM_SIZE =
SAI_MAX_VECTOR_TERM_SIZE.getSizeInBytes();
+public static final String TERM_OVERSIZE_MESSAGE = "Can't add term of
column %s to index for key: %s, term size %s " +
+ "max allowed size %s,
use analyzed = true (if not yet set) for that column.";
private static final Set> EQ_ONLY_TYPES =
ImmutableSet.of(UTF8Type.instance,
AsciiType.instance,
@@ -98,6 +113,8 @@ public class IndexContext
private final AbstractAnalyzer.AnalyzerFactory analyzerFactory;
private final PrimaryKey.Factory primaryKeyFactory;
+private final long maxTermSize;
+
public IndexContext(String keyspace,
String table,
AbstractType partitionKeyType,
@@ -128,6 +145,8 @@ public class IndexContext
this.analyzerFactory = indexMetadata == null ?
AbstractAnalyzer.fromOptions(getValidator(), Collections.emptyMap())
:
AbstractAnalyzer.fromOptions(getValidator(), indexMetadata.options);
+
+maxTermSize = isVector() ? MAX_VECTOR_TERM_SIZE : (isFrozen() ?
MAX_FROZEN_TERM_SIZE : MAX_STRING_TERM_SIZE);
}
public boolean hasClustering()
@@ -509,4 +528,61 @@ public class IndexContext
.mapToLong(SSTableIndex::indexFileCacheSize)
(cassandra) 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit a77951ec0c1919f6510e3ec9c4451e3124a19e92 Merge: c6cb8523b3 b91c0fcb6e Author: Andrés de la Peña AuthorDate: Tue Nov 14 11:48:35 2023 + Merge branch 'cassandra-5.0' into trunk .build/docker/ubuntu2004_test.docker | 2 +- .circleci/config.yml | 170 ++--- .circleci/config.yml.FREE | 170 ++--- .circleci/config.yml.PAID | 170 ++--- .circleci/config_template.yml | 26 ++-- .circleci/config_template.yml.PAID.patch | 16 +- CHANGES.txt| 1 + NEWS.txt | 2 + bin/cqlsh | 4 +- bin/cqlsh.py | 4 +- .../cassandra/pages/installing/installing.adoc | 2 +- pylib/Dockerfile.ubuntu.py3| 2 +- pylib/README.asc | 2 +- pylib/cassandra-cqlsh-tests.sh | 2 +- 14 files changed, 288 insertions(+), 285 deletions(-) diff --cc .circleci/config_template.yml.PAID.patch index 7ca86ce04f,53d3644476..422dddc650 --- a/.circleci/config_template.yml.PAID.patch +++ b/.circleci/config_template.yml.PAID.patch @@@ -1,5 -1,5 +1,5 @@@ - --- config_template.yml 2023-08-11 10:06:39.0 -0400 - +++ config_template.yml.PAID 2023-08-11 10:22:35.0 -0400 config_template.yml 2023-11-10 15:39:02.663105186 + -+++ config_template.yml.PAID 2023-11-10 15:39:24.839456426 + ++--- config_template.yml 2023-11-10 16:50:30.517080584 + + config_template.yml.PAID 2023-11-10 16:51:58.40394 + @@ -157,20 +157,20 @@ j11_par_executor: _par_executor executor: diff --cc CHANGES.txt index c77d936885,8988be9fda..bfa639fff2 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,8 -1,5 +1,9 @@@ -5.0-beta1 +5.1 + * Add ELAPSED command to cqlsh (CASSANDRA-18861) + * Add the ability to disable bulk loading of SSTables (CASSANDRA-18781) + * Clean up obsolete functions and simplify cql_version handling in cqlsh (CASSANDRA-18787) +Merged from 5.0: + * Make cqlsh's min required Python version 3.7+ instead of 3.6+ (CASSANDRA-18960) * Fix incorrect seeking through the sstable iterator by IndexState (CASSANDRA-18932) * Upgrade Python driver to 3.28.0 (CASSANDRA-18960) * Add retries to IR messages (CASSANDRA-18962) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch trunk updated (c6cb8523b3 -> a77951ec0c)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from c6cb8523b3 Merge branch 'cassandra-5.0' into trunk new b91c0fcb6e Change cqlsh's min required Python version from 3.6+ to 3.7+ new a77951ec0c Merge branch 'cassandra-5.0' into trunk The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .build/docker/ubuntu2004_test.docker | 2 +- .circleci/config.yml | 170 ++--- .circleci/config.yml.FREE | 170 ++--- .circleci/config.yml.PAID | 170 ++--- .circleci/config_template.yml | 26 ++-- .circleci/config_template.yml.PAID.patch | 16 +- CHANGES.txt| 1 + NEWS.txt | 2 + bin/cqlsh | 4 +- bin/cqlsh.py | 4 +- .../cassandra/pages/installing/installing.adoc | 2 +- pylib/Dockerfile.ubuntu.py3| 2 +- pylib/README.asc | 2 +- pylib/cassandra-cqlsh-tests.sh | 2 +- 14 files changed, 288 insertions(+), 285 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch cassandra-5.0 updated: Upgrade Python driver to 3.28.0
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cassandra-5.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-5.0 by this push:
new f41ecf586c Upgrade Python driver to 3.28.0
f41ecf586c is described below
commit f41ecf586c0459476d68e755c6431119f08ff4e1
Author: Andrés de la Peña
AuthorDate: Thu Nov 2 13:59:58 2023 +
Upgrade Python driver to 3.28.0
patch by Andrés de la Peña and Jeremiah Jordan; reviewed by Brandon
Williams for CASSANDRA-18960
Co-authored-by: Jeremiah Jordan
Co-authored-by: Andrés de la Peña
---
CHANGES.txt | 1 +
lib/cassandra-driver-internal-only-3.25.0.zip | Bin 345177 -> 0 bytes
lib/cassandra-driver-internal-only-3.28.0.zip | Bin 0 -> 356837 bytes
pylib/cqlshlib/formatting.py | 15 +++
pylib/cqlshlib/test/test_cqlsh_output.py | 22 ++
5 files changed, 34 insertions(+), 4 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index f288f3206e..f54569f342 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
5.0-alpha2
+ * Upgrade Python driver to 3.28.0 (CASSANDRA-18960)
* Add retries to IR messages (CASSANDRA-18962)
* Add metrics and logging to repair retries (CASSANDRA-18952)
* Remove deprecated code in Cassandra 1.x and 2.x (CASSANDRA-18959)
diff --git a/lib/cassandra-driver-internal-only-3.25.0.zip
b/lib/cassandra-driver-internal-only-3.25.0.zip
deleted file mode 100644
index ecfd6a946e..00
Binary files a/lib/cassandra-driver-internal-only-3.25.0.zip and /dev/null
differ
diff --git a/lib/cassandra-driver-internal-only-3.28.0.zip
b/lib/cassandra-driver-internal-only-3.28.0.zip
new file mode 100644
index 00..07e0379447
Binary files /dev/null and b/lib/cassandra-driver-internal-only-3.28.0.zip
differ
diff --git a/pylib/cqlshlib/formatting.py b/pylib/cqlshlib/formatting.py
index 4eb36581c6..23bf1efbbf 100644
--- a/pylib/cqlshlib/formatting.py
+++ b/pylib/cqlshlib/formatting.py
@@ -149,8 +149,8 @@ class CqlType:
def parse(self, typestring, ksmeta):
"""
Parse the typestring by looking at this pattern: *<*>. If there is no
match then the type
-is either a simple type or a user type, otherwise it must be a
composite type
-for which we need to look-up the sub-types. For user types the sub
types can be extracted
+is either a simple type or a user type, otherwise it must be a
composite type or a vector type,
+for which we need to look up the subtypes. For user types the subtypes
can be extracted
from the keyspace metadata.
"""
while True:
@@ -167,8 +167,15 @@ class CqlType:
typestring = m.group(2)
continue
-name = m.group(1) # a composite type, parse sub types
-return name, self.parse_sub_types(m.group(2), ksmeta),
self._get_formatter(name)
+name = m.group(1) # a composite or vector type, parse subtypes
+try:
+# Vector types are parameterized as name so add
custom handling for that here
+type_args = m.group(2).split(',')
+vector_type = CqlType(type_args[0])
+vector_size = int(type_args[1])
+return name, [vector_type for _ in range(vector_size)],
self._get_formatter(name)
+except (ValueError, IndexError):
+return name, self.parse_sub_types(m.group(2), ksmeta),
self._get_formatter(name)
@staticmethod
def _get_formatter(name):
diff --git a/pylib/cqlshlib/test/test_cqlsh_output.py
b/pylib/cqlshlib/test/test_cqlsh_output.py
index c17def624a..9ca7988d2e 100644
--- a/pylib/cqlshlib/test/test_cqlsh_output.py
+++ b/pylib/cqlshlib/test/test_cqlsh_output.py
@@ -935,6 +935,28 @@ class TestCqlshOutput(BaseTestCase):
"""),
))
+def test_vectors_output(self):
+self.assertQueriesGiveColoredOutput((
+("SELECT vectorcol FROM has_all_types;", r"""
+ vectorcol
+ M
+---
+
+ [1, -2, NaN]
+ BGBBGGBBGGGB
+ [1, 2, 3]
+ BGBBGBBGB
+ [0, 0, 0]
+ BGBBGBBGB
+ null
+
+ [1e+08, 1e+08, 1e+08]
+ BGBBGBBGB
+
+(5 rows)
+"""),
+))
+
def test_expanded_output_counts_past_page(self):
query = "PAGING 5; EXPAND ON; SELECT * FROM twenty_rows_table;"
output, result = testcall_cqlsh(prompt=None, env=
(cassandra) 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 53c25f8faa68065040c37f29f3f7f0f0b7161a98
Merge: bb686fc27d f41ecf586c
Author: Andrés de la Peña
AuthorDate: Thu Nov 2 17:10:13 2023 +
Merge branch 'cassandra-5.0' into trunk
CHANGES.txt | 1 +
lib/cassandra-driver-internal-only-3.25.0.zip | Bin 345177 -> 0 bytes
lib/cassandra-driver-internal-only-3.28.0.zip | Bin 0 -> 356837 bytes
pylib/cqlshlib/formatting.py | 15 +++
pylib/cqlshlib/test/test_cqlsh_output.py | 22 ++
5 files changed, 34 insertions(+), 4 deletions(-)
diff --cc CHANGES.txt
index c7bbed34a0,f54569f342..d7a0a03b6d
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,8 -1,5 +1,9 @@@
-5.0-alpha2
+5.1
+ * Add ELAPSED command to cqlsh (CASSANDRA-18861)
+ * Add the ability to disable bulk loading of SSTables (CASSANDRA-18781)
+ * Clean up obsolete functions and simplify cql_version handling in cqlsh
(CASSANDRA-18787)
+Merged from 5.0:
+ * Upgrade Python driver to 3.28.0 (CASSANDRA-18960)
* Add retries to IR messages (CASSANDRA-18962)
* Add metrics and logging to repair retries (CASSANDRA-18952)
* Remove deprecated code in Cassandra 1.x and 2.x (CASSANDRA-18959)
diff --cc pylib/cqlshlib/test/test_cqlsh_output.py
index 3cb6d51300,9ca7988d2e..65ac5be482
--- a/pylib/cqlshlib/test/test_cqlsh_output.py
+++ b/pylib/cqlshlib/test/test_cqlsh_output.py
@@@ -935,9 -935,31 +935,31 @@@ class TestCqlshOutput(BaseTestCase)
"""),
))
+ def test_vectors_output(self):
+ self.assertQueriesGiveColoredOutput((
+ ("SELECT vectorcol FROM has_all_types;", r"""
+ vectorcol
+ M
+ ---
+
+ [1, -2, NaN]
+ BGBBGGBBGGGB
+ [1, 2, 3]
+ BGBBGBBGB
+ [0, 0, 0]
+ BGBBGBBGB
+ null
+
+ [1e+08, 1e+08, 1e+08]
+ BGBBGBBGB
+
+ (5 rows)
+ """),
+ ))
+
def test_expanded_output_counts_past_page(self):
query = "PAGING 5; EXPAND ON; SELECT * FROM twenty_rows_table;"
-output, result = testcall_cqlsh(prompt=None, env=self.default_env,
+output, result = cqlsh_testcall(prompt=None, env=self.default_env,
tty=False, input=query)
self.assertEqual(0, result)
# format is "@ Row 1"
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch trunk updated (bb686fc27d -> 53c25f8faa)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from bb686fc27d Merge branch 'cassandra-5.0' into trunk new f41ecf586c Upgrade Python driver to 3.28.0 new 53c25f8faa Merge branch 'cassandra-5.0' into trunk The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt | 1 + lib/cassandra-driver-internal-only-3.25.0.zip | Bin 345177 -> 0 bytes lib/cassandra-driver-internal-only-3.28.0.zip | Bin 0 -> 356837 bytes pylib/cqlshlib/formatting.py | 15 +++ pylib/cqlshlib/test/test_cqlsh_output.py | 22 ++ 5 files changed, 34 insertions(+), 4 deletions(-) delete mode 100644 lib/cassandra-driver-internal-only-3.25.0.zip create mode 100644 lib/cassandra-driver-internal-only-3.28.0.zip - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit d6159cfe151316964918407b2b4099d5678892eb Merge: 59eb8ba043 c4d11c4372 Author: Andrés de la Peña AuthorDate: Mon Oct 30 19:04:54 2023 + Merge branch 'cassandra-5.0' into trunk NEWS.txt| 1 + .../cassandra/index/sai/plan/QueryController.java | 6 +++--- .../index/sai/cql/VectorUpdateDeleteTest.java | 21 - 3 files changed, 8 insertions(+), 20 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch trunk updated (59eb8ba043 -> d6159cfe15)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from 59eb8ba043 Merge branch 'cassandra-5.0' into trunk new c4d11c4372 Fix VectorUpdateDeleteTest for JDK 17 Removed use of reflection and directly set relevant property to avoid jdk 17 errors new d6159cfe15 Merge branch 'cassandra-5.0' into trunk The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: NEWS.txt| 1 + .../cassandra/index/sai/plan/QueryController.java | 6 +++--- .../index/sai/cql/VectorUpdateDeleteTest.java | 21 - 3 files changed, 8 insertions(+), 20 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
(cassandra) branch cassandra-5.0 updated: Fix VectorUpdateDeleteTest for JDK 17 Removed use of reflection and directly set relevant property to avoid jdk 17 errors
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cassandra-5.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-5.0 by this push:
new c4d11c4372 Fix VectorUpdateDeleteTest for JDK 17 Removed use of
reflection and directly set relevant property to avoid jdk 17 errors
c4d11c4372 is described below
commit c4d11c4372906ae1dea9e6c31c1136f122e8a1b2
Author: Mike Adamson
AuthorDate: Mon Oct 30 09:46:52 2023 +
Fix VectorUpdateDeleteTest for JDK 17
Removed use of reflection and directly set
relevant property to avoid jdk 17 errors
patch by Mike Adamson; reviewed by Stefan Miklosovic, Michael Semb Wever
and Andrés de la Peña for CASSANDRA-18715
---
.../cassandra/index/sai/plan/QueryController.java | 6 +++---
.../index/sai/cql/VectorUpdateDeleteTest.java | 21 -
2 files changed, 7 insertions(+), 20 deletions(-)
diff --git a/src/java/org/apache/cassandra/index/sai/plan/QueryController.java
b/src/java/org/apache/cassandra/index/sai/plan/QueryController.java
index 018785647a..61101cabf6 100644
--- a/src/java/org/apache/cassandra/index/sai/plan/QueryController.java
+++ b/src/java/org/apache/cassandra/index/sai/plan/QueryController.java
@@ -67,8 +67,6 @@ import static
org.apache.cassandra.config.CassandraRelevantProperties.SAI_VECTOR
public class QueryController
{
-public static final int ORDER_CHUNK_SIZE =
SAI_VECTOR_SEARCH_ORDER_CHUNK_SIZE.getInt();
-
private final ColumnFamilyStore cfs;
private final ReadCommand command;
private final QueryContext queryContext;
@@ -79,6 +77,7 @@ public class QueryController
private final PrimaryKey.Factory keyFactory;
private final PrimaryKey firstPrimaryKey;
private final PrimaryKey lastPrimaryKey;
+private final int orderChunkSize;
public QueryController(ColumnFamilyStore cfs,
ReadCommand command,
@@ -98,6 +97,7 @@ public class QueryController
this.keyFactory = new PrimaryKey.Factory(cfs.getPartitioner(),
cfs.getComparator());
this.firstPrimaryKey = keyFactory.create(mergeRange.left.getToken());
this.lastPrimaryKey = keyFactory.create(mergeRange.right.getToken());
+this.orderChunkSize = SAI_VECTOR_SEARCH_ORDER_CHUNK_SIZE.getInt();
}
public PrimaryKey.Factory primaryKeyFactory()
@@ -275,7 +275,7 @@ public class QueryController
// This is a hybrid query. We apply all other predicates before ordering
and limiting.
public KeyRangeIterator getTopKRows(KeyRangeIterator source,
RowFilter.Expression expression)
{
-return new KeyRangeOrderingIterator(source, ORDER_CHUNK_SIZE, list ->
this.getTopKRows(list, expression));
+return new KeyRangeOrderingIterator(source, orderChunkSize, list ->
this.getTopKRows(list, expression));
}
private KeyRangeIterator getTopKRows(List rawSourceKeys,
RowFilter.Expression expression)
diff --git
a/test/unit/org/apache/cassandra/index/sai/cql/VectorUpdateDeleteTest.java
b/test/unit/org/apache/cassandra/index/sai/cql/VectorUpdateDeleteTest.java
index f2fa362b4e..20842e6184 100644
--- a/test/unit/org/apache/cassandra/index/sai/cql/VectorUpdateDeleteTest.java
+++ b/test/unit/org/apache/cassandra/index/sai/cql/VectorUpdateDeleteTest.java
@@ -18,14 +18,11 @@
package org.apache.cassandra.index.sai.cql;
-import java.lang.reflect.Field;
-import java.lang.reflect.Modifier;
-
import org.apache.cassandra.cql3.UntypedResultSet;
-import org.apache.cassandra.index.sai.plan.QueryController;
import org.junit.Test;
+import static
org.apache.cassandra.config.CassandraRelevantProperties.SAI_VECTOR_SEARCH_ORDER_CHUNK_SIZE;
import static
org.apache.cassandra.index.sai.cql.VectorTypeTest.assertContainsInt;
import static org.assertj.core.api.Assertions.assertThat;
@@ -528,7 +525,7 @@ public class VectorUpdateDeleteTest extends VectorTester
// We also query for different LIMITs
for (int i = 1; i <= 100; i++)
{
-setChunkSize(i);
+SAI_VECTOR_SEARCH_ORDER_CHUNK_SIZE.setInt(i);
var results = execute("SELECT pk FROM %s WHERE str_val = 'A'
ORDER BY vec ANN OF [1,1] LIMIT 1");
assertRows(results, row(1));
results = execute("SELECT pk FROM %s WHERE str_val = 'A' ORDER
BY vec ANN OF [1,1] LIMIT 3");
@@ -543,19 +540,9 @@ public class VectorUpdateDeleteTest extends VectorTester
finally
{
// Revert to prevent interference with other tests. Note that a
decreased chunk size can impact
-// wether we compute the topk with brute force because it
determines how many vectors get sent to the
+// whether we compute the topk with brute force because it
determines how many vectors get sent to the
[cassandra] branch cassandra-5.0 updated: Add cqlsh autocompletion for the vector data type
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cassandra-5.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-5.0 by this push:
new 5eba52feb8 Add cqlsh autocompletion for the vector data type
5eba52feb8 is described below
commit 5eba52feb83d5cc68b7c2dea39eb64cad2e5338d
Author: Andrés de la Peña
AuthorDate: Fri Oct 20 13:31:25 2023 +0100
Add cqlsh autocompletion for the vector data type
patch by Andrés de la Peña; reviewed by Brandon Williams for CASSANDRA-18946
---
CHANGES.txt | 1 +
pylib/cqlshlib/cql3handling.py | 20 +---
pylib/cqlshlib/test/test_cqlsh_completion.py | 36
pylib/cqlshlib/test/test_cqlsh_output.py | 12 ++
pylib/cqlshlib/test/test_keyspace_init.cql | 30 ---
5 files changed, 73 insertions(+), 26 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 517a20cdb8..317954aafb 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
5.0-alpha2
+ * Add cqlsh autocompletion for the vector data type (CASSANDRA-18946)
* Fix nodetool tablehistograms output to avoid printing repeated information
and ensure at most two arguments (CASSANDRA-18955)
* Change the checksum algorithm SAI-related files use from CRC32 to CRC32C
(CASSANDRA-18836)
* Correctly remove Index.Group from IndexRegistry (CASSANDRA-18905)
diff --git a/pylib/cqlshlib/cql3handling.py b/pylib/cqlshlib/cql3handling.py
index 39b08fcb1b..3c6518a0b9 100644
--- a/pylib/cqlshlib/cql3handling.py
+++ b/pylib/cqlshlib/cql3handling.py
@@ -21,7 +21,7 @@ from cqlshlib.cqlhandling import CqlParsingRuleSet, Hint
simple_cql_types = {'ascii', 'bigint', 'blob', 'boolean', 'counter', 'date',
'decimal', 'double', 'duration', 'float',
'inet', 'int', 'smallint', 'text', 'time', 'timestamp',
'timeuuid', 'tinyint', 'uuid', 'varchar',
'varint'}
-simple_cql_types.difference_update(('set', 'map', 'list'))
+simple_cql_types.difference_update(('set', 'map', 'list', 'vector'))
cqldocs = helptopics.CQL3HelpTopics()
@@ -305,7 +305,7 @@ JUNK ::= /([
\t\r\f\v]+|(--|[/][/])[^\n\r]*([\n\r]|$)|[/][*].*?[*][/])/ ;
::= utname= ;
- ::= ( | |
| ) ( )? ;
+ ::= ( | |
| | ) ( )? ;
::= "MASKED" "WITH" ( "DEFAULT" |
);
@@ -321,6 +321,8 @@ JUNK ::= /([
\t\r\f\v]+|(--|[/][/])[^\n\r]*([\n\r]|$)|[/][*].*?[*][/])/ ;
| "frozen" "<" "set" "<" ">" ">"
;
+ ::= "vector" "<" "," ">" ;
+
::= ( ksname= dot="." )? cfname= ;
::= ( ksname= dot="." )?
mvname= ;
@@ -922,9 +924,9 @@ def insert_newval_completer(ctxt, cass):
return []
curcol = insertcols[len(valuesdone)]
coltype = layout.columns[curcol].cql_type
-if coltype in ('map', 'set'):
+if coltype.startswith('map<') or coltype.startswith('set<'):
return ['{']
-if coltype == 'list':
+if coltype.startswith('list<') or coltype.startswith('vector<'):
return ['[']
if coltype == 'boolean':
return ['true', 'false']
@@ -999,10 +1001,10 @@ def update_countername_completer(ctxt, cass):
coltype = layout.columns[curcol].cql_type
if coltype == 'counter':
return [maybe_escape_name(curcol)]
-if coltype in ('map', 'set'):
-return ["{"]
-if coltype == 'list':
-return ["["]
+if coltype.startswith('map<') or coltype.startswith('set<'):
+return ['{']
+if coltype.startswith('list<') or coltype.startswith('vector<'):
+return ['[']
return [Hint('' % coltype)]
@@ -1065,7 +1067,7 @@ def assignment_udt_field_completer(ctxt, cass):
def _is_usertype(layout, curcol):
coltype = layout.columns[curcol].cql_type
-return coltype not in simple_cql_types and coltype not in ('map', 'set',
'list')
+return coltype not in simple_cql_types and coltype not in ('map', 'set',
'list', 'vector')
def _usertype_fields(ctxt, cass, layout, curcol):
diff --git a/pylib/cqlshlib/test/test_cqlsh_completion.py
b/pylib/cqlshlib/test/test_cqlsh_completion.py
index db59297a97..3fe46e878f 100644
--- a/pylib/cqlshlib/test/test_cqlsh_completion.py
+++ b/pylib/cqlshlib/test/test_cqlsh_completion.py
@@ -210,17 +210,17 @@ class TestCqlshCompletion(CqlshCompletionCase):
self.trycompletions(
'INSERT INTO twenty_rows_composite_table (a, b, c) VALUES (',
-[''],
+choices=[''],
split_completed_lines=False)
self.trycompletions(
"INSERT INTO twenty_rows_composite_table (a, b, c) VALUES ('",
-[''],
[cassandra] 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit e8c5fe31a28ec6e5aac9a87f3aad138d4e913494
Merge: 3d022d3c5f 5eba52feb8
Author: Andrés de la Peña
AuthorDate: Wed Oct 25 12:15:26 2023 +0100
Merge branch 'cassandra-5.0' into trunk
CHANGES.txt | 1 +
pylib/cqlshlib/cql3handling.py | 20 +---
pylib/cqlshlib/test/test_cqlsh_completion.py | 36
pylib/cqlshlib/test/test_cqlsh_output.py | 12 ++
pylib/cqlshlib/test/test_keyspace_init.cql | 30 ---
5 files changed, 73 insertions(+), 26 deletions(-)
diff --cc CHANGES.txt
index 763b8ecbbd,317954aafb..e3a8ff051b
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,8 -1,5 +1,9 @@@
-5.0-alpha2
+5.1
+ * Add ELAPSED command to cqlsh (CASSANDRA-18861)
+ * Add the ability to disable bulk loading of SSTables (CASSANDRA-18781)
+ * Clean up obsolete functions and simplify cql_version handling in cqlsh
(CASSANDRA-18787)
+Merged from 5.0:
+ * Add cqlsh autocompletion for the vector data type (CASSANDRA-18946)
* Fix nodetool tablehistograms output to avoid printing repeated information
and ensure at most two arguments (CASSANDRA-18955)
* Change the checksum algorithm SAI-related files use from CRC32 to CRC32C
(CASSANDRA-18836)
* Correctly remove Index.Group from IndexRegistry (CASSANDRA-18905)
diff --cc pylib/cqlshlib/cql3handling.py
index 220c197629,3c6518a0b9..d19febceaf
--- a/pylib/cqlshlib/cql3handling.py
+++ b/pylib/cqlshlib/cql3handling.py
@@@ -21,8 -21,10 +21,8 @@@ from cqlshlib.cqlhandling import CqlPar
simple_cql_types = {'ascii', 'bigint', 'blob', 'boolean', 'counter', 'date',
'decimal', 'double', 'duration', 'float',
'inet', 'int', 'smallint', 'text', 'time', 'timestamp',
'timeuuid', 'tinyint', 'uuid', 'varchar',
'varint'}
- simple_cql_types.difference_update(('set', 'map', 'list'))
+ simple_cql_types.difference_update(('set', 'map', 'list', 'vector'))
-cqldocs = helptopics.CQL3HelpTopics()
-
class UnexpectedTableStructure(UserWarning):
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (3d022d3c5f -> e8c5fe31a2)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from 3d022d3c5f Merge branch 'cassandra-5.0' into trunk new 5eba52feb8 Add cqlsh autocompletion for the vector data type new e8c5fe31a2 Merge branch 'cassandra-5.0' into trunk The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt | 1 + pylib/cqlshlib/cql3handling.py | 20 +--- pylib/cqlshlib/test/test_cqlsh_completion.py | 36 pylib/cqlshlib/test/test_cqlsh_output.py | 12 ++ pylib/cqlshlib/test/test_keyspace_init.cql | 30 --- 5 files changed, 73 insertions(+), 26 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (4b84c4332c -> 31729b8f92)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from 4b84c4332c Merge branch 'cassandra-5.0' into trunk new 3b9881bfa6 Fix vector type to support DDM's mask_default function new 31729b8f92 Merge branch 'cassandra-5.0' into trunk The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt | 1 + .../cassandra/examples/CQL/ddm_create_table_with_udf.cql | 2 +- doc/modules/cassandra/partials/masking_functions.adoc| 11 ++- src/java/org/apache/cassandra/cql3/CQL3Type.java | 16 src/java/org/apache/cassandra/db/marshal/VectorType.java | 13 + .../masking/ColumnMaskQueryWithDefaultTest.java | 9 - .../masking/ColumnMaskQueryWithReplaceTest.java | 7 +++ .../cassandra/cql3/functions/masking/ColumnMaskTest.java | 16 .../cql3/functions/masking/MaskingFunctionTester.java| 16 .../functions/masking/ReplaceMaskingFunctionTest.java| 5 +++-- .../apache/cassandra/db/marshal/AbstractTypeTest.java| 16 +++- 11 files changed, 106 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-5.0 updated: Fix vector type to support DDM's mask_default function
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cassandra-5.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-5.0 by this push:
new 3b9881bfa6 Fix vector type to support DDM's mask_default function
3b9881bfa6 is described below
commit 3b9881bfa65d2806e3918ba2c73198cb17c7071b
Author: Andrés de la Peña
AuthorDate: Wed Sep 27 21:56:15 2023 +0100
Fix vector type to support DDM's mask_default function
patch by Andrés de la Peña; reviewed by Berenguer Blasi and Maxwell Guo for
CASSANDRA-18889
---
CHANGES.txt | 1 +
.../cassandra/examples/CQL/ddm_create_table_with_udf.cql | 2 +-
doc/modules/cassandra/partials/masking_functions.adoc| 11 ++-
src/java/org/apache/cassandra/cql3/CQL3Type.java | 16
src/java/org/apache/cassandra/db/marshal/VectorType.java | 13 +
.../masking/ColumnMaskQueryWithDefaultTest.java | 9 -
.../masking/ColumnMaskQueryWithReplaceTest.java | 7 +++
.../cassandra/cql3/functions/masking/ColumnMaskTest.java | 16
.../cql3/functions/masking/MaskingFunctionTester.java| 16
.../functions/masking/ReplaceMaskingFunctionTest.java| 5 +++--
.../apache/cassandra/db/marshal/AbstractTypeTest.java| 16 +++-
11 files changed, 106 insertions(+), 6 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 66f613d26d..1e0fe54855 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
5.0-alpha2
+ * Fix vector type to support DDM's mask_default function (CASSANDRA-18889)
* Remove unnecessary reporter-config3 dependency (CASSANDRA-18907)
* Remove support for empty values on the vector data type (CASSANDRA-18876)
* Upgrade Dropwizard Metrics to 4.2.19 (CASSANDRA-14667)
diff --git a/doc/modules/cassandra/examples/CQL/ddm_create_table_with_udf.cql
b/doc/modules/cassandra/examples/CQL/ddm_create_table_with_udf.cql
index 23db48c1b5..85d6555f3a 100644
--- a/doc/modules/cassandra/examples/CQL/ddm_create_table_with_udf.cql
+++ b/doc/modules/cassandra/examples/CQL/ddm_create_table_with_udf.cql
@@ -2,7 +2,7 @@ CREATE FUNCTION redact(input text)
CALLED ON NULL INPUT
RETURNS text
LANGUAGE java
- AS 'return "redacted";
+ AS 'return "redacted";';
CREATE TABLE patients (
id timeuuid PRIMARY KEY,
diff --git a/doc/modules/cassandra/partials/masking_functions.adoc
b/doc/modules/cassandra/partials/masking_functions.adoc
index 43fb25c38d..a91f168067 100644
--- a/doc/modules/cassandra/partials/masking_functions.adoc
+++ b/doc/modules/cassandra/partials/masking_functions.adoc
@@ -10,7 +10,12 @@ Examples:
`mask_null(123)` -> `null`
-| `mask_default(value)` | Replaces its argument by an arbitrary, fixed default
value of the same type. This will be `\***\***` for text values, zero for
numeric values, `false` for booleans, etc.
+| `mask_default(value)` | Replaces its argument by an arbitrary, fixed default
value of the same type.
+This will be `\***\***` for text values, zero for numeric values, `false` for
booleans, etc.
+
+Variable-length multivalued types such as lists, sets and maps are masked as
empty collections.
+
+Fixed-length multivalued types such as tuples, UDTs and vectors are masked by
replacing each of their values by the default masking value of the value type.
Examples:
@@ -18,6 +23,10 @@ Examples:
`mask_default(123)` -> `0`
+`mask_default((list) [1, 2, 3])` -> `[]`
+
+`mask_default((vector) [1, 2, 3])` -> `[0, 0, 0]`
+
| `mask_replace(value, replacement])` | Replaces the first argument by the
replacement value on the second argument. The replacement value needs to have
the same type as the replaced value.
Examples:
diff --git a/src/java/org/apache/cassandra/cql3/CQL3Type.java
b/src/java/org/apache/cassandra/cql3/CQL3Type.java
index 074a8e515e..6b8b97877d 100644
--- a/src/java/org/apache/cassandra/cql3/CQL3Type.java
+++ b/src/java/org/apache/cassandra/cql3/CQL3Type.java
@@ -20,6 +20,7 @@ package org.apache.cassandra.cql3;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.List;
+import java.util.Objects;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -566,6 +567,21 @@ public interface CQL3Type
return sb.toString();
}
+@Override
+public boolean equals(Object o)
+{
+if (this == o) return true;
+if (o == null || getClass() != o.getClass()) return false;
+Vector vector = (Vector) o;
+return Objects.equals(type, vector.type);
+}
+
+@Override
+public int hashCode()
+{
+return Objects.hash(type);
+}
+
@Override
public String toString()
{
diff --git a/src/java/org/apache/cassandra/d
[cassandra] 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 31729b8f9241e842cdc93f9fbb7169b5aba2ca4e Merge: 4b84c4332c 3b9881bfa6 Author: Andrés de la Peña AuthorDate: Tue Oct 10 15:03:06 2023 +0100 Merge branch 'cassandra-5.0' into trunk CHANGES.txt | 1 + .../cassandra/examples/CQL/ddm_create_table_with_udf.cql | 2 +- doc/modules/cassandra/partials/masking_functions.adoc| 11 ++- src/java/org/apache/cassandra/cql3/CQL3Type.java | 16 src/java/org/apache/cassandra/db/marshal/VectorType.java | 13 + .../masking/ColumnMaskQueryWithDefaultTest.java | 9 - .../masking/ColumnMaskQueryWithReplaceTest.java | 7 +++ .../cassandra/cql3/functions/masking/ColumnMaskTest.java | 16 .../cql3/functions/masking/MaskingFunctionTester.java| 16 .../functions/masking/ReplaceMaskingFunctionTest.java| 5 +++-- .../apache/cassandra/db/marshal/AbstractTypeTest.java| 16 +++- 11 files changed, 106 insertions(+), 6 deletions(-) diff --cc CHANGES.txt index 03ebe157c0,1e0fe54855..1b377ce775 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,8 -1,5 +1,9 @@@ -5.0-alpha2 +5.1 + * Add ELAPSED command to cqlsh (CASSANDRA-18861) + * Add the ability to disable bulk loading of SSTables (CASSANDRA-18781) + * Clean up obsolete functions and simplify cql_version handling in cqlsh (CASSANDRA-18787) +Merged from 5.0: + * Fix vector type to support DDM's mask_default function (CASSANDRA-18889) * Remove unnecessary reporter-config3 dependency (CASSANDRA-18907) * Remove support for empty values on the vector data type (CASSANDRA-18876) * Upgrade Dropwizard Metrics to 4.2.19 (CASSANDRA-14667) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 90e0ef27e4b0ebb3611142ae40e03c58473b5cc5 Merge: a1de444331 6befb178aa Author: Andrés de la Peña AuthorDate: Wed Oct 4 17:33:15 2023 +0100 Merge branch 'cassandra-5.0' into trunk CHANGES.txt| 1 + src/java/org/apache/cassandra/cql3/Constants.java | 2 +- .../db/marshal/AbstractCompositeType.java | 6 +++ .../cassandra/db/marshal/AbstractTimeUUIDType.java | 7 .../apache/cassandra/db/marshal/AbstractType.java | 2 +- .../apache/cassandra/db/marshal/BooleanType.java | 7 .../org/apache/cassandra/db/marshal/ByteType.java | 6 --- .../org/apache/cassandra/db/marshal/BytesType.java | 6 +++ .../cassandra/db/marshal/CollectionType.java | 6 --- .../cassandra/db/marshal/CounterColumnType.java| 7 .../apache/cassandra/db/marshal/DecimalType.java | 7 .../apache/cassandra/db/marshal/DoubleType.java| 7 .../apache/cassandra/db/marshal/DurationType.java | 6 --- .../org/apache/cassandra/db/marshal/FloatType.java | 7 .../cassandra/db/marshal/InetAddressType.java | 7 .../org/apache/cassandra/db/marshal/Int32Type.java | 7 .../apache/cassandra/db/marshal/IntegerType.java | 7 .../cassandra/db/marshal/LexicalUUIDType.java | 7 .../org/apache/cassandra/db/marshal/LongType.java | 7 .../org/apache/cassandra/db/marshal/ShortType.java | 6 --- .../cassandra/db/marshal/SimpleDateType.java | 6 --- .../apache/cassandra/db/marshal/StringType.java| 6 +++ .../org/apache/cassandra/db/marshal/TimeType.java | 6 --- .../apache/cassandra/db/marshal/TimestampType.java | 6 +++ .../org/apache/cassandra/db/marshal/TupleType.java | 6 +++ .../org/apache/cassandra/db/marshal/UUIDType.java | 7 .../apache/cassandra/db/marshal/VectorType.java| 44 +++--- .../cql3/validation/operations/CQLVectorTest.java | 43 + 28 files changed, 190 insertions(+), 52 deletions(-) diff --cc CHANGES.txt index 1870ba0ac2,12c8fbc0e3..57e241c7cf --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,8 -1,5 +1,9 @@@ -5.0-alpha2 +5.1 + * Add ELAPSED command to cqlsh (CASSANDRA-18861) + * Add the ability to disable bulk loading of SSTables (CASSANDRA-18781) + * Clean up obsolete functions and simplify cql_version handling in cqlsh (CASSANDRA-18787) +Merged from 5.0: + * Remove support for empty values on the vector data type (CASSANDRA-18876) * Upgrade Dropwizard Metrics to 4.2.19 (CASSANDRA-14667) * Upgrade caffeine cache and fix CIDR permissions cache invalidation (CASSANDRA-18805) * Remove deprecated properties in CompressionParams (CASSANDRA-18742) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (a1de444331 -> 90e0ef27e4)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from a1de444331 Merge branch 'cassandra-5.0' into trunk new 6befb178aa Remove support for empty values on the vector data type new 90e0ef27e4 Merge branch 'cassandra-5.0' into trunk The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + src/java/org/apache/cassandra/cql3/Constants.java | 2 +- .../db/marshal/AbstractCompositeType.java | 6 +++ .../cassandra/db/marshal/AbstractTimeUUIDType.java | 7 .../apache/cassandra/db/marshal/AbstractType.java | 2 +- .../apache/cassandra/db/marshal/BooleanType.java | 7 .../org/apache/cassandra/db/marshal/ByteType.java | 6 --- .../org/apache/cassandra/db/marshal/BytesType.java | 6 +++ .../cassandra/db/marshal/CollectionType.java | 6 --- .../cassandra/db/marshal/CounterColumnType.java| 7 .../apache/cassandra/db/marshal/DecimalType.java | 7 .../apache/cassandra/db/marshal/DoubleType.java| 7 .../apache/cassandra/db/marshal/DurationType.java | 6 --- .../org/apache/cassandra/db/marshal/FloatType.java | 7 .../cassandra/db/marshal/InetAddressType.java | 7 .../org/apache/cassandra/db/marshal/Int32Type.java | 7 .../apache/cassandra/db/marshal/IntegerType.java | 7 .../cassandra/db/marshal/LexicalUUIDType.java | 7 .../org/apache/cassandra/db/marshal/LongType.java | 7 .../org/apache/cassandra/db/marshal/ShortType.java | 6 --- .../cassandra/db/marshal/SimpleDateType.java | 6 --- .../apache/cassandra/db/marshal/StringType.java| 6 +++ .../org/apache/cassandra/db/marshal/TimeType.java | 6 --- .../apache/cassandra/db/marshal/TimestampType.java | 6 +++ .../org/apache/cassandra/db/marshal/TupleType.java | 6 +++ .../org/apache/cassandra/db/marshal/UUIDType.java | 7 .../apache/cassandra/db/marshal/VectorType.java| 44 +++--- .../cql3/validation/operations/CQLVectorTest.java | 43 + 28 files changed, 190 insertions(+), 52 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-5.0 updated: Remove support for empty values on the vector data type
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cassandra-5.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-5.0 by this push:
new 6befb178aa Remove support for empty values on the vector data type
6befb178aa is described below
commit 6befb178aa0b7ad669f5c30e8b53fdd69a225a7a
Author: Andrés de la Peña
AuthorDate: Fri Sep 22 14:09:35 2023 +0100
Remove support for empty values on the vector data type
patch by Andrés de la Peña; reviewed by David Capwell and Maxwell Guo for
CASSANDRA-18876
---
CHANGES.txt| 1 +
src/java/org/apache/cassandra/cql3/Constants.java | 2 +-
.../db/marshal/AbstractCompositeType.java | 6 +++
.../cassandra/db/marshal/AbstractTimeUUIDType.java | 7
.../apache/cassandra/db/marshal/AbstractType.java | 2 +-
.../apache/cassandra/db/marshal/BooleanType.java | 7
.../org/apache/cassandra/db/marshal/ByteType.java | 6 ---
.../org/apache/cassandra/db/marshal/BytesType.java | 6 +++
.../cassandra/db/marshal/CollectionType.java | 6 ---
.../cassandra/db/marshal/CounterColumnType.java| 7
.../apache/cassandra/db/marshal/DecimalType.java | 7
.../apache/cassandra/db/marshal/DoubleType.java| 7
.../apache/cassandra/db/marshal/DurationType.java | 6 ---
.../org/apache/cassandra/db/marshal/FloatType.java | 7
.../cassandra/db/marshal/InetAddressType.java | 7
.../org/apache/cassandra/db/marshal/Int32Type.java | 7
.../apache/cassandra/db/marshal/IntegerType.java | 7
.../cassandra/db/marshal/LexicalUUIDType.java | 7
.../org/apache/cassandra/db/marshal/LongType.java | 7
.../org/apache/cassandra/db/marshal/ShortType.java | 6 ---
.../cassandra/db/marshal/SimpleDateType.java | 6 ---
.../apache/cassandra/db/marshal/StringType.java| 6 +++
.../org/apache/cassandra/db/marshal/TimeType.java | 6 ---
.../apache/cassandra/db/marshal/TimestampType.java | 6 +++
.../org/apache/cassandra/db/marshal/TupleType.java | 6 +++
.../org/apache/cassandra/db/marshal/UUIDType.java | 7
.../apache/cassandra/db/marshal/VectorType.java| 44 +++---
.../cql3/validation/operations/CQLVectorTest.java | 43 +
28 files changed, 190 insertions(+), 52 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index bca22c4580..12c8fbc0e3 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
5.0-alpha2
+ * Remove support for empty values on the vector data type (CASSANDRA-18876)
* Upgrade Dropwizard Metrics to 4.2.19 (CASSANDRA-14667)
* Upgrade caffeine cache and fix CIDR permissions cache invalidation
(CASSANDRA-18805)
* Remove deprecated properties in CompressionParams (CASSANDRA-18742)
diff --git a/src/java/org/apache/cassandra/cql3/Constants.java
b/src/java/org/apache/cassandra/cql3/Constants.java
index 03341d22bd..be81bed3b0 100644
--- a/src/java/org/apache/cassandra/cql3/Constants.java
+++ b/src/java/org/apache/cassandra/cql3/Constants.java
@@ -292,7 +292,7 @@ public abstract class Constants
public AssignmentTestable.TestResult testAssignment(String keyspace,
ColumnSpecification receiver)
{
CQL3Type receiverType = receiver.type.asCQL3Type();
-if (receiverType.isCollection() || receiverType.isUDT())
+if (receiverType.isCollection() || receiverType.isUDT() ||
receiverType.isVector())
return AssignmentTestable.TestResult.NOT_ASSIGNABLE;
if (!(receiverType instanceof CQL3Type.Native))
diff --git
a/src/java/org/apache/cassandra/db/marshal/AbstractCompositeType.java
b/src/java/org/apache/cassandra/db/marshal/AbstractCompositeType.java
index 350c124fa1..170363594f 100644
--- a/src/java/org/apache/cassandra/db/marshal/AbstractCompositeType.java
+++ b/src/java/org/apache/cassandra/db/marshal/AbstractCompositeType.java
@@ -41,6 +41,12 @@ public abstract class AbstractCompositeType extends
AbstractType
super(ComparisonType.CUSTOM);
}
+@Override
+public boolean allowsEmpty()
+{
+return true;
+}
+
public int compareCustom(VL left, ValueAccessor accessorL, VR
right, ValueAccessor accessorR)
{
if (accessorL.isEmpty(left) || accessorR.isEmpty(right))
diff --git a/src/java/org/apache/cassandra/db/marshal/AbstractTimeUUIDType.java
b/src/java/org/apache/cassandra/db/marshal/AbstractTimeUUIDType.java
index 1c4088ee97..35778aff24 100644
--- a/src/java/org/apache/cassandra/db/marshal/AbstractTimeUUIDType.java
+++ b/src/java/org/apache/cassandra/db/marshal/AbstractTimeUUIDType.java
@@ -42,6 +42,13 @@ public abstract class AbstractTimeUUIDType extends
TemporalType
super(ComparisonType.CUSTOM);
} // singleton
+@Override
+public boolean allowsEmpty()
+{
+return true
[cassandra] 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit a67f4904f7cfda90239d285333903c42ec1596b2 Merge: 2a5e1b77c9 c9b934df67 Author: Andrés de la Peña AuthorDate: Mon Sep 11 16:49:29 2023 +0100 Merge branch 'cassandra-5.0' into trunk .../upgrade/MixedModeAvailabilityTestBase.java | 181 - ...edModeAvailabilityUpgradedCoordinatorTest.java} | 11 +- ... MixedModeAvailabilityUpgradedReplicaTest.java} | 11 +- .../MixedModeAvailabilityV30OneAllTest.java| 32 4 files changed, 74 insertions(+), 161 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (2a5e1b77c9 -> a67f4904f7)
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
from 2a5e1b77c9 Merge branch 'cassandra-5.0' into trunk
new c9b934df67 Fix MixedModeAvailabilityTest
new a67f4904f7 Merge branch 'cassandra-5.0' into trunk
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../upgrade/MixedModeAvailabilityTestBase.java | 181 -
...edModeAvailabilityUpgradedCoordinatorTest.java} | 12 +-
... MixedModeAvailabilityUpgradedReplicaTest.java} | 12 +-
.../MixedModeAvailabilityV30AllOneTest.java| 32
.../MixedModeAvailabilityV30OneAllTest.java| 32
.../MixedModeAvailabilityV30QuorumQuorumTest.java | 32
6 files changed, 74 insertions(+), 227 deletions(-)
copy
test/distributed/org/apache/cassandra/distributed/upgrade/{MixedModeIndexV40Test.java
=> MixedModeAvailabilityUpgradedCoordinatorTest.java} (78%)
copy
test/distributed/org/apache/cassandra/distributed/upgrade/{MixedModeIndexV40Test.java
=> MixedModeAvailabilityUpgradedReplicaTest.java} (78%)
delete mode 100644
test/distributed/org/apache/cassandra/distributed/upgrade/MixedModeAvailabilityV30AllOneTest.java
delete mode 100644
test/distributed/org/apache/cassandra/distributed/upgrade/MixedModeAvailabilityV30OneAllTest.java
delete mode 100644
test/distributed/org/apache/cassandra/distributed/upgrade/MixedModeAvailabilityV30QuorumQuorumTest.java
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-5.0 updated: Fix MixedModeAvailabilityTest
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cassandra-5.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-5.0 by this push:
new c9b934df67 Fix MixedModeAvailabilityTest
c9b934df67 is described below
commit c9b934df67ed4ea0779e1f07cf762b67ba6362c5
Author: Andrés de la Peña
AuthorDate: Fri Aug 25 16:08:45 2023 +0100
Fix MixedModeAvailabilityTest
patch by Andrés de la Peña; reviewed by Berenguer Blasi for CASSANDRA-18564
---
.../upgrade/MixedModeAvailabilityTestBase.java | 181 -
...edModeAvailabilityUpgradedCoordinatorTest.java} | 11 +-
... MixedModeAvailabilityUpgradedReplicaTest.java} | 11 +-
.../MixedModeAvailabilityV30OneAllTest.java| 32
4 files changed, 74 insertions(+), 161 deletions(-)
diff --git
a/test/distributed/org/apache/cassandra/distributed/upgrade/MixedModeAvailabilityTestBase.java
b/test/distributed/org/apache/cassandra/distributed/upgrade/MixedModeAvailabilityTestBase.java
index 49f327b026..d26f6761d5 100644
---
a/test/distributed/org/apache/cassandra/distributed/upgrade/MixedModeAvailabilityTestBase.java
+++
b/test/distributed/org/apache/cassandra/distributed/upgrade/MixedModeAvailabilityTestBase.java
@@ -18,6 +18,8 @@
package org.apache.cassandra.distributed.upgrade;
+import java.util.EnumMap;
+import java.util.Map;
import java.util.UUID;
import java.util.concurrent.RejectedExecutionException;
@@ -25,22 +27,17 @@ import org.junit.Test;
import org.apache.cassandra.distributed.api.ConsistencyLevel;
import org.apache.cassandra.distributed.api.ICoordinator;
-import org.apache.cassandra.exceptions.ReadFailureException;
-import org.apache.cassandra.exceptions.ReadTimeoutException;
-import org.apache.cassandra.exceptions.WriteFailureException;
-import org.apache.cassandra.exceptions.WriteTimeoutException;
-import org.apache.cassandra.net.Verb;
-import org.assertj.core.api.Assertions;
import static java.lang.String.format;
-import static java.util.concurrent.TimeUnit.SECONDS;
+import static java.util.concurrent.TimeUnit.MINUTES;
import static org.apache.cassandra.distributed.api.ConsistencyLevel.ALL;
import static org.apache.cassandra.distributed.api.ConsistencyLevel.ONE;
import static org.apache.cassandra.distributed.api.ConsistencyLevel.QUORUM;
+import static org.apache.cassandra.distributed.api.Feature.GOSSIP;
+import static org.apache.cassandra.distributed.api.Feature.NATIVE_PROTOCOL;
+import static org.apache.cassandra.distributed.api.Feature.NETWORK;
import static org.apache.cassandra.distributed.shared.AssertUtils.assertRows;
import static org.apache.cassandra.distributed.shared.AssertUtils.row;
-import static org.apache.cassandra.net.Verb.READ_REQ;
-import static org.junit.Assert.assertFalse;
public abstract class MixedModeAvailabilityTestBase extends UpgradeTestBase
@@ -49,54 +46,32 @@ public abstract class MixedModeAvailabilityTestBase extends
UpgradeTestBase
private static final int COORDINATOR = 1;
private static final String INSERT = withKeyspace("INSERT INTO %s.t (k, c,
v) VALUES (?, ?, ?)");
private static final String SELECT = withKeyspace("SELECT * FROM %s.t
WHERE k = ?");
+private static final Map
CONSISTENCY_LEVELS = new EnumMap<>(ConsistencyLevel.class)
+{{
+put(ALL, ONE);
+put(ONE, ALL);
+put(QUORUM, QUORUM);
+}};
-private final ConsistencyLevel writeConsistencyLevel;
-private final ConsistencyLevel readConsistencyLevel;
-
-public MixedModeAvailabilityTestBase(ConsistencyLevel
writeConsistencyLevel, ConsistencyLevel readConsistencyLevel)
-{
-this.writeConsistencyLevel = writeConsistencyLevel;
-this.readConsistencyLevel = readConsistencyLevel;
-}
-
-@Test
-public void testAvailabilityCoordinatorNotUpgraded() throws Throwable
-{
-testAvailability(false, writeConsistencyLevel, readConsistencyLevel);
-}
+protected abstract boolean upgradedCoordinator();
@Test
-public void testAvailabilityCoordinatorUpgraded() throws Throwable
-{
-testAvailability(true, writeConsistencyLevel, readConsistencyLevel);
-}
-
-protected static void testAvailability(ConsistencyLevel
writeConsistencyLevel,
- ConsistencyLevel
readConsistencyLevel) throws Throwable
-{
-testAvailability(true, writeConsistencyLevel, readConsistencyLevel);
-testAvailability(false, writeConsistencyLevel, readConsistencyLevel);
-}
-
-private static void testAvailability(boolean upgradedCoordinator,
- ConsistencyLevel
writeConsistencyLevel,
- ConsistencyLevel
readConsistencyLevel) throws Throwable
+public void testAvailability() throws Throwable
{
new
[cassandra] branch cassandra-5.0 updated: SAI result retriever is filtering too many rows
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch cassandra-5.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/cassandra-5.0 by this push: new 9ce86e0ff8 SAI result retriever is filtering too many rows 9ce86e0ff8 is described below commit 9ce86e0ff8b6344b528a0640f9dafa23f97dd85a Author: Mike Adamson AuthorDate: Tue Aug 8 17:07:01 2023 +0100 SAI result retriever is filtering too many rows This patch fixes a bug in the SegmentMetadata that was only storing the partition key for min and max primary keys for a segment. It also contains some refactoring of the PrimaryKey to remove the deferred loading of PrimaryKeys by the PrimaryKeyMaps. Patch by Mike Adamson; reviewed by Caleb Rackliffe and Andrés de la Peña for CASSANDRA-18734 --- CHANGES.txt| 1 + .../apache/cassandra/index/sai/IndexContext.java | 9 +- .../cassandra/index/sai/StorageAttachedIndex.java | 1 + .../index/sai/StorageAttachedIndexGroup.java | 2 +- .../cassandra/index/sai/disk/PrimaryKeyMap.java| 3 +- .../index/sai/disk/StorageAttachedIndexWriter.java | 3 +- .../index/sai/disk/format/IndexDescriptor.java | 11 +- .../index/sai/disk/v1/MemtableIndexWriter.java | 14 +- .../index/sai/disk/v1/SkinnyPrimaryKeyMap.java | 40 +-- .../index/sai/disk/v1/WidePrimaryKeyMap.java | 37 +-- .../index/sai/disk/v1/keystore/KeyLookup.java | 6 +- .../index/sai/disk/v1/segment/SegmentMetadata.java | 33 +- .../index/sai/memory/TrieMemoryIndex.java | 3 +- .../cassandra/index/sai/plan/QueryController.java | 13 +- .../sai/plan/StorageAttachedIndexSearcher.java | 24 +- .../cassandra/index/sai/utils/PrimaryKey.java | 368 ++--- .../test/microbench/sai/KeyLookupBench.java| 4 +- .../org/apache/cassandra/index/sai/SAITester.java | 4 +- .../index/sai/disk/format/IndexDescriptorTest.java | 5 +- .../sai/disk/v1/InvertedIndexSearcherTest.java | 10 +- .../index/sai/disk/v1/SegmentFlushTest.java| 16 +- .../index/sai/disk/v1/WideRowPrimaryKeyTest.java | 3 +- .../v1/bbtree/BlockBalancedTreeIndexBuilder.java | 8 +- .../index/sai/disk/v1/keystore/KeyLookupTest.java | 28 +- .../sai/iterators/KeyRangeConcatIteratorTest.java | 6 +- .../index/sai/iterators/LongIterator.java | 2 +- .../AbstractInMemoryKeyRangeIteratorTester.java| 16 +- .../PriorityInMemoryKeyRangeIteratorTest.java | 4 +- .../index/sai/memory/TrieMemoryIndexTest.java | 1 + .../index/sai/utils/AbstractPrimaryKeyTester.java | 15 +- .../index/sai/utils/IndexInputLeakDetector.java| 2 +- .../cassandra/index/sai/utils/PrimaryKeyTest.java | 61 +++- 32 files changed, 449 insertions(+), 304 deletions(-) diff --git a/CHANGES.txt b/CHANGES.txt index 953ca7f3dc..c65a1e6671 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 5.0-alpha2 + * Fix SAI's SegmentMetadata min and max primary keys (CASSANDRA-18734) * Remove commons-codec dependency (CASSANDRA-18772) Merged from 4.1: Merged from 4.0: diff --git a/src/java/org/apache/cassandra/index/sai/IndexContext.java b/src/java/org/apache/cassandra/index/sai/IndexContext.java index ac33c837da..61eb844b02 100644 --- a/src/java/org/apache/cassandra/index/sai/IndexContext.java +++ b/src/java/org/apache/cassandra/index/sai/IndexContext.java @@ -44,6 +44,7 @@ import org.apache.cassandra.db.marshal.UTF8Type; import org.apache.cassandra.db.marshal.UUIDType; import org.apache.cassandra.db.rows.Cell; import org.apache.cassandra.db.rows.Row; +import org.apache.cassandra.dht.IPartitioner; import org.apache.cassandra.index.sai.analyzer.AbstractAnalyzer; import org.apache.cassandra.index.sai.disk.SSTableIndex; import org.apache.cassandra.index.sai.disk.format.Version; @@ -96,6 +97,7 @@ public class IndexContext public IndexContext(String keyspace, String table, AbstractType partitionKeyType, +IPartitioner partitioner, ClusteringComparator clusteringComparator, ColumnMetadata columnMetadata, IndexTarget.Type indexType, @@ -108,7 +110,7 @@ public class IndexContext this.columnMetadata = Objects.requireNonNull(columnMetadata); this.indexType = Objects.requireNonNull(indexType); this.validator = TypeUtil.cellValueType(columnMetadata, indexType); -this.primaryKeyFactory = new PrimaryKey.Factory(clusteringComparator); +this.primaryKeyFactory = new PrimaryKey.Factory(partitioner, clusteringComparator); this.indexMetadata = indexMetadata; this.memtableIndexManager = indexMetadata == null ? null : new MemtableIndexManager(this); @@ -122,6 +124,11 @@ public class
[cassandra] branch trunk updated (f8d5a1aa16 -> 1ba458c900)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from f8d5a1aa16 Merge branch 'cassandra-5.0' into trunk new 9ce86e0ff8 SAI result retriever is filtering too many rows new 1ba458c900 Merge branch 'cassandra-5.0' into trunk The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + .../apache/cassandra/index/sai/IndexContext.java | 9 +- .../cassandra/index/sai/StorageAttachedIndex.java | 1 + .../index/sai/StorageAttachedIndexGroup.java | 2 +- .../cassandra/index/sai/disk/PrimaryKeyMap.java| 3 +- .../index/sai/disk/StorageAttachedIndexWriter.java | 3 +- .../index/sai/disk/format/IndexDescriptor.java | 11 +- .../index/sai/disk/v1/MemtableIndexWriter.java | 14 +- .../index/sai/disk/v1/SkinnyPrimaryKeyMap.java | 40 +-- .../index/sai/disk/v1/WidePrimaryKeyMap.java | 37 +-- .../index/sai/disk/v1/keystore/KeyLookup.java | 6 +- .../index/sai/disk/v1/segment/SegmentMetadata.java | 33 +- .../index/sai/memory/TrieMemoryIndex.java | 3 +- .../cassandra/index/sai/plan/QueryController.java | 13 +- .../sai/plan/StorageAttachedIndexSearcher.java | 24 +- .../cassandra/index/sai/utils/PrimaryKey.java | 368 ++--- .../test/microbench/sai/KeyLookupBench.java| 4 +- .../org/apache/cassandra/index/sai/SAITester.java | 4 +- .../index/sai/disk/format/IndexDescriptorTest.java | 5 +- .../sai/disk/v1/InvertedIndexSearcherTest.java | 10 +- .../index/sai/disk/v1/SegmentFlushTest.java| 16 +- .../index/sai/disk/v1/WideRowPrimaryKeyTest.java | 3 +- .../v1/bbtree/BlockBalancedTreeIndexBuilder.java | 8 +- .../index/sai/disk/v1/keystore/KeyLookupTest.java | 28 +- .../sai/iterators/KeyRangeConcatIteratorTest.java | 6 +- .../index/sai/iterators/LongIterator.java | 2 +- .../AbstractInMemoryKeyRangeIteratorTester.java| 16 +- .../PriorityInMemoryKeyRangeIteratorTest.java | 4 +- .../index/sai/memory/TrieMemoryIndexTest.java | 1 + .../index/sai/utils/AbstractPrimaryKeyTester.java | 15 +- .../index/sai/utils/IndexInputLeakDetector.java| 2 +- .../cassandra/index/sai/utils/PrimaryKeyTest.java | 61 +++- 32 files changed, 449 insertions(+), 304 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 1ba458c900fd5f5c41e341a195402b9f7acb12ab Merge: f8d5a1aa16 9ce86e0ff8 Author: Andrés de la Peña AuthorDate: Thu Sep 7 11:52:02 2023 +0100 Merge branch 'cassandra-5.0' into trunk CHANGES.txt| 1 + .../apache/cassandra/index/sai/IndexContext.java | 9 +- .../cassandra/index/sai/StorageAttachedIndex.java | 1 + .../index/sai/StorageAttachedIndexGroup.java | 2 +- .../cassandra/index/sai/disk/PrimaryKeyMap.java| 3 +- .../index/sai/disk/StorageAttachedIndexWriter.java | 3 +- .../index/sai/disk/format/IndexDescriptor.java | 11 +- .../index/sai/disk/v1/MemtableIndexWriter.java | 14 +- .../index/sai/disk/v1/SkinnyPrimaryKeyMap.java | 40 +-- .../index/sai/disk/v1/WidePrimaryKeyMap.java | 37 +-- .../index/sai/disk/v1/keystore/KeyLookup.java | 6 +- .../index/sai/disk/v1/segment/SegmentMetadata.java | 33 +- .../index/sai/memory/TrieMemoryIndex.java | 3 +- .../cassandra/index/sai/plan/QueryController.java | 13 +- .../sai/plan/StorageAttachedIndexSearcher.java | 24 +- .../cassandra/index/sai/utils/PrimaryKey.java | 368 ++--- .../test/microbench/sai/KeyLookupBench.java| 4 +- .../org/apache/cassandra/index/sai/SAITester.java | 4 +- .../index/sai/disk/format/IndexDescriptorTest.java | 5 +- .../sai/disk/v1/InvertedIndexSearcherTest.java | 10 +- .../index/sai/disk/v1/SegmentFlushTest.java| 16 +- .../index/sai/disk/v1/WideRowPrimaryKeyTest.java | 3 +- .../v1/bbtree/BlockBalancedTreeIndexBuilder.java | 8 +- .../index/sai/disk/v1/keystore/KeyLookupTest.java | 28 +- .../sai/iterators/KeyRangeConcatIteratorTest.java | 6 +- .../index/sai/iterators/LongIterator.java | 2 +- .../AbstractInMemoryKeyRangeIteratorTester.java| 16 +- .../PriorityInMemoryKeyRangeIteratorTest.java | 4 +- .../index/sai/memory/TrieMemoryIndexTest.java | 1 + .../index/sai/utils/AbstractPrimaryKeyTester.java | 15 +- .../index/sai/utils/IndexInputLeakDetector.java| 2 +- .../cassandra/index/sai/utils/PrimaryKeyTest.java | 61 +++- 32 files changed, 449 insertions(+), 304 deletions(-) diff --cc CHANGES.txt index 26dab609c1,c65a1e6671..30cf57cba5 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,7 -1,13 +1,8 @@@ -5.0-alpha2 +5.1 + * Clean up obsolete functions and simplify cql_version handling in cqlsh (CASSANDRA-18787) +Merged from 5.0: + * Fix SAI's SegmentMetadata min and max primary keys (CASSANDRA-18734) * Remove commons-codec dependency (CASSANDRA-18772) -Merged from 4.1: -Merged from 4.0: -Merged from 3.11: -Merged from 3.0: - * Add cqlshrc.sample and credentials.sample into Debian package (CASSANDRA-18818) - -5.0-alpha1 * Fix rpm installations to work with jdk17 (CASSANDRA-18817) * Fix Depends and Build-Depends for Java for Debian packages (CASSANDRA-18809) * Update command line flags --add-exports and --add-opens for JDK17 (CASSANDRA-18439) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-5.0 updated: Add missing features to the page listing the 5.0 new features
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch cassandra-5.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/cassandra-5.0 by this push: new 637a427274 Add missing features to the page listing the 5.0 new features 637a427274 is described below commit 637a427274fa24ab137943693ce2808cdacb03bb Author: Andrés de la Peña AuthorDate: Thu Aug 24 11:06:31 2023 +0100 Add missing features to the page listing the 5.0 new features patch by Andrés de la Peña; reviewed by Stefan Miklosovic for CASSANDRA-18794 --- doc/modules/cassandra/pages/new/index.adoc | 13 +++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/doc/modules/cassandra/pages/new/index.adoc b/doc/modules/cassandra/pages/new/index.adoc index 38c932f384..25d66c562c 100644 --- a/doc/modules/cassandra/pages/new/index.adoc +++ b/doc/modules/cassandra/pages/new/index.adoc @@ -7,11 +7,20 @@ This section covers the new features in Apache Cassandra 5.0. * https://cwiki.apache.org/confluence/x/FQRACw[ACID Transactions (Accord)] * https://issues.apache.org/jira/browse/CASSANDRA-16052[Storage Attached Indexes] * https://issues.apache.org/jira/browse/CASSANDRA-17240[Trie Memtables] +* https://issues.apache.org/jira/browse/CASSANDRA-18398[Trie SSTables] * https://github.com/apache/cassandra/blob/trunk/NEWS.txt[More Guardrails] * https://issues.apache.org/jira/browse/CASSANDRA-8877[TTL and writetime on collections and UDTs] * https://cwiki.apache.org/confluence/x/YyD1D[Transactional Cluster Metadata] * https://issues.apache.org/jira/browse/CASSANDRA-16895[JDK 17] * https://issues.apache.org/jira/browse/CASSANDRA-17221[New Mathematical CQL functions: abs, exp, log, log10 and round] * https://issues.apache.org/jira/browse/CASSANDRA-18060[New CQL native scalar functions for collections] -** The new functions are mostly analogous to the existing aggregation functions, but they operate on the elements of collection columns. The new functions are `map_keys`, `map_values`, `collection_count`, `collection_min`, `collection_max`, `collection_sum` and `collection_avg`. -* https://issues.apache.org/jira/browse/CASSANDRA-14227[Extend maximum expiration date (TTL)] +* https://issues.apache.org/jira/browse/CASSANDRA-14227[Extend maximum expiration date (TTL)] +* https://issues.apache.org/jira/browse/CASSANDRA-17940[Dynamic Data Masking] +* https://issues.apache.org/jira/browse/CASSANDRA-18504[New vector data type] +* https://issues.apache.org/jira/browse/CASSANDRA-18640[New vector similarity functions] +* https://issues.apache.org/jira/browse/CASSANDRA-18397[Unified Compaction Strategy] +* https://issues.apache.org/jira/browse/CASSANDRA-8720[New sstablepartitions offline tool to find large partitions] +* https://issues.apache.org/jira/browse/CASSANDRA-18646[New snitch for Microsoft Azure] +* https://issues.apache.org/jira/browse/CASSANDRA-18592[New CIDR authorizer] +* https://issues.apache.org/jira/browse/CASSANDRA-18624[Pluggable crypto provider] +* https://issues.apache.org/jira/browse/CASSANDRA-17948[New virtual table to view system logs] - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (16dc2a2393 -> e9b9f9a35f)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from 16dc2a2393 Merge branch 'cassandra-5.0' into trunk new 637a427274 Add missing features to the page listing the 5.0 new features new e9b9f9a35f Merge branch 'cassandra-5.0' into trunk The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: doc/modules/cassandra/pages/new/index.adoc | 11 ++- 1 file changed, 10 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit e9b9f9a35f18e464cc89f22e223f0c3f4b83babb Merge: 16dc2a2393 637a427274 Author: Andrés de la Peña AuthorDate: Wed Aug 30 15:21:33 2023 +0100 Merge branch 'cassandra-5.0' into trunk doc/modules/cassandra/pages/new/index.adoc | 11 ++- 1 file changed, 10 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-5.0 updated: Fix spelling errors in the documentation for DDM
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch cassandra-5.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/cassandra-5.0 by this push: new 410018ab16 Fix spelling errors in the documentation for DDM 410018ab16 is described below commit 410018ab165b54c378648d52fb4ec815c557e80e Author: Andrés de la Peña AuthorDate: Wed Aug 23 10:54:11 2023 +0100 Fix spelling errors in the documentation for DDM patch by Andrés de la Peña; reviewed by Brandon Williams for CASSANDRA-18790 Co-authored-by: Andrés de la Peña Co-authored-by: Lorina Poland --- .../pages/developing/cql/dynamic_data_masking.adoc| 19 ++- doc/modules/cassandra/partials/masking_functions.adoc | 2 +- 2 files changed, 11 insertions(+), 10 deletions(-) diff --git a/doc/modules/cassandra/pages/developing/cql/dynamic_data_masking.adoc b/doc/modules/cassandra/pages/developing/cql/dynamic_data_masking.adoc index 8f523ac2ed..16599c9053 100644 --- a/doc/modules/cassandra/pages/developing/cql/dynamic_data_masking.adoc +++ b/doc/modules/cassandra/pages/developing/cql/dynamic_data_masking.adoc @@ -1,7 +1,7 @@ = Dynamic Data Masking Dynamic data masking (DDM) allows to obscure sensitive information while still allowing access to the masked columns. -DDM doesn't change the stored data. Instead, it just presents the data on their obscured form during `SELECT` queries. +DDM doesn't change the stored data. Instead, it just presents the data in its obscured form during `SELECT` queries. This aims to provide some degree of protection against accidental data exposure. However, it's important to know that anyone with direct access to the sstable files will be able to read the clear data. @@ -20,10 +20,10 @@ include::example$CQL/select_with_mask_functions.cql[] == Attaching masking functions to table columns -The masking functions can be permanently attached to the columns of a table. -In that case, `SELECT` queries will always return the column values in their masked form. -The masking will be transparent for the users running `SELECT` queries, -so their only way to know that a column is masked will be consulting the table definition. +The masking functions can be permanently attached to any column of a table. +If a masking columns is defined, `SELECT` queries will always return the column values in their masked form. +The masking will be transparent to the users running `SELECT` queries, +so their only way to know that a column is masked will be to consult the table definition. This is an optional feature that should be enabled with the `dynamic_data_masking_enabled` property in `cassandra.yaml`, since it's disabled by default. @@ -74,8 +74,9 @@ include::example$CQL/ddm_drop_mask.cql[] == Permissions The `UNMASK` permission allows users to retrieve the unmasked values of masked columns. -The masks will only be applied to the results of a `SELECT` query if the user doesn't have the `UNMASK` permission. -Ordinary users are created without the `UNMASK` permission, whereas superusers do have it. +Ordinary users are created without the `UNMASK` permission and will see masked values. +Superusers are created with the `UNMASK` permission, +and will be able to see the unmasked values in a SELECT query results. As an example, suppose that we have a table with masked columns: @@ -104,7 +105,7 @@ the users: include::example$CQL/ddm_create_users.cql[] -We can now see that the user with the `UNMASK` permission can see the clear data, without any masking: +We can now see that the user with the `UNMASK` permission can see the clear data, unmasked: [source,cql] @@ -130,7 +131,7 @@ Since it includes the `UNMASK` permission, that anonymous user will always see t In other words, attaching data masking functions to columns only makes sense if authentication is enabled. Users without the `UNMASK` permission are not allowed to use masked columns in the `WHERE` clause of a `SELECT` query. -This prevents malicious users from figuring out the clear data by running exhaustive queries. For instance: +This prevents malicious users from figuring out the clear data by running exhaustive queries. For example: [source,cql] diff --git a/doc/modules/cassandra/partials/masking_functions.adoc b/doc/modules/cassandra/partials/masking_functions.adoc index afadcd001e..43fb25c38d 100644 --- a/doc/modules/cassandra/partials/masking_functions.adoc +++ b/doc/modules/cassandra/partials/masking_functions.adoc @@ -2,7 +2,7 @@ |=== |Function | Description -| `mask_null(value)` | Replaces the first argument by a `null` column. The returned value is always an absent column, as it didn't exist, and not a not-null column representing a `null` value. +| `mask_null(value)` | Replaces the first argument with a `null` column. The returned value is always
[cassandra] 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit cbaef9094e83364e6812c65b8411ff7dbffaf9c6 Merge: 4226cbbec6 410018ab16 Author: Andrés de la Peña AuthorDate: Thu Aug 24 11:23:23 2023 +0100 Merge branch 'cassandra-5.0' into trunk .../pages/developing/cql/dynamic_data_masking.adoc| 19 ++- doc/modules/cassandra/partials/masking_functions.adoc | 2 +- 2 files changed, 11 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (4226cbbec6 -> cbaef9094e)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from 4226cbbec6 Merge branch 'cassandra-5.0' into trunk new 410018ab16 Fix spelling errors in the documentation for DDM new cbaef9094e Merge branch 'cassandra-5.0' into trunk The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../pages/developing/cql/dynamic_data_masking.adoc| 19 ++- doc/modules/cassandra/partials/masking_functions.adoc | 2 +- 2 files changed, 11 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-5.0 updated: Forbid SAI indexes with analysis options on primary key columns
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cassandra-5.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-5.0 by this push:
new b265b4658e Forbid SAI indexes with analysis options on primary key
columns
b265b4658e is described below
commit b265b4658e007b6943d543a11c609b7ba5fd979f
Author: Andrés de la Peña
AuthorDate: Wed Jul 19 17:40:50 2023 +0100
Forbid SAI indexes with analysis options on primary key columns
patch by Andrés de la Peña; reviewed by Caleb Rackliffe for CASSANDRA-18782
---
CHANGES.txt| 1 +
.../cassandra/index/sai/StorageAttachedIndex.java | 11 +++-
.../index/sai/analyzer/AbstractAnalyzer.java | 12 ++---
.../index/sai/analyzer/NonTokenizingOptions.java | 5
.../index/sai/cql/StorageAttachedIndexDDLTest.java | 31 ++
5 files changed, 56 insertions(+), 4 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 262a2edcb6..68515312fe 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
5.0-alpha2
+ * Forbid SAI indexes with analysis options on primary key columns
(CASSANDRA-18782)
Merged from 4.1:
Merged from 4.0:
Merged from 3.11:
diff --git a/src/java/org/apache/cassandra/index/sai/StorageAttachedIndex.java
b/src/java/org/apache/cassandra/index/sai/StorageAttachedIndex.java
index e8c4b53f4e..805a6b65e1 100644
--- a/src/java/org/apache/cassandra/index/sai/StorageAttachedIndex.java
+++ b/src/java/org/apache/cassandra/index/sai/StorageAttachedIndex.java
@@ -48,6 +48,7 @@ import org.slf4j.LoggerFactory;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.cql3.CQL3Type;
+import org.apache.cassandra.cql3.CqlBuilder;
import org.apache.cassandra.cql3.Operator;
import org.apache.cassandra.cql3.statements.schema.IndexTarget;
import org.apache.cassandra.db.CassandraWriteContext;
@@ -96,6 +97,9 @@ import org.apache.cassandra.utils.concurrent.OpOrder;
public class StorageAttachedIndex implements Index
{
public static final String NAME = "sai";
+
+@VisibleForTesting
+public static final String ANALYSIS_ON_KEY_COLUMNS_MESSAGE = "Analysis
options are not supported on primary key columns, but found ";
private static final Logger logger =
LoggerFactory.getLogger(StorageAttachedIndex.class);
@@ -253,7 +257,12 @@ public class StorageAttachedIndex implements Index
throw new InvalidRequestException("Unsupported type: " +
type.asCQL3Type());
}
-AbstractAnalyzer.fromOptions(type, options);
+Map analysisOptions =
AbstractAnalyzer.getAnalyzerOptions(options);
+if (target.left.isPrimaryKeyColumn() && !analysisOptions.isEmpty())
+{
+throw new InvalidRequestException(ANALYSIS_ON_KEY_COLUMNS_MESSAGE
+ new CqlBuilder().append(analysisOptions));
+}
+AbstractAnalyzer.fromOptions(type, analysisOptions);
return Collections.emptyMap();
}
diff --git
a/src/java/org/apache/cassandra/index/sai/analyzer/AbstractAnalyzer.java
b/src/java/org/apache/cassandra/index/sai/analyzer/AbstractAnalyzer.java
index 1f5e339657..da0b2a4d19 100644
--- a/src/java/org/apache/cassandra/index/sai/analyzer/AbstractAnalyzer.java
+++ b/src/java/org/apache/cassandra/index/sai/analyzer/AbstractAnalyzer.java
@@ -22,6 +22,7 @@ import java.nio.ByteBuffer;
import java.util.Iterator;
import java.util.Map;
import java.util.NoSuchElementException;
+import java.util.stream.Collectors;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.db.marshal.StringType;
@@ -103,9 +104,14 @@ public abstract class AbstractAnalyzer implements
Iterator
private static boolean hasNonTokenizingOptions(Map options)
{
-return options.get(NonTokenizingOptions.ASCII) != null ||
- options.containsKey(NonTokenizingOptions.CASE_SENSITIVE) ||
- options.containsKey(NonTokenizingOptions.NORMALIZE);
+return
options.keySet().stream().anyMatch(NonTokenizingOptions::hasOption);
+}
+
+public static Map getAnalyzerOptions(Map
options)
+{
+return options.entrySet().stream()
+ .filter(e -> NonTokenizingOptions.hasOption(e.getKey()))
+ .collect(Collectors.toMap(Map.Entry::getKey,
Map.Entry::getValue));
}
}
diff --git
a/src/java/org/apache/cassandra/index/sai/analyzer/NonTokenizingOptions.java
b/src/java/org/apache/cassandra/index/sai/analyzer/NonTokenizingOptions.java
index ab6485acf5..66c7740a53 100644
--- a/src/java/org/apache/cassandra/index/sai/analyzer/NonTokenizingOptions.java
+++ b/src/java/org/apache/cassandra/index/sai/analyzer/NonTokenizingOptions.java
@@ -65,6 +65,11 @@ public class NonTokenizingOptions
this.normalized = normalized;
}
+
[cassandra] branch trunk updated (77cf2185b6 -> 4226cbbec6)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from 77cf2185b6 Add jmh microbenchmarks to eclipse IDE new b265b4658e Forbid SAI indexes with analysis options on primary key columns new 4226cbbec6 Merge branch 'cassandra-5.0' into trunk The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + .../cassandra/index/sai/StorageAttachedIndex.java | 11 +++- .../index/sai/analyzer/AbstractAnalyzer.java | 12 ++--- .../index/sai/analyzer/NonTokenizingOptions.java | 5 .../index/sai/cql/StorageAttachedIndexDDLTest.java | 31 ++ 5 files changed, 56 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 4226cbbec6c2ac92ed8d3834140f8ea3e3260e59 Merge: 77cf2185b6 b265b4658e Author: Andrés de la Peña AuthorDate: Thu Aug 24 10:14:40 2023 +0100 Merge branch 'cassandra-5.0' into trunk CHANGES.txt| 1 + .../cassandra/index/sai/StorageAttachedIndex.java | 11 +++- .../index/sai/analyzer/AbstractAnalyzer.java | 12 ++--- .../index/sai/analyzer/NonTokenizingOptions.java | 5 .../index/sai/cql/StorageAttachedIndexDDLTest.java | 31 ++ 5 files changed, 56 insertions(+), 4 deletions(-) diff --cc CHANGES.txt index cfc0a9aace,68515312fe..a412b7a8d9 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,5 -1,12 +1,6 @@@ -5.0-alpha2 +5.1 +Merged from 5.0: + * Forbid SAI indexes with analysis options on primary key columns (CASSANDRA-18782) -Merged from 4.1: -Merged from 4.0: -Merged from 3.11: -Merged from 3.0: - * Make alternation of a user type validate the same way as creation of a user type does (CASSANDRA-18585) - -5.0-alpha1 * Expose bootstrap failure state via JMX and nodetool info (CASSANDRA-18749) * Reduce size of per-SSTable index components for SAI (CASSANDRA-18673) * Remove unnecessary Netty dependencies after upgrade to version 4.1.96 (CASSANDRA-18729) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit d6534e5c4fb1e8dff0c0848a53f07ebfc391b9b3 Merge: f8baabd95a 5817799605 Author: Andrés de la Peña AuthorDate: Mon Aug 21 15:17:24 2023 +0100 Merge branch 'cassandra-5.0' into trunk .../pages/developing/cql/dynamic_data_masking.adoc | 178 + .../cassandra/pages/developing/cql/index.adoc | 2 +- 2 files changed, 179 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (f8baabd95a -> d6534e5c4f)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from f8baabd95a Merge branch 'cassandra-5.0' into trunk new 5817799605 Restore accidentally deleted DDM doc new d6534e5c4f Merge branch 'cassandra-5.0' into trunk The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../pages/developing/cql/dynamic_data_masking.adoc | 178 + .../cassandra/pages/developing/cql/index.adoc | 2 +- 2 files changed, 179 insertions(+), 1 deletion(-) create mode 100644 doc/modules/cassandra/pages/developing/cql/dynamic_data_masking.adoc - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-5.0 updated: Restore accidentally deleted DDM doc
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch cassandra-5.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/cassandra-5.0 by this push: new 5817799605 Restore accidentally deleted DDM doc 5817799605 is described below commit 58177996058a57e4909b11bcc6e754c8a6e38f6d Author: Andrés de la Peña AuthorDate: Fri Aug 18 11:01:45 2023 +0100 Restore accidentally deleted DDM doc patch by Andrés de la Peña; reviewed by Brandon Williams and Berenguer Blasi for CASSANDRA-18776 --- .../pages/developing/cql/dynamic_data_masking.adoc | 178 + .../cassandra/pages/developing/cql/index.adoc | 2 +- 2 files changed, 179 insertions(+), 1 deletion(-) diff --git a/doc/modules/cassandra/pages/developing/cql/dynamic_data_masking.adoc b/doc/modules/cassandra/pages/developing/cql/dynamic_data_masking.adoc new file mode 100644 index 00..8f523ac2ed --- /dev/null +++ b/doc/modules/cassandra/pages/developing/cql/dynamic_data_masking.adoc @@ -0,0 +1,178 @@ += Dynamic Data Masking + +Dynamic data masking (DDM) allows to obscure sensitive information while still allowing access to the masked columns. +DDM doesn't change the stored data. Instead, it just presents the data on their obscured form during `SELECT` queries. +This aims to provide some degree of protection against accidental data exposure. However, it's important to know that +anyone with direct access to the sstable files will be able to read the clear data. + +== Masking functions + +DDM is based on a set of CQL native functions that obscure sensitive information. The available functions are: + +include::partial$masking_functions.adoc[] + +Those functions can be discretionarily used on `SELECT` queries to get an obscured view of the data. For example: + +[source,cql] + +include::example$CQL/select_with_mask_functions.cql[] + + +== Attaching masking functions to table columns + +The masking functions can be permanently attached to the columns of a table. +In that case, `SELECT` queries will always return the column values in their masked form. +The masking will be transparent for the users running `SELECT` queries, +so their only way to know that a column is masked will be consulting the table definition. + +This is an optional feature that should be enabled with the `dynamic_data_masking_enabled` property in `cassandra.yaml`, +since it's disabled by default. + +The masks of the columns of a table can be defined on `CREATE TABLE` queries: + +[source,cql] + +include::example$CQL/ddm_create_table.cql[] + + +Note that in the example above we are referencing the `mask_inner` function with two arguments. +However, that CQL function actually has three arguments when explicitely used on `SELECT` queries. +The first argument is always ommitted when attaching the function to a schema column. +The value of that first argument is always interpreted as the value of the masked column, in this case a `text` column. +For the same reason the call to `mask_default` attached to the column doesn't have any argument, +even when that function requires one argument when explicitely used on `SELECT` queries. + +Data can be inserted into the masked table as usual. For example: + +[source,cql] + +include::example$CQL/ddm_insert_data.cql[] + + +The attached column masks will make `SELECT` queries automatically return masked data, +without the need of including the masking function on the query: + +[source,cql] + +include::example$CQL/ddm_select_with_masked_columns.cql[] + + +The masking function attached to a column can be changed with an `ALTER TABLE` query: + +[source,cql] + +include::example$CQL/ddm_alter_mask.cql[] + + +In a similar way, a masking function can be dettached from a column with an `ALTER TABLE` query: + +[source,cql] + +include::example$CQL/ddm_drop_mask.cql[] + + +== Permissions + +The `UNMASK` permission allows users to retrieve the unmasked values of masked columns. +The masks will only be applied to the results of a `SELECT` query if the user doesn't have the `UNMASK` permission. +Ordinary users are created without the `UNMASK` permission, whereas superusers do have it. + +As an example, suppose that we have a table with masked columns: + +[source,cql] + +include::example$CQL/ddm_create_table.cql[] + + +And we insert some data into the table: + +[source,cql] + +include::example$CQL/ddm_insert_data.cql[] + + +[source,cql] + +include::example$CQL/ddm_select_without_unmask_permission.cql[] + + +Then we create two users with `SELECT` permission for the table, but we only grant the `UNMASK` permission to one of +the users: + +[source,cql] + +include::example$CQL/ddm_create_users.cql[] + + +We can now see that the user with the `UNMASK` permission can see the clear data, without any masking: + +[source
[cassandra] branch trunk updated (ab40d85cd1 -> de0a556c5d)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from ab40d85cd1 Merge branch 'cassandra-5.0' into trunk new 7482d88f48 Add guardrail for vector dimensions new de0a556c5d Merge branch 'cassandra-5.0' into trunk The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 13 +- NEWS.txt | 1 + conf/cassandra.yaml| 5 + src/java/org/apache/cassandra/config/Config.java | 2 + .../apache/cassandra/config/GuardrailsOptions.java | 26 +++ src/java/org/apache/cassandra/cql3/CQL3Type.java | 53 ++ .../statements/schema/AlterTableStatement.java | 7 + .../cql3/statements/schema/AlterTypeStatement.java | 1 + .../statements/schema/CreateTableStatement.java| 15 +- .../statements/schema/CreateTypeStatement.java | 5 + .../apache/cassandra/db/guardrails/Guardrails.java | 30 .../cassandra/db/guardrails/GuardrailsConfig.java | 10 ++ .../cassandra/db/guardrails/GuardrailsMBean.java | 16 ++ .../guardrails/GuardrailVectorDimensionsTest.java | 184 + 14 files changed, 360 insertions(+), 8 deletions(-) create mode 100644 test/unit/org/apache/cassandra/db/guardrails/GuardrailVectorDimensionsTest.java - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-5.0 updated: Add guardrail for vector dimensions
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch cassandra-5.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/cassandra-5.0 by this push: new 7482d88f48 Add guardrail for vector dimensions 7482d88f48 is described below commit 7482d88f48662480c6879b4e500bcdf93c78e5e0 Author: Andrés de la Peña AuthorDate: Mon Aug 7 12:46:08 2023 +0100 Add guardrail for vector dimensions patch by Andrés de la Peña; reviewed by Brandon Williams and Maxwell Guo for CASSANDRA-18730 --- CHANGES.txt| 1 + NEWS.txt | 1 + conf/cassandra.yaml| 5 + src/java/org/apache/cassandra/config/Config.java | 2 + .../apache/cassandra/config/GuardrailsOptions.java | 26 +++ src/java/org/apache/cassandra/cql3/CQL3Type.java | 53 ++ .../statements/schema/AlterTableStatement.java | 7 + .../cql3/statements/schema/AlterTypeStatement.java | 1 + .../statements/schema/CreateTableStatement.java| 15 +- .../statements/schema/CreateTypeStatement.java | 5 + .../apache/cassandra/db/guardrails/Guardrails.java | 30 .../cassandra/db/guardrails/GuardrailsConfig.java | 10 ++ .../cassandra/db/guardrails/GuardrailsMBean.java | 16 ++ .../guardrails/GuardrailVectorDimensionsTest.java | 184 + 14 files changed, 350 insertions(+), 6 deletions(-) diff --git a/CHANGES.txt b/CHANGES.txt index 0de34d2bf4..7528daf808 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 5.0 + * Add guardrail for vector dimensions (CASSANDRA-18730) * Upgraded Netty to 4.1.96, added BouncyCastle dependency (CASSANDRA-17992) * Fix for (unsupported) big endian unaligned architecture, eg s390x (CASSANDRA-17723) * CIDR filtering authorizer (CASSANDRA-18592) diff --git a/NEWS.txt b/NEWS.txt index 9b8441f857..788e4cadef 100644 --- a/NEWS.txt +++ b/NEWS.txt @@ -116,6 +116,7 @@ New features - Column value size - Partition size - Partition tombstones + - Vector dimensions - It is possible to list ephemeral snapshots by nodetool listsnaphots command when flag "-e" is specified. - Added a new flag to `nodetool profileload` and JMX endpoint to set up recurring profile load generation on specified intervals (see CASSANDRA-17821) diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml index 26d3579f03..4d6d4f1d42 100644 --- a/conf/cassandra.yaml +++ b/conf/cassandra.yaml @@ -1989,6 +1989,11 @@ drop_compact_storage_enabled: false # fields_per_udt_warn_threshold: -1 # fields_per_udt_fail_threshold: -1 # +# Guardrail to warn or fail when creating a vector column with more dimensions than threshold. +# Default -1 to disable. +# vector_dimensions_warn_threshold: -1 +# vector_dimensions_fail_threshold: -1 +# # Guardrail to indicate whether or not users are allowed to use ALTER TABLE commands to make column changes to tables # alter_table_enabled: true # diff --git a/src/java/org/apache/cassandra/config/Config.java b/src/java/org/apache/cassandra/config/Config.java index 831bf93cfc..2f6408f3fd 100644 --- a/src/java/org/apache/cassandra/config/Config.java +++ b/src/java/org/apache/cassandra/config/Config.java @@ -892,6 +892,8 @@ public class Config public volatile int items_per_collection_fail_threshold = -1; public volatile int fields_per_udt_warn_threshold = -1; public volatile int fields_per_udt_fail_threshold = -1; +public volatile int vector_dimensions_warn_threshold = -1; +public volatile int vector_dimensions_fail_threshold = -1; public volatile int data_disk_usage_percentage_warn_threshold = -1; public volatile int data_disk_usage_percentage_fail_threshold = -1; public volatile DataStorageSpec.LongBytesBound data_disk_usage_max_disk_size = null; diff --git a/src/java/org/apache/cassandra/config/GuardrailsOptions.java b/src/java/org/apache/cassandra/config/GuardrailsOptions.java index f58408e8b2..b0cb259c26 100644 --- a/src/java/org/apache/cassandra/config/GuardrailsOptions.java +++ b/src/java/org/apache/cassandra/config/GuardrailsOptions.java @@ -82,6 +82,7 @@ public class GuardrailsOptions implements GuardrailsConfig validateSizeThreshold(config.collection_size_warn_threshold, config.collection_size_fail_threshold, false, "collection_size"); validateMaxIntThreshold(config.items_per_collection_warn_threshold, config.items_per_collection_fail_threshold, "items_per_collection"); validateMaxIntThreshold(config.fields_per_udt_warn_threshold, config.fields_per_udt_fail_threshold, "fields_per_udt"); +validateMaxIntThreshold(config.vector_dimensions_warn_threshold, config.vector_dimensions_fail_threshold, "vector_dimensions"); validatePercentageThreshol
[cassandra] 01/01: Merge branch 'cassandra-5.0' into trunk
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit de0a556c5d354827381a9bef9469db061724b82d Merge: ab40d85cd1 7482d88f48 Author: Andrés de la Peña AuthorDate: Wed Aug 9 11:32:26 2023 +0100 Merge branch 'cassandra-5.0' into trunk CHANGES.txt| 13 +- NEWS.txt | 1 + conf/cassandra.yaml| 5 + src/java/org/apache/cassandra/config/Config.java | 2 + .../apache/cassandra/config/GuardrailsOptions.java | 26 +++ src/java/org/apache/cassandra/cql3/CQL3Type.java | 53 ++ .../statements/schema/AlterTableStatement.java | 7 + .../cql3/statements/schema/AlterTypeStatement.java | 1 + .../statements/schema/CreateTableStatement.java| 15 +- .../statements/schema/CreateTypeStatement.java | 5 + .../apache/cassandra/db/guardrails/Guardrails.java | 30 .../cassandra/db/guardrails/GuardrailsConfig.java | 10 ++ .../cassandra/db/guardrails/GuardrailsMBean.java | 16 ++ .../guardrails/GuardrailVectorDimensionsTest.java | 184 + 14 files changed, 360 insertions(+), 8 deletions(-) diff --cc CHANGES.txt index 0de34d2bf4,7528daf808..bda248627e --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,4 -1,5 +1,15 @@@ -5.0 ++5.1 ++Merged from 5.0: + * Add guardrail for vector dimensions (CASSANDRA-18730) ++Merged from 4.1: ++Merged from 4.0: ++Merged from 3.11: ++Merged from 3.0: ++ * Fix CQLSH online help topic link (CASSANDRA-17534) ++ * Remove unused suppressions (CASSANDRA-18724) ++ ++ +5.0 * Upgraded Netty to 4.1.96, added BouncyCastle dependency (CASSANDRA-17992) * Fix for (unsupported) big endian unaligned architecture, eg s390x (CASSANDRA-17723) * CIDR filtering authorizer (CASSANDRA-18592) @@@ -180,8 -181,8 +191,6 @@@ Merged from 4.0 * Partial compaction can resurrect deleted data (CASSANDRA-18507) Merged from 3.11: Merged from 3.0: -- * Fix CQLSH online help topic link (CASSANDRA-17534) -- * Remove unused suppressions (CASSANDRA-18724) 4.1.4 - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Remove 3.x from the versions checked for prepared statement behaviour
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new 98680e8708 Remove 3.x from the versions checked for prepared statement
behaviour
98680e8708 is described below
commit 98680e8708e36e5120b08e1b8ad9f7622d3bd74b
Author: Andrés de la Peña
AuthorDate: Wed Jul 26 12:49:28 2023 +0100
Remove 3.x from the versions checked for prepared statement behaviour
patch by Andrés de la Peña; reviewed by Mick Semb Wever for CASSANDRA-18695
---
CHANGES.txt| 1 +
src/java/org/apache/cassandra/cql3/QueryProcessor.java | 17 ++---
2 files changed, 7 insertions(+), 11 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 3f78b9769e..7a60b6ae95 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
5.0
+ * Remove 3.x from the versions checked for prepared statement behaviour
(CASSANDRA-18695)
* Add vector similarity functions (CASSANDRA-18640)
* Lift MessagingService.minimum_version to 40 (CASSANDRA-18314)
* Introduce pluggable crypto providers and default to
AmazonCorrettoCryptoProvider (CASSANDRA-18624)
diff --git a/src/java/org/apache/cassandra/cql3/QueryProcessor.java
b/src/java/org/apache/cassandra/cql3/QueryProcessor.java
index 479e35fe6c..02369f373a 100644
--- a/src/java/org/apache/cassandra/cql3/QueryProcessor.java
+++ b/src/java/org/apache/cassandra/cql3/QueryProcessor.java
@@ -82,8 +82,6 @@ public class QueryProcessor implements QueryHandler
public static final CassandraVersion CQL_VERSION = new
CassandraVersion("3.4.7");
// See comments on QueryProcessor #prepare
-public static final CassandraVersion
NEW_PREPARED_STATEMENT_BEHAVIOUR_SINCE_30 = new CassandraVersion("3.0.26");
-public static final CassandraVersion
NEW_PREPARED_STATEMENT_BEHAVIOUR_SINCE_3X = new CassandraVersion("3.11.12");
public static final CassandraVersion
NEW_PREPARED_STATEMENT_BEHAVIOUR_SINCE_40 = new CassandraVersion("4.0.2");
public static final QueryProcessor instance = new QueryProcessor();
@@ -644,10 +642,7 @@ public class QueryProcessor implements QueryHandler
synchronized (this)
{
CassandraVersion minVersion =
Gossiper.instance.getMinVersion(DatabaseDescriptor.getWriteRpcTimeout(TimeUnit.MILLISECONDS),
TimeUnit.MILLISECONDS);
-if (minVersion != null &&
-((minVersion.major == 3 && minVersion.minor == 0 &&
minVersion.compareTo(NEW_PREPARED_STATEMENT_BEHAVIOUR_SINCE_30) >= 0) ||
- (minVersion.major == 3 && minVersion.minor > 0 &&
minVersion.compareTo(NEW_PREPARED_STATEMENT_BEHAVIOUR_SINCE_3X) >= 0) ||
-
(minVersion.compareTo(NEW_PREPARED_STATEMENT_BEHAVIOUR_SINCE_40, true) >= 0)))
+if (minVersion != null &&
minVersion.compareTo(NEW_PREPARED_STATEMENT_BEHAVIOUR_SINCE_40, true) >= 0)
{
logger.info("Fully upgraded to at least {}", minVersion);
newPreparedStatementBehaviour = true;
@@ -661,17 +656,17 @@ public class QueryProcessor implements QueryHandler
* This method got slightly out of hand, but this is with best intentions:
to allow users to be upgraded from any
* prior version, and help implementers avoid previous mistakes by clearly
separating fully qualified and non-fully
* qualified statement behaviour.
- *
+ *
* Basically we need to handle 4 different hashes here;
* 1. fully qualified query with keyspace
* 2. fully qualified query without keyspace
* 3. unqualified query with keyspace
* 4. unqualified query without keyspace
- *
- * The correct combination to return is 2/3 - the problem is during
upgrades (assuming upgrading from < 3.0.26)
+ *
+ * The correct combination to return is 2/3 - the problem is during
upgrades (assuming upgrading from < 4.0.2)
* - Existing clients have hash 1 or 3
- * - Query prepared on a 3.0.26/3.11.12/4.0.2 instance needs to return
hash 1/3 to be able to execute it on a 3.0.25 instance
- * - This is handled by the useNewPreparedStatementBehaviour flag - while
there still are 3.0.25 instances in
+ * - Query prepared on a post-4.0.2 instance needs to return hash 1/3 to
be able to execute it on a pre-4.0.2 instance
+ * - This is handled by the useNewPreparedStatementBehaviour flag - while
there still are pre-4.0.2 instances in
* the cluster we always return hash 1/3
* - Once fully upgraded we start returning hash 2/3, this will cause a
prepared statement id mismatch for existing
* clients, but they will be able to continue using the old prepared
statement id after that exception si
[cassandra] branch trunk updated: Add vector similarity functions
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/trunk by this push: new 5a82c04fd3 Add vector similarity functions 5a82c04fd3 is described below commit 5a82c04fd363030d712fb2935b6c591577ba25ec Author: Andrés de la Peña AuthorDate: Wed Jun 28 16:43:58 2023 +0100 Add vector similarity functions patch by Andrés de la Peña; reviewed by Ekaterina Dimitrova and Maxwell Guo for CASSANDRA-18640 Co-authored-by: Jonathan Ellis Co-authored-by: Andrés de la Peña --- CHANGES.txt| 1 + NEWS.txt | 1 + .../cassandra/pages/developing/cql/changes.adoc| 1 + .../cassandra/pages/developing/cql/functions.adoc | 7 + .../cassandra/partials/vector_functions.adoc | 41 + src/java/org/apache/cassandra/cql3/CQL3Type.java | 14 +- .../cassandra/cql3/functions/FunctionFactory.java | 14 +- .../cql3/functions/FunctionParameter.java | 86 -- .../cassandra/cql3/functions/NativeFunctions.java | 1 + .../cassandra/cql3/functions/VectorFcts.java | 105 .../functions/masking/ReplaceMaskingFunction.java | 4 +- .../cassandra/cql3/functions/types/VectorType.java | 2 +- .../cassandra/cql3/selection/Selectable.java | 25 ++- .../apache/cassandra/db/marshal/VectorType.java| 2 +- .../cassandra/cql3/functions/VectorFctsTest.java | 179 + .../cql3/validation/operations/CQLVectorTest.java | 4 +- .../cassandra/utils/AbstractTypeGenerators.java| 6 +- 17 files changed, 453 insertions(+), 40 deletions(-) diff --git a/CHANGES.txt b/CHANGES.txt index 12691ced64..3f78b9769e 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 5.0 + * Add vector similarity functions (CASSANDRA-18640) * Lift MessagingService.minimum_version to 40 (CASSANDRA-18314) * Introduce pluggable crypto providers and default to AmazonCorrettoCryptoProvider (CASSANDRA-18624) * Improved DeletionTime serialization (CASSANDRA-18648) diff --git a/NEWS.txt b/NEWS.txt index 1c5be180e8..378ae6e93c 100644 --- a/NEWS.txt +++ b/NEWS.txt @@ -84,6 +84,7 @@ New features class. Further details and documentation can be found in src/java/org/apache/cassandra/db/compaction/UnifiedCompactionStrategy.md - New `VectorType` (cql `vector`) which adds new fixed-length element arrays. See CASSANDRA-18504 +- Added new vector similarity functions `similarity_cosine`, `similarity_euclidean` and `similarity_dot_product`. - Removed UDT type migration logic for 3.6+ clusters upgrading to 4.0. If migration has been disabled, it must be enabled before upgrading to 5.0 if the cluster used UDTs. See CASSANDRA-18504 - Entended max expiration time from 2038-01-19T03:14:06+00:00 to 2106-02-07T06:28:13+00:00 diff --git a/doc/modules/cassandra/pages/developing/cql/changes.adoc b/doc/modules/cassandra/pages/developing/cql/changes.adoc index 9dc3af00cb..7bf6bd6d03 100644 --- a/doc/modules/cassandra/pages/developing/cql/changes.adoc +++ b/doc/modules/cassandra/pages/developing/cql/changes.adoc @@ -4,6 +4,7 @@ The following describes the changes in each version of CQL. == 3.4.7 +* Add vector similarity functions (`18640`) * Remove deprecated functions `dateOf` and `unixTimestampOf`, replaced by `toTimestamp` and `toUnixTimestamp` (`18328`) * Added support for attaching masking functions to table columns (`18068`) * Add UNMASK permission (`18069`) diff --git a/doc/modules/cassandra/pages/developing/cql/functions.adoc b/doc/modules/cassandra/pages/developing/cql/functions.adoc index 6474b72cb5..244af3e7cf 100644 --- a/doc/modules/cassandra/pages/developing/cql/functions.adoc +++ b/doc/modules/cassandra/pages/developing/cql/functions.adoc @@ -284,6 +284,13 @@ A number of functions allow to obscure the real contents of a column containing include::partial$masking_functions.adoc[] +[[vector-similarity-functions]] += Vector similarity functions + +A number of functions allow to obtain the similarity score between vectors of floats. + +include::partial$vector_functions.adoc[] + [[user-defined-scalar-functions]] User-defined functions diff --git a/doc/modules/cassandra/partials/vector_functions.adoc b/doc/modules/cassandra/partials/vector_functions.adoc new file mode 100644 index 00..daa4b2b8ce --- /dev/null +++ b/doc/modules/cassandra/partials/vector_functions.adoc @@ -0,0 +1,41 @@ +[cols=",",options="header",] +|=== +|Function | Description + +| `similarity_cosine(vector, vector)` | Calculates the cosine similarity score between two float vectors of the same dimension. + +Examples: + +`similarity_cosine([0.1, 0.2], null)` -> `null` + +`similarity_cosine([0.1, 0.2], [0.1, 0.2])` -> `1` + +`s
[cassandra] branch cep-7-sai updated: Fix KeyRangeIntersectionIterator count
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cep-7-sai
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cep-7-sai by this push:
new 21a30509b0 Fix KeyRangeIntersectionIterator count
21a30509b0 is described below
commit 21a30509b0ee95ce72c44c403f1b2804a0ade5af
Author: Jonathan Ellis
AuthorDate: Mon Jul 17 14:23:09 2023 +0200
Fix KeyRangeIntersectionIterator count
patch by Jonathan Ellis; reviewed by Andrés de la Peña
---
.../iterators/KeyRangeIntersectionIterator.java| 12 ++-
.../index/sai/iterators/KeyRangeIterator.java | 23 --
.../KeyRangeIntersectionIteratorTest.java | 6 +++---
3 files changed, 31 insertions(+), 10 deletions(-)
diff --git
a/src/java/org/apache/cassandra/index/sai/iterators/KeyRangeIntersectionIterator.java
b/src/java/org/apache/cassandra/index/sai/iterators/KeyRangeIntersectionIterator.java
index f5bca7948b..63f273a2a5 100644
---
a/src/java/org/apache/cassandra/index/sai/iterators/KeyRangeIntersectionIterator.java
+++
b/src/java/org/apache/cassandra/index/sai/iterators/KeyRangeIntersectionIterator.java
@@ -249,6 +249,8 @@ public class KeyRangeIntersectionIterator extends
KeyRangeIterator
private static class IntersectionStatistics extends
KeyRangeIterator.Builder.Statistics
{
+private boolean empty = true;
+
@Override
public void update(KeyRangeIterator range)
{
@@ -256,7 +258,15 @@ public class KeyRangeIntersectionIterator extends
KeyRangeIterator
min = nullSafeMax(min, range.getMinimum());
// maximum of the intersection is the smallest maximum of
individual iterators
max = nullSafeMin(max, range.getMaximum());
-count += range.getCount();
+if (empty)
+{
+empty = false;
+count = range.getCount();
+}
+else
+{
+count = Math.min(count, range.getCount());
+}
}
}
diff --git
a/src/java/org/apache/cassandra/index/sai/iterators/KeyRangeIterator.java
b/src/java/org/apache/cassandra/index/sai/iterators/KeyRangeIterator.java
index f8bd38291b..84bf36bd59 100644
--- a/src/java/org/apache/cassandra/index/sai/iterators/KeyRangeIterator.java
+++ b/src/java/org/apache/cassandra/index/sai/iterators/KeyRangeIterator.java
@@ -18,10 +18,10 @@
package org.apache.cassandra.index.sai.iterators;
import java.io.Closeable;
-import java.util.List;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterables;
import org.apache.cassandra.index.sai.utils.PrimaryKey;
import org.apache.cassandra.utils.AbstractGuavaIterator;
@@ -29,6 +29,8 @@ import org.apache.cassandra.utils.AbstractGuavaIterator;
/**
* An abstract implementation of {@link AbstractGuavaIterator} that supports
the building and management of
* concatanation, union and intersection iterators.
+ *
+ * Only certain methods are designed to be overriden. The others are marked
private or final.
*/
public abstract class KeyRangeIterator extends
AbstractGuavaIterator implements Closeable
{
@@ -81,7 +83,8 @@ public abstract class KeyRangeIterator extends
AbstractGuavaIterator
*
* @param nextKey value to skip the iterator forward until matching
*
- * @return The next current key after the skip was performed
+ * @return The key skipped to, which will be the key returned by the
+ * next call to {@link #next()}, i.e., we are "peeking" at the next key as
part of the skip.
*/
public final PrimaryKey skipTo(PrimaryKey nextKey)
{
@@ -106,14 +109,22 @@ public abstract class KeyRangeIterator extends
AbstractGuavaIterator
return recomputeNext();
}
+/**
+ * Skip to nextKey.
+ *
+ * That is, implementations should set up the iterator state such that
+ * calling computeNext() will return nextKey if present,
+ * or the first one after it if not present.
+ */
protected abstract void performSkipTo(PrimaryKey nextKey);
-protected PrimaryKey recomputeNext()
+private PrimaryKey recomputeNext()
{
return tryToComputeNext() ? peek() : endOfData();
}
-protected boolean tryToComputeNext()
+@Override
+protected final boolean tryToComputeNext()
{
boolean hasNext = super.tryToComputeNext();
current = hasNext ? next : getMaximum();
@@ -159,9 +170,9 @@ public abstract class KeyRangeIterator extends
AbstractGuavaIterator
return statistics.count;
}
-public Builder add(List ranges)
+public Builder add(Iterable ranges)
{
-if (ranges == null || ranges.isEmpty())
+if (ranges == null || Iterables.isEm
[cassandra] branch cep-7-sai updated: Fix concurrency in bbtree reader by cloning state
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cep-7-sai
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cep-7-sai by this push:
new 83419e074d Fix concurrency in bbtree reader by cloning state
83419e074d is described below
commit 83419e074d131a68112470fc8761c3edafb5e675
Author: Mike Adamson
AuthorDate: Thu Jul 13 11:24:55 2023 +0100
Fix concurrency in bbtree reader by cloning state
patch by Mike Adamson; reviewed by Andrés de la Peña and Caleb Rackliffe
for CASSANDRA-18669
---
.../disk/v1/bbtree/BlockBalancedTreeReader.java| 33 ++--
.../disk/v1/bbtree/BlockBalancedTreeWalker.java| 152 +---
.../v1/bbtree/BlockBalancedTreeReaderTest.java | 192 +
.../sai/disk/v1/bbtree/BlockBalancedTreeTest.java | 54 +++---
4 files changed, 265 insertions(+), 166 deletions(-)
diff --git
a/src/java/org/apache/cassandra/index/sai/disk/v1/bbtree/BlockBalancedTreeReader.java
b/src/java/org/apache/cassandra/index/sai/disk/v1/bbtree/BlockBalancedTreeReader.java
index 59271e3c14..53cac195e0 100644
---
a/src/java/org/apache/cassandra/index/sai/disk/v1/bbtree/BlockBalancedTreeReader.java
+++
b/src/java/org/apache/cassandra/index/sai/disk/v1/bbtree/BlockBalancedTreeReader.java
@@ -78,17 +78,17 @@ public class BlockBalancedTreeReader extends
BlockBalancedTreeWalker implements
this.indexContext = indexContext;
this.postingsFile = postingsFile;
this.postingsIndex = new BlockBalancedTreePostingsIndex(postingsFile,
treePostingsRoot);
-leafOrderMapBitsRequired =
DirectWriter.unsignedBitsRequired(state.maxPointsInLeafNode - 1);
+leafOrderMapBitsRequired =
DirectWriter.unsignedBitsRequired(maxValuesInLeafNode - 1);
}
public int getBytesPerValue()
{
-return state.bytesPerValue;
+return bytesPerValue;
}
public long getPointCount()
{
-return state.valueCount;
+return valueCount;
}
@Override
@@ -101,7 +101,7 @@ public class BlockBalancedTreeReader extends
BlockBalancedTreeWalker implements
@SuppressWarnings({"resource", "RedundantSuppression"})
public PostingList intersect(IntersectVisitor visitor,
QueryEventListener.BalancedTreeEventListener listener, QueryContext context)
{
-Relation relation = visitor.compare(state.minPackedValue,
state.maxPackedValue);
+Relation relation = visitor.compare(minPackedValue, maxPackedValue);
if (relation == Relation.CELL_OUTSIDE_QUERY)
{
@@ -113,7 +113,6 @@ public class BlockBalancedTreeReader extends
BlockBalancedTreeWalker implements
IndexInput treeInput =
IndexFileUtils.instance.openInput(treeIndexFile);
IndexInput postingsInput =
IndexFileUtils.instance.openInput(postingsFile);
IndexInput postingsSummaryInput =
IndexFileUtils.instance.openInput(postingsFile);
-state.reset();
Intersection intersection = relation == Relation.CELL_INSIDE_QUERY
? new Intersection(treeInput,
postingsInput, postingsSummaryInput, listener, context)
@@ -131,6 +130,7 @@ public class BlockBalancedTreeReader extends
BlockBalancedTreeWalker implements
private final Stopwatch queryExecutionTimer =
Stopwatch.createStarted();
final QueryContext context;
+final TraversalState state;
final IndexInput treeInput;
final IndexInput postingsInput;
final IndexInput postingsSummaryInput;
@@ -140,12 +140,13 @@ public class BlockBalancedTreeReader extends
BlockBalancedTreeWalker implements
Intersection(IndexInput treeInput, IndexInput postingsInput,
IndexInput postingsSummaryInput,
QueryEventListener.BalancedTreeEventListener listener,
QueryContext context)
{
+this.state = newTraversalState();
this.treeInput = treeInput;
this.postingsInput = postingsInput;
this.postingsSummaryInput = postingsSummaryInput;
this.listener = listener;
this.context = context;
-postingLists = new PriorityQueue<>(state.numLeaves, COMPARATOR);
+postingLists = new PriorityQueue<>(numLeaves, COMPARATOR);
}
public PostingList execute()
@@ -247,14 +248,14 @@ public class BlockBalancedTreeReader extends
BlockBalancedTreeWalker implements
{
super(treeInput, postingsInput, postingsSummaryInput, listener,
context);
this.visitor = visitor;
-this.packedValue = new byte[state.bytesPerValue];
-this.origIndex = new short[state.maxPointsInLeafNode];
+this.packedValue = new byte[bytesPerValue];
+this.origIndex = new short[maxValuesInLeafNode];
}
@Override
publ
[cassandra] branch cep-7-sai updated (05dd58783a -> f6159654f4)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch cep-7-sai in repository https://gitbox.apache.org/repos/asf/cassandra.git omit 05dd58783a Add basic text analysis to SAI, including "case_sensitive", "normalize", and "ascii" modes omit e5337eb911 Query all ranges at once for SAI distributed queries omit 9057ff3e6d Various changes to SAI index validation omit de26b9089b Numeric on-disk index write and search omit 5d3f257477 Stream all compatible components registered by an SSTable omit f3865ad744 Upgrade to lucene-core 9.7.0 omit b018587fa0 fix Segment#intersects to compare bound instead of token omit 2aa3197aed send SSTableAddedNotification before MemtableDiscardedNotification to fix index data visilibity issue omit 6ac577308c post-rebase fixes for the rebase on trunk at fad1f7457032544ab6a7b40c5d38ecb8b25899bb omit 26f2c942ab Unify CQLTester#waitForIndex and SAITester#waitForIndexQueryable omit b661711765 Allow CQL queries on multiple indexes without ALLOW FILTERING omit 3914fb6674 Literal on-disk index and index write path (#9) omit 14ed6a7d73 In-memory index implementation with query path omit 07ad4d8491 Add Index Group Interface for Storage Attached Index add 914484dfd8 Ninja fix missing IDEA sun.rmi.transport.tcp compiler export after CASSANDRA-18511 add 1adbea5a06 Extend maximum expiration date add c32a848008 Run checkstyle only on j8 add e28ef4089e Suppress CVE-2023-2976 add 9c970cc117 Merge branch 'cassandra-3.0' into cassandra-3.11 add be8cd8ab6b Merge branch 'cassandra-3.11' into cassandra-4.0 add 6b99db84ba Merge branch 'cassandra-4.0' into cassandra-4.1 add 60200ba613 Merge branch 'cassandra-4.1' into trunk add 5655a33bc0 Fix Down nodes counter in nodetool describecluster add aa7ed179d3 Merge branch 'cassandra-4.0' into cassandra-4.1 add ed58b0ffc3 Merge branch 'cassandra-4.1' into trunk add a2dc44f072 Remove dependency on pytz library for setting CQLSH timezones on Python version >= 3.9 add 03da864bab Add keyspace and table name to exception message during ColumnSubselection deserialization add f368b9dc1e Merge branch 'cassandra-3.11' into cassandra-4.0 add 867c074dda Merge branch 'cassandra-4.0' into cassandra-4.1 add 984f519bd9 Merge branch 'cassandra-4.1' into trunk add 8d3c30dde4 Validate the comparison git branch when generating CircleCI configs add 3855c8a194 Merge branch 'cassandra-3.0' into cassandra-3.11 add 2393bb16f3 Merge branch 'cassandra-3.11' into cassandra-4.0 add 2b2d9f9b72 Merge branch 'cassandra-4.0' into cassandra-4.1 add e4f31b73c2 Merge branch 'cassandra-4.1' into trunk add 70389abcf7 Bti shouldn't be available in compatibility mode add c91e2714b9 Generate.sh -s param to skip autodetection of tests add 7de3e61d75 Merge branch 'cassandra-3.0' into cassandra-3.11 add c94eb0e37b Merge branch 'cassandra-3.11' into cassandra-4.0 add 2973a7bdcd Merge branch 'cassandra-4.0' into cassandra-4.1 add dcb763ec2c Merge branch 'cassandra-4.1' into trunk add 5143bd81e8 Track the amount of read data per row add b0cf9ba219 Merge branch 'cassandra-4.0' into cassandra-4.1 add ba3ad7487a Merge branch 'cassandra-4.1' into trunk add 4f3cb5de37 Avoid unnecessary deserialization of terminal arguments when executing CQL functions add 41a669a100 Deduplicate the MixedMode* upgrade jvm-dtests add b7c00d7d13 Slow builds due to checkstyle add 43d90928a8 Merge branch 'cassandra-4.1' into trunk add 1d961679a0 Switch Java and Python upgrade tests from running on JDK8 to JDK11 in CircleCI patch by Ekaterina Dimitrova; reviewed by Berenguer Blasi for CASSANDRA-18285 add 1eccb2bc1f Fix error message handling when trying to use CLUSTERING ORDER with non-clustering column add 2fcdaa5b76 Merge branch 'cassandra-3.11' into cassandra-4.0 add 1f6b37d189 Merge branch 'cassandra-4.0' into cassandra-4.1 add 4f5cb2a6fa Merge branch 'cassandra-4.1' into trunk add c37bcbf7e9 Include TLSv1.2 in negotiatedProtocolMustBeAcceptedProtocolTest Add a comment about the use of disabled TLSv1.1 with JDK 8 and higher to negotiatedProtocolMustBeAcceptedProtocolTest add 622397e7e5 Merge branch 'cassandra-4.0' into cassandra-4.1 add a55d4183f5 Merge branch 'cassandra-4.1' into trunk add 64d953fdca Wait for live endpoints in gossip waiting to settle add 49dba85ec8 Merge branch 'cassandra-3.11' into cassandra-4.0 add 2b98d487d0 Merge branch 'cassandra-4.0' into cassandra-4.1 add c7260399d6 Merge branch 'cassandra-4.1' into trunk add 83a3316246 CircleCI: Add separate approval step for oa utests on separate workflows add cc3e742c73 Fix SSTabledump errors when dumping data from index add 92d0a40c96 Fix assertion error when describin
[cassandra] 08/14: fix Segment#intersects to compare bound instead of token
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cep-7-sai
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit a47baeb341a29b69bf0cb35912386bf6a82ed1bf
Author: Zhao Yang
AuthorDate: Wed Jun 14 08:55:01 2023 +0800
fix Segment#intersects to compare bound instead of token
---
.../index/sai/disk/v1/segment/Segment.java | 19 --
.../index/sai/cql/TokenRangeReadTest.java | 44 ++
.../cassandra/index/sai/disk/v1/SegmentTest.java | 16
3 files changed, 58 insertions(+), 21 deletions(-)
diff --git
a/src/java/org/apache/cassandra/index/sai/disk/v1/segment/Segment.java
b/src/java/org/apache/cassandra/index/sai/disk/v1/segment/Segment.java
index a36a3504dc..5f108a638c 100644
--- a/src/java/org/apache/cassandra/index/sai/disk/v1/segment/Segment.java
+++ b/src/java/org/apache/cassandra/index/sai/disk/v1/segment/Segment.java
@@ -42,9 +42,7 @@ import org.apache.cassandra.io.util.FileUtils;
*/
public class Segment implements Closeable
{
-private final Token minKey;
private final Token.KeyBound minKeyBound;
-private final Token maxKey;
private final Token.KeyBound maxKeyBound;
// per sstable
@@ -56,10 +54,8 @@ public class Segment implements Closeable
public Segment(IndexContext indexContext, SSTableContext sstableContext,
PerColumnIndexFiles indexFiles, SegmentMetadata metadata) throws IOException
{
-this.minKey = metadata.minKey.token();
-this.minKeyBound = minKey.minKeyBound();
-this.maxKey = metadata.maxKey.token();
-this.maxKeyBound = maxKey.maxKeyBound();
+this.minKeyBound = metadata.minKey.token().minKeyBound();
+this.maxKeyBound = metadata.maxKey.token().maxKeyBound();
this.primaryKeyMapFactory = sstableContext.primaryKeyMapFactory;
this.metadata = metadata;
@@ -72,9 +68,7 @@ public class Segment implements Closeable
{
this.primaryKeyMapFactory = null;
this.metadata = null;
-this.minKey = minKey;
this.minKeyBound = minKey.minKeyBound();
-this.maxKey = maxKey;
this.maxKeyBound = maxKey.maxKeyBound();
this.index = null;
}
@@ -87,14 +81,15 @@ public class Segment implements Closeable
if (keyRange instanceof Range && ((Range)keyRange).isWrapAround())
return keyRange.contains(minKeyBound) ||
keyRange.contains(maxKeyBound);
-int cmp = keyRange.right.getToken().compareTo(minKey);
+int cmp = keyRange.right.compareTo(minKeyBound);
// if right is minimum, it means right is the max token and bigger
than maxKey.
-// if right bound is less than minKey, no intersection
+// if right bound is less than minKeyBound, no intersection
if (!keyRange.right.isMinimum() && (!keyRange.inclusiveRight() && cmp
== 0 || cmp < 0))
return false;
-// if left bound is bigger than maxKey, no intersection
-return keyRange.isStartInclusive() ||
keyRange.left.getToken().compareTo(maxKey) < 0;
+cmp = keyRange.left.compareTo(maxKeyBound);
+// if left bound is bigger than maxKeyBound, no intersection
+return (keyRange.isStartInclusive() || cmp != 0) && cmp <= 0;
}
public long indexFileCacheSize()
diff --git
a/test/unit/org/apache/cassandra/index/sai/cql/TokenRangeReadTest.java
b/test/unit/org/apache/cassandra/index/sai/cql/TokenRangeReadTest.java
new file mode 100644
index 00..67a030ef9f
--- /dev/null
+++ b/test/unit/org/apache/cassandra/index/sai/cql/TokenRangeReadTest.java
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.index.sai.cql;
+
+import org.junit.Test;
+
+import org.apache.cassandra.index.sai.SAITester;
+import org.apache.cassandra.index.sai.StorageAttachedIndex;
+
+import static java.lang.String.format;
+
+public class TokenRangeReadTest extends SAITester
+{
+@Test
+public void testTokenRangeRead() throws Throwable
+{
+createTable("CREATE TABLE %s (k1 int, v1 text, PRIMARY KEY (
[cassandra] 10/14: Stream all compatible components registered by an SSTable
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cep-7-sai
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 04fab6d0cdd74e4510a3d9ba985d28d1b975cde2
Author: Piotr Kołaczkowski
AuthorDate: Thu Jun 15 15:07:18 2023 +0200
Stream all compatible components registered by an SSTable
patch by Piotr Kołaczkowski; reviewed by Andrés de la Peña and Caleb
Rackliffe for CASSANDRA-18345
---
.../CassandraEntireSSTableStreamReader.java| 2 +-
.../db/streaming/CassandraOutgoingFile.java| 4 +-
.../cassandra/db/streaming/ComponentContext.java | 6 +-
.../cassandra/db/streaming/ComponentManifest.java | 12 +-
.../index/sai/disk/format/IndexComponent.java | 18 +++
.../cassandra/index/sai/disk/format/Version.java | 9 +-
.../index/sai/disk/v1/V1OnDiskFormat.java | 14 +-
.../org/apache/cassandra/io/sstable/Component.java | 26 ++--
.../org/apache/cassandra/io/sstable/SSTable.java | 11 ++
.../io/sstable/SSTableZeroCopyWriter.java | 26 ++--
.../org/apache/cassandra/io/sstable/SSTable_API.md | 14 +-
.../cassandra/io/sstable/format/SSTableFormat.java | 19 ++-
.../cassandra/io/sstable/format/big/BigFormat.java | 20 +--
.../cassandra/io/sstable/format/bti/BtiFormat.java | 19 +--
.../distributed/test/sai/IndexStreamingTest.java | 155 +
.../microbench/ZeroCopyStreamingBenchmark.java | 2 +-
.../CassandraEntireSSTableStreamWriterTest.java| 4 +-
.../db/streaming/CassandraStreamHeaderTest.java| 4 +-
.../apache/cassandra/io/sstable/ComponentTest.java | 20 +--
.../io/sstable/SSTableZeroCopyWriterTest.java | 2 +-
...ntireSSTableStreamingCorrectFilesCountTest.java | 2 +-
21 files changed, 278 insertions(+), 111 deletions(-)
diff --git
a/src/java/org/apache/cassandra/db/streaming/CassandraEntireSSTableStreamReader.java
b/src/java/org/apache/cassandra/db/streaming/CassandraEntireSSTableStreamReader.java
index 9c5e048a12..98e2b6f7ef 100644
---
a/src/java/org/apache/cassandra/db/streaming/CassandraEntireSSTableStreamReader.java
+++
b/src/java/org/apache/cassandra/db/streaming/CassandraEntireSSTableStreamReader.java
@@ -121,7 +121,7 @@ public class CassandraEntireSSTableStreamReader implements
IStreamReader
prettyPrintMemory(bytesRead),
prettyPrintMemory(totalSize));
-writer.writeComponent(component.type, in, length);
+writer.writeComponent(component, in, length);
session.progress(writer.descriptor.fileFor(component).toString(),
ProgressInfo.Direction.IN, length, length, length);
bytesRead += length;
diff --git
a/src/java/org/apache/cassandra/db/streaming/CassandraOutgoingFile.java
b/src/java/org/apache/cassandra/db/streaming/CassandraOutgoingFile.java
index 88ecff8a44..9569769eba 100644
--- a/src/java/org/apache/cassandra/db/streaming/CassandraOutgoingFile.java
+++ b/src/java/org/apache/cassandra/db/streaming/CassandraOutgoingFile.java
@@ -65,7 +65,7 @@ public class CassandraOutgoingFile implements OutgoingStream
this.filename = sstable.getFilename();
this.shouldStreamEntireSSTable = computeShouldStreamEntireSSTables();
-ComponentManifest manifest =
ComponentManifest.create(sstable.descriptor);
+ComponentManifest manifest = ComponentManifest.create(sstable);
this.header = makeHeader(sstable, operation, sections, estimatedKeys,
shouldStreamEntireSSTable, manifest);
}
@@ -154,7 +154,7 @@ public class CassandraOutgoingFile implements OutgoingStream
// redistribution, otherwise file sizes recorded in component
manifest will be different from actual
// file sizes.
// Recreate the latest manifest and hard links for mutatable
components in case they are modified.
-try (ComponentContext context = sstable.runWithLock(ignored ->
ComponentContext.create(sstable.descriptor)))
+try (ComponentContext context = sstable.runWithLock(ignored ->
ComponentContext.create(sstable)))
{
CassandraStreamHeader current = makeHeader(sstable, operation,
sections, estimatedKeys, true, context.manifest());
CassandraStreamHeader.serializer.serialize(current, out,
version);
diff --git a/src/java/org/apache/cassandra/db/streaming/ComponentContext.java
b/src/java/org/apache/cassandra/db/streaming/ComponentContext.java
index 164dd6ba5c..c03e7b4c34 100644
--- a/src/java/org/apache/cassandra/db/streaming/ComponentContext.java
+++ b/src/java/org/apache/cassandra/db/streaming/ComponentContext.java
@@ -28,6 +28,7 @@ import org.slf4j.LoggerFactory;
import org.apache.cassandra.io.sstable.Component;
import org.apache.cassandra.io.sstable.Descriptor;
+import org.apache.cassandra.io.sstable.SSTable;
import org.apache.cassandra.io.util.File;
[cassandra] 09/14: Upgrade to lucene-core 9.7.0
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cep-7-sai
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 2eeab3377dc17655a1e974c79a48e5deede38529
Author: Jonathan Ellis
AuthorDate: Mon Jun 26 14:50:01 2023 -0500
Upgrade to lucene-core 9.7.0
Notes on the upgrade path:
- RamIndexOutput is replaced with ResettableByteBuffersIndexOutput, an
extension of ByteBuffersIndexOutput, which was the closest thing to a
replacement of RamIndexOutput.
- Lucene exposes the code we needed from DirectReaders more or less
directly in DirectReader now, so the old copied code has been deleted.
- Lucene changed its data files to be little endian, but to keep its
compatibility story simple it retained BE for the header and footer ints.
That's the cause of the changes in SAICodecUtils.
- We could gain a bit of performance making our own code natively little
endian but that is too big of a change for this patch.
patch by Jonathan Ellis; reviewed by Andrés de la Peña, Caleb Rackliffe,
and Mike Adamson for CASSANDRA-18494
---
.build/parent-pom-template.xml | 2 +-
.../sai/disk/ResettableByteBuffersIndexOutput.java | 153 ++
.../index/sai/disk/io/IndexInputReader.java| 35 +---
.../index/sai/disk/io/IndexOutputWriter.java | 5 +
.../index/sai/disk/io/RAMIndexOutput.java | 86
.../cassandra/index/sai/disk/v1/DirectReaders.java | 230 +
.../index/sai/disk/v1/MetadataSource.java | 7 +-
.../index/sai/disk/v1/MetadataWriter.java | 6 +-
.../cassandra/index/sai/disk/v1/SAICodecUtils.java | 24 ++-
.../disk/v1/bitpack/AbstractBlockPackedReader.java | 9 +-
.../disk/v1/bitpack/AbstractBlockPackedWriter.java | 8 +-
.../sai/disk/v1/bitpack/BlockPackedReader.java | 8 +-
.../v1/bitpack/MonotonicBlockPackedReader.java | 7 +-
.../sai/disk/v1/bitpack/NumericValuesMeta.java | 4 +-
.../index/sai/disk/v1/postings/PostingsReader.java | 74 ---
.../index/sai/disk/v1/postings/PostingsWriter.java | 25 ++-
.../index/sai/disk/v1/segment/SegmentMetadata.java | 12 +-
.../sai/disk/v1/sortedterms/SortedTermsMeta.java | 4 +-
.../index/sai/memory/ByteSliceReader.java | 12 +-
.../cassandra/index/sai/disk/v1/MetadataTest.java | 6 +-
.../index/sai/disk/v1/SAICodecUtilsTest.java | 44 ++--
.../index/sai/disk/v1/TermsReaderTest.java | 2 +-
22 files changed, 302 insertions(+), 461 deletions(-)
diff --git a/.build/parent-pom-template.xml b/.build/parent-pom-template.xml
index 47cf919bee..c0a48b2dda 100644
--- a/.build/parent-pom-template.xml
+++ b/.build/parent-pom-template.xml
@@ -1047,7 +1047,7 @@
org.apache.lucene
lucene-core
-7.5.0
+9.7.0
com.carrotsearch.randomizedtesting
diff --git
a/src/java/org/apache/cassandra/index/sai/disk/ResettableByteBuffersIndexOutput.java
b/src/java/org/apache/cassandra/index/sai/disk/ResettableByteBuffersIndexOutput.java
new file mode 100644
index 00..19430a5087
--- /dev/null
+++
b/src/java/org/apache/cassandra/index/sai/disk/ResettableByteBuffersIndexOutput.java
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.index.sai.disk;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.lucene.store.ByteBuffersDataOutput;
+import org.apache.lucene.store.ByteBuffersIndexOutput;
+import org.apache.lucene.store.DataInput;
+import org.apache.lucene.store.IndexOutput;
+
+/***
+ * A wrapper around {@link ByteBuffersIndexOutput} that adds several methods
that interact
+ * with the underlying delegate.
+ */
+public class ResettableByteBuffersIndexOutput extends IndexOutput
+{
+private final ByteBuffersIndexOutput bbio;
+private final ByteBuffersDataOutput delegate;
+
+public ResettableByteBuffersIndexOutput(String name)
+{
+//TODO CASSANDRA-18280 to investigate the initial size allocation
+this(128, name);
+}
+
+public ResettableByteBuffersIndexOut
[cassandra] 13/14: Query all ranges at once for SAI distributed queries
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cep-7-sai
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit b3024f3d280ea90ea7452eb316bba92ac4c6e4fe
Author: Mike Adamson
AuthorDate: Wed May 10 15:05:15 2023 +0100
Query all ranges at once for SAI distributed queries
patch by Mike Adamson; reviewed by Caleb Rackliffe, Andres de la Peña, and
Berenguer Blasi for CASSANDRA-18515
---
.../service/reads/range/RangeCommands.java | 31 +++--
.../test/sai/ConcurrencyFactorTest.java| 133 +
2 files changed, 154 insertions(+), 10 deletions(-)
diff --git
a/src/java/org/apache/cassandra/service/reads/range/RangeCommands.java
b/src/java/org/apache/cassandra/service/reads/range/RangeCommands.java
index 865b03d487..ca1ceebe76 100644
--- a/src/java/org/apache/cassandra/service/reads/range/RangeCommands.java
+++ b/src/java/org/apache/cassandra/service/reads/range/RangeCommands.java
@@ -79,19 +79,30 @@ public class RangeCommands
keyspace,
consistencyLevel);
-// our estimate of how many result rows there will be per-range
-float resultsPerRange = estimateResultsPerRange(command, keyspace);
-// underestimate how many rows we will get per-range in order to
increase the likelihood that we'll
-// fetch enough rows in the first round
-resultsPerRange -= resultsPerRange * CONCURRENT_SUBREQUESTS_MARGIN;
int maxConcurrencyFactor = Math.min(replicaPlans.size(),
MAX_CONCURRENT_RANGE_REQUESTS);
-int concurrencyFactor = resultsPerRange == 0.0
+int concurrencyFactor = maxConcurrencyFactor;
+Index.QueryPlan queryPlan = command.indexQueryPlan();
+if (queryPlan == null || queryPlan.shouldEstimateInitialConcurrency())
+{
+// our estimate of how many result rows there will be per-range
+float resultsPerRange = estimateResultsPerRange(command, keyspace);
+// underestimate how many rows we will get per-range in order to
increase the likelihood that we'll
+// fetch enough rows in the first round
+resultsPerRange -= resultsPerRange * CONCURRENT_SUBREQUESTS_MARGIN;
+concurrencyFactor = resultsPerRange == 0.0
? 1
: Math.max(1, Math.min(maxConcurrencyFactor,
(int) Math.ceil(command.limits().count() / resultsPerRange)));
-logger.trace("Estimated result rows per range: {}; requested rows: {},
ranges.size(): {}; concurrent range requests: {}",
- resultsPerRange, command.limits().count(),
replicaPlans.size(), concurrencyFactor);
-Tracing.trace("Submitting range requests on {} ranges with a
concurrency of {} ({} rows per range expected)",
- replicaPlans.size(), concurrencyFactor, resultsPerRange);
+logger.trace("Estimated result rows per range: {}; requested rows:
{}, ranges.size(): {}; concurrent range requests: {}",
+ resultsPerRange, command.limits().count(),
replicaPlans.size(), concurrencyFactor);
+Tracing.trace("Submitting range requests on {} ranges with a
concurrency of {} ({} rows per range expected)",
+ replicaPlans.size(), concurrencyFactor,
resultsPerRange);
+}
+else
+{
+logger.trace("Max concurrent range requests: {}; requested rows:
{}, ranges.size(): {}; concurrent range requests: {}",
+ MAX_CONCURRENT_RANGE_REQUESTS,
command.limits().count(), replicaPlans.size(), concurrencyFactor);
+Tracing.trace("Submitting range requests on {} ranges with a
concurrency of {}", replicaPlans.size(), concurrencyFactor);
+}
ReplicaPlanMerger mergedReplicaPlans = new
ReplicaPlanMerger(replicaPlans, keyspace, consistencyLevel);
return new RangeCommandIterator(mergedReplicaPlans,
diff --git
a/test/distributed/org/apache/cassandra/distributed/test/sai/ConcurrencyFactorTest.java
b/test/distributed/org/apache/cassandra/distributed/test/sai/ConcurrencyFactorTest.java
new file mode 100644
index 00..6c39591c2f
--- /dev/null
+++
b/test/distributed/org/apache/cassandra/distributed/test/sai/ConcurrencyFactorTest.java
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of
[cassandra] 12/14: Various changes to SAI index validation
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cep-7-sai
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 73dcc94a140d2afcc21ff5fb2a2fe8c62061fe10
Author: Piotr Kołaczkowski
AuthorDate: Tue Jun 20 16:03:03 2023 +0200
Various changes to SAI index validation
- Fix checksum calculation in IncrementalChecksumSequentialWriter
- Checksum per-SSTable and per-column components after streaming
- Avoid validating indexes when full rebuild is requested
patch by Piotr Kołaczkowski; reviewed by Caleb Rackliffe and Andres de la
Peña for CASSANDRA-18490
Co-authored-by: Piotr Kołaczkowski
Co-authored-by: Caleb Rackliffe
---
.../apache/cassandra/index/sai/IndexContext.java | 10 +--
.../cassandra/index/sai/IndexValidation.java | 38 +
.../cassandra/index/sai/SSTableContextManager.java | 6 +-
.../cassandra/index/sai/StorageAttachedIndex.java | 13 +--
.../index/sai/StorageAttachedIndexBuilder.java | 10 ++-
.../index/sai/StorageAttachedIndexGroup.java | 16 ++--
.../index/sai/disk/format/IndexDescriptor.java | 28 +++
.../index/sai/disk/io/IndexFileUtils.java | 69 +++-
.../index/sai/disk/io/IndexOutputWriter.java | 12 ++-
.../index/sai/disk/v1/V1OnDiskFormat.java | 13 ++-
.../cassandra/index/sai/view/IndexViewManager.java | 9 ++-
.../org/apache/cassandra/index/sai/SAITester.java | 15 ++--
.../index/sai/disk/v1/trie/TrieValidationTest.java | 93 ++
13 files changed, 212 insertions(+), 120 deletions(-)
diff --git a/src/java/org/apache/cassandra/index/sai/IndexContext.java
b/src/java/org/apache/cassandra/index/sai/IndexContext.java
index f16607395d..14a6db6c4b 100644
--- a/src/java/org/apache/cassandra/index/sai/IndexContext.java
+++ b/src/java/org/apache/cassandra/index/sai/IndexContext.java
@@ -162,9 +162,9 @@ public class IndexContext
/**
* @return A set of SSTables which have attached to them invalid index
components.
*/
-public Collection
onSSTableChanged(Collection oldSSTables,
Collection newSSTables, boolean validate)
+public Collection
onSSTableChanged(Collection oldSSTables,
Collection newSSTables, IndexValidation validation)
{
-return viewManager.update(oldSSTables, newSSTables, validate);
+return viewManager.update(oldSSTables, newSSTables, validation);
}
public ColumnMetadata getDefinition()
@@ -408,7 +408,7 @@ public class IndexContext
* @return the indexes that are built on the given SSTables on the left
and corrupted indexes'
* corresponding contexts on the right
*/
-public Pair, Collection>
getBuiltIndexes(Collection sstableContexts, boolean validate)
+public Pair, Collection>
getBuiltIndexes(Collection sstableContexts, IndexValidation
validation)
{
Set valid = new HashSet<>(sstableContexts.size());
Set invalid = new HashSet<>();
@@ -433,9 +433,9 @@ public class IndexContext
try
{
-if (validate)
+if (validation != IndexValidation.NONE)
{
-if
(!sstableContext.indexDescriptor.validatePerIndexComponents(this))
+if
(!sstableContext.indexDescriptor.validatePerIndexComponents(this, validation))
{
logger.warn(logMessage("Invalid per-column component
for SSTable {}"), sstableContext.descriptor());
invalid.add(sstableContext);
diff --git a/src/java/org/apache/cassandra/index/sai/IndexValidation.java
b/src/java/org/apache/cassandra/index/sai/IndexValidation.java
new file mode 100644
index 00..edd9e0fd1c
--- /dev/null
+++ b/src/java/org/apache/cassandra/index/sai/IndexValidation.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.index.sai;
+
+public enum IndexValidation
+{
+/**
+ * No validation to be performed
+ */
+NONE,
+
+/**
+ * Basic header/footer validation, but no data validation (fast)
+
[cassandra] 14/14: Add basic text analysis to SAI, including "case_sensitive", "normalize", and "ascii" modes
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cep-7-sai
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit f6159654f4d816504466214f0f1d9d0cfc87f916
Author: Mike Adamson
AuthorDate: Tue May 9 12:29:01 2023 +0100
Add basic text analysis to SAI, including "case_sensitive", "normalize",
and "ascii" modes
patch by Mike Adamson; reviewed by Caleb Rackliffe and Andres de la Peña
for CASSANDRA-18479
---
.build/cassandra-deps-template.xml | 8 +-
.build/parent-pom-template.xml | 10 +-
.../apache/cassandra/index/sai/IndexContext.java | 31 ++--
.../cassandra/index/sai/StorageAttachedIndex.java | 6 +-
.../index/sai/analyzer/AbstractAnalyzer.java | 26
.../index/sai/analyzer/NonTokenizingAnalyzer.java | 147 +++
.../index/sai/analyzer/NonTokenizingOptions.java | 156 +
.../index/sai/analyzer/filter/BasicFilters.java| 82 +++
.../index/sai/analyzer/filter/FilterPipeline.java | 69 +
.../analyzer/filter/FilterPipelineExecutor.java| 40 ++
.../index/sai/disk/v1/SSTableIndexWriter.java | 2 +-
.../index/sai/memory/TrieMemoryIndex.java | 2 +-
.../cassandra/index/sai/plan/Expression.java | 2 +-
.../apache/cassandra/index/sai/plan/Operation.java | 2 +-
.../sai/plan/StorageAttachedIndexSearcher.java | 2 +-
.../index/sai/virtual/ColumnIndexesSystemView.java | 2 +-
.../index/sasi/analyzer/filter/StemmerFactory.java | 55 +---
.../test/sai/ReplicaFilteringProtectionTest.java | 67 +
.../sai/analyzer/NonTokenizingAnalyzerTest.java| 75 ++
.../sai/analyzer/filter/BasicFiltersTest.java | 73 ++
.../index/sai/cql/StorageAttachedIndexDDLTest.java | 106 ++
.../index/sai/virtual/IndexesSystemViewTest.java | 2 +-
22 files changed, 904 insertions(+), 61 deletions(-)
diff --git a/.build/cassandra-deps-template.xml
b/.build/cassandra-deps-template.xml
index 2115b78f02..3cc7fc741e 100644
--- a/.build/cassandra-deps-template.xml
+++ b/.build/cassandra-deps-template.xml
@@ -324,10 +324,6 @@
org.hdrhistogram
HdrHistogram
-
- com.github.rholder
- snowball-stemmer
-
com.googlecode.concurrent-trees
concurrent-trees
@@ -376,5 +372,9 @@
org.apache.lucene
lucene-core
+
+ org.apache.lucene
+ lucene-analysis-common
+
diff --git a/.build/parent-pom-template.xml b/.build/parent-pom-template.xml
index c0a48b2dda..7566d78bc4 100644
--- a/.build/parent-pom-template.xml
+++ b/.build/parent-pom-template.xml
@@ -876,11 +876,6 @@
-
-com.github.rholder
-snowball-stemmer
-1.3.0.581.1
-
com.googlecode.concurrent-trees
concurrent-trees
@@ -1049,6 +1044,11 @@
lucene-core
9.7.0
+
+org.apache.lucene
+lucene-analysis-common
+9.7.0
+
com.carrotsearch.randomizedtesting
randomizedtesting-runner
diff --git a/src/java/org/apache/cassandra/index/sai/IndexContext.java
b/src/java/org/apache/cassandra/index/sai/IndexContext.java
index 14a6db6c4b..d0a5e08b16 100644
--- a/src/java/org/apache/cassandra/index/sai/IndexContext.java
+++ b/src/java/org/apache/cassandra/index/sai/IndexContext.java
@@ -90,8 +90,7 @@ public class IndexContext
private final IndexViewManager viewManager;
private final IndexMetrics indexMetrics;
private final ColumnQueryMetrics columnQueryMetrics;
-private final AbstractAnalyzer.AnalyzerFactory indexAnalyzerFactory;
-private final AbstractAnalyzer.AnalyzerFactory queryAnalyzerFactory;
+private final AbstractAnalyzer.AnalyzerFactory analyzerFactory;
private final PrimaryKey.Factory primaryKeyFactory;
public IndexContext(String keyspace,
@@ -119,9 +118,8 @@ public class IndexContext
this.columnQueryMetrics = isLiteral() ? new
ColumnQueryMetrics.TrieIndexMetrics(this)
: new
ColumnQueryMetrics.BalancedTreeIndexMetrics(this);
-// We currently only support the NoOpAnalyzer
-this.indexAnalyzerFactory =
AbstractAnalyzer.fromOptions(getValidator(), Collections.emptyMap());
-this.queryAnalyzerFactory =
AbstractAnalyzer.fromOptions(getValidator(), Collections.emptyMap());
+this.analyzerFactory = indexMetadata == null ?
AbstractAnalyzer.fromOptions(getValidator(), Collections.emptyMap())
+ :
AbstractAnalyzer.fromOptions(getValidator(), indexMetadata.options);
}
public AbstractType keyValidator()
@@ -199,23 +197,12 @@ public class IndexContext
}
/**
- * Returns an {@code AnalyzerFactory} for use by the write path
[cassandra] 07/14: send SSTableAddedNotification before MemtableDiscardedNotification to fix index data visilibity issue
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch cep-7-sai in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 9f6d3b5324f220558863e51f843196308eedade7 Author: Zhao Yang AuthorDate: Tue Jun 13 14:02:34 2023 +0800 send SSTableAddedNotification before MemtableDiscardedNotification to fix index data visilibity issue --- src/java/org/apache/cassandra/db/lifecycle/Tracker.java | 5 +++-- test/unit/org/apache/cassandra/db/lifecycle/TrackerTest.java | 12 ++-- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/src/java/org/apache/cassandra/db/lifecycle/Tracker.java b/src/java/org/apache/cassandra/db/lifecycle/Tracker.java index 575af6cff5..94c83f041c 100644 --- a/src/java/org/apache/cassandra/db/lifecycle/Tracker.java +++ b/src/java/org/apache/cassandra/db/lifecycle/Tracker.java @@ -411,11 +411,12 @@ public class Tracker Throwable fail; fail = updateSizeTracking(emptySet(), sstables, null); -notifyDiscarded(memtable); - // TODO: if we're invalidated, should we notifyadded AND removed, or just skip both? fail = notifyAdded(sstables, false, memtable, fail); +// make sure index sees flushed index files before dicarding memtable index +notifyDiscarded(memtable); + if (!isDummy() && !cfstore.isValid()) dropSSTables(); diff --git a/test/unit/org/apache/cassandra/db/lifecycle/TrackerTest.java b/test/unit/org/apache/cassandra/db/lifecycle/TrackerTest.java index d6e3742dcc..7192d50031 100644 --- a/test/unit/org/apache/cassandra/db/lifecycle/TrackerTest.java +++ b/test/unit/org/apache/cassandra/db/lifecycle/TrackerTest.java @@ -327,9 +327,9 @@ public class TrackerTest tracker.replaceFlushed(prev2, singleton(reader)); Assert.assertEquals(1, tracker.getView().sstables.size()); Assert.assertEquals(2, listener.received.size()); -Assert.assertEquals(prev2, ((MemtableDiscardedNotification) listener.received.get(0)).memtable); -Assert.assertEquals(singleton(reader), ((SSTableAddedNotification) listener.received.get(1)).added); -Assert.assertEquals(Optional.of(prev2), ((SSTableAddedNotification) listener.received.get(1)).memtable()); +Assert.assertEquals(singleton(reader), ((SSTableAddedNotification) listener.received.get(0)).added); +Assert.assertEquals(Optional.of(prev2), ((SSTableAddedNotification) listener.received.get(0)).memtable()); +Assert.assertEquals(prev2, ((MemtableDiscardedNotification) listener.received.get(1)).memtable); listener.received.clear(); if (reader instanceof KeyCacheSupport) Assert.assertTrue(((KeyCacheSupport) reader).getKeyCache().isEnabled()); @@ -350,9 +350,9 @@ public class TrackerTest Assert.assertEquals(0, cfs.metric.liveDiskSpaceUsed.getCount()); Assert.assertEquals(5, listener.received.size()); Assert.assertEquals(prev1, ((MemtableSwitchedNotification) listener.received.get(0)).memtable); -Assert.assertEquals(prev1, ((MemtableDiscardedNotification) listener.received.get(1)).memtable); -Assert.assertEquals(singleton(reader), ((SSTableAddedNotification) listener.received.get(2)).added); -Assert.assertEquals(Optional.of(prev1), ((SSTableAddedNotification) listener.received.get(2)).memtable()); +Assert.assertEquals(singleton(reader), ((SSTableAddedNotification) listener.received.get(1)).added); +Assert.assertEquals(Optional.of(prev1), ((SSTableAddedNotification) listener.received.get(1)).memtable()); +Assert.assertEquals(prev1, ((MemtableDiscardedNotification) listener.received.get(2)).memtable); Assert.assertTrue(listener.received.get(3) instanceof SSTableDeletingNotification); Assert.assertEquals(1, ((SSTableListChangedNotification) listener.received.get(4)).removed.size()); DatabaseDescriptor.setIncrementalBackupsEnabled(backups); - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 05/14: Unify CQLTester#waitForIndex and SAITester#waitForIndexQueryable
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cep-7-sai
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 2a21b349afab10a6d885571bdd8f348a0d8252b5
Author: Andrés de la Peña
AuthorDate: Thu May 11 15:01:00 2023 +0100
Unify CQLTester#waitForIndex and SAITester#waitForIndexQueryable
patch by Andrés de la Peña; reviewed by Caleb Rackliffe for CASSANDRA-18521
---
.../cassandra/cql3/functions/types/ParseUtils.java | 4 +-
test/unit/org/apache/cassandra/cql3/CQLTester.java | 241 +++--
.../org/apache/cassandra/cql3/CQLTesterTest.java | 82 +++
.../org/apache/cassandra/cql3/KeyCacheCqlTest.java | 4 +-
.../entities/SecondaryIndexOnMapEntriesTest.java | 2 +-
.../validation/entities/SecondaryIndexTest.java| 38 ++--
.../operations/InsertUpdateIfConditionTest.java| 2 -
.../db/guardrails/GuardrailTablesTest.java | 2 +-
.../cassandra/index/SecondaryIndexManagerTest.java | 127 +--
.../index/internal/CassandraIndexTest.java | 9 +-
.../org/apache/cassandra/index/sai/SAITester.java | 41 +---
.../index/sai/cql/AllowFilteringTest.java | 6 -
.../cassandra/index/sai/cql/BaseDataModel.java | 29 ++-
.../cassandra/index/sai/cql/BooleanTypeTest.java | 1 -
.../index/sai/cql/CollectionIndexingTest.java | 3 -
.../index/sai/cql/DecimalLargeValueTest.java | 2 -
.../index/sai/cql/DuplicateRowIDTest.java | 1 -
.../sai/cql/MixedIndexImplementationsTest.java | 4 -
.../index/sai/cql/MultipleColumnIndexTest.java | 2 -
.../index/sai/cql/SingleNodeExecutor.java | 4 +-
.../index/sai/cql/StorageAttachedIndexDDLTest.java | 36 ++-
.../index/sai/cql/types/IndexingTypeSupport.java | 2 -
.../cassandra/index/sai/disk/NodeStartupTest.java | 5 +-
.../index/sai/disk/SelectiveIntersectionTest.java | 2 -
.../index/sai/disk/SingleNodeQueryFailureTest.java | 1 -
.../index/sai/functional/CompactionTest.java | 2 -
.../index/sai/functional/DiskSpaceTest.java| 1 -
.../index/sai/functional/DropTableTest.java| 1 -
.../index/sai/functional/FailureTest.java | 2 +-
.../index/sai/functional/GroupComponentsTest.java | 3 -
.../index/sai/functional/NodeRestartTest.java | 9 +-
.../index/sai/functional/SnapshotTest.java | 2 -
.../index/sai/metrics/QueryMetricsTest.java| 1 -
.../sai/metrics/SegmentFlushingFailureTester.java | 1 -
.../index/sai/metrics/StateMetricsTest.java| 6 +-
.../cassandra/index/sai/plan/OperationTest.java| 2 +-
.../index/sai/virtual/IndexesSystemViewTest.java | 4 +-
.../index/sai/virtual/SSTablesSystemViewTest.java | 2 -
.../index/sai/virtual/SegmentsSystemViewTest.java | 1 -
39 files changed, 383 insertions(+), 304 deletions(-)
diff --git a/src/java/org/apache/cassandra/cql3/functions/types/ParseUtils.java
b/src/java/org/apache/cassandra/cql3/functions/types/ParseUtils.java
index 8972beed15..8d0f29b45b 100644
--- a/src/java/org/apache/cassandra/cql3/functions/types/ParseUtils.java
+++ b/src/java/org/apache/cassandra/cql3/functions/types/ParseUtils.java
@@ -292,7 +292,7 @@ public abstract class ParseUtils
* @param value The string to un-double quote.
* @return The un-double quoted string.
*/
-static String unDoubleQuote(String value)
+public static String unDoubleQuote(String value)
{
return unquote(value, '"');
}
@@ -482,7 +482,7 @@ public abstract class ParseUtils
* @return {@code true} if the given string is surrounded by the quote
character, and {@code
* false} otherwise.
*/
-private static boolean isQuoted(String value, char quoteChar)
+public static boolean isQuoted(String value, char quoteChar)
{
return value != null
&& value.length() > 1
diff --git a/test/unit/org/apache/cassandra/cql3/CQLTester.java
b/test/unit/org/apache/cassandra/cql3/CQLTester.java
index fa34d5cdc7..c5da07aeb8 100644
--- a/test/unit/org/apache/cassandra/cql3/CQLTester.java
+++ b/test/unit/org/apache/cassandra/cql3/CQLTester.java
@@ -51,7 +51,7 @@ import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
import java.util.stream.Stream;
-
+import javax.annotation.Nullable;
import javax.management.MBeanServerConnection;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
@@ -103,6 +103,7 @@ import org.apache.cassandra.config.DataStorageSpec;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.config.EncryptionOptions;
import org.apache.cassandra.cql3.functions.FunctionName;
+import org.apache.cassandra.cql3.functions.types.ParseUtils;
import org.apache.cassandra.db.ColumnFamilyStore;
import org.apache.cassandra.db.Directories;
import org.apache.cassa
[cassandra] 06/14: post-rebase fixes for the rebase on trunk at fad1f7457032544ab6a7b40c5d38ecb8b25899bb
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cep-7-sai
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit d46f8ef9ec5355b825ffd1afe4eae1437142b6c8
Author: Caleb Rackliffe
AuthorDate: Thu Jun 1 14:43:20 2023 -0500
post-rebase fixes for the rebase on trunk at
fad1f7457032544ab6a7b40c5d38ecb8b25899bb
---
.../apache/cassandra/config/CassandraRelevantProperties.java | 7 ++-
src/java/org/apache/cassandra/index/IndexStatusManager.java | 12
src/java/org/apache/cassandra/index/sai/QueryContext.java| 4 +++-
.../cassandra/index/sai/disk/v1/SSTableIndexWriter.java | 7 +--
.../cassandra/index/sai/disk/v1/trie/TriePrefixSearcher.java | 4 ++--
src/java/org/apache/cassandra/io/tries/TrieNode.java | 2 +-
src/java/org/apache/cassandra/io/tries/ValueIterator.java| 2 +-
test/unit/org/apache/cassandra/index/sai/SAITester.java | 11 ---
.../org/apache/cassandra/index/sai/plan/OperationTest.java | 3 ++-
.../cassandra/index/sai/virtual/SegmentsSystemViewTest.java | 4 +++-
test/unit/org/apache/cassandra/io/tries/WalkerTest.java | 1 -
test/unit/org/apache/cassandra/io/util/SizedIntsTest.java| 1 -
.../cassandra/io/util/TailOverridingRebuffererTest.java | 4 +---
13 files changed, 36 insertions(+), 26 deletions(-)
diff --git
a/src/java/org/apache/cassandra/config/CassandraRelevantProperties.java
b/src/java/org/apache/cassandra/config/CassandraRelevantProperties.java
index a41652606e..aeb45903fc 100644
--- a/src/java/org/apache/cassandra/config/CassandraRelevantProperties.java
+++ b/src/java/org/apache/cassandra/config/CassandraRelevantProperties.java
@@ -416,12 +416,15 @@ public enum CassandraRelevantProperties
RING_DELAY("cassandra.ring_delay_ms"),
// SAI specific properties
+
/** Controls the maximum number of index query intersections that will
take part in a query */
SAI_INTERSECTION_CLAUSE_LIMIT("cassandra.sai.intersection.clause.limit",
"2"),
/** Latest version to be used for SAI index writing */
SAI_LATEST_VERSION("cassandra.sai.latest_version", "aa"),
+SAI_MAX_FROZEN_TERM_SIZE("cassandra.sai.max_frozen_term_size_kb", "5"),
+SAI_MAX_STRING_TERM_SIZE("cassandra.sai.max_string_term_size_kb", "1"),
+SAI_TEST_DISABLE_TIMEOUT("cassandra.sai.test.disable.timeout", "false"),
-/** Defines how often schema definitions are pulled from the other nodes */
SCHEMA_PULL_INTERVAL_MS("cassandra.schema_pull_interval_ms", "6"),
SCHEMA_UPDATE_HANDLER_FACTORY_CLASS("cassandra.schema.update_handler_factory.class"),
SEARCH_CONCURRENCY_FACTOR("cassandra.search_concurrency_factor", "1"),
@@ -486,6 +489,7 @@ public enum CassandraRelevantProperties
TEST_DEBUG_REF_COUNT("cassandra.debugrefcount"),
TEST_DRIVER_CONNECTION_TIMEOUT_MS("cassandra.test.driver.connection_timeout_ms",
"5000"),
TEST_DRIVER_READ_TIMEOUT_MS("cassandra.test.driver.read_timeout_ms",
"12000"),
+TEST_ENCRYPTION("cassandra.test.encryption", "false"),
TEST_FAIL_MV_LOCKS_COUNT("cassandra.test.fail_mv_locks_count", "0"),
TEST_FAIL_WRITES_KS("cassandra.test.fail_writes_ks", ""),
/** Flush changes of {@link org.apache.cassandra.schema.SchemaKeyspace}
after each schema modification. In production,
@@ -498,6 +502,7 @@ public enum CassandraRelevantProperties
TEST_JVM_DTEST_DISABLE_SSL("cassandra.test.disable_ssl"),
TEST_LEGACY_SSTABLE_ROOT("legacy-sstable-root"),
TEST_ORG_CAFFINITAS_OHC_SEGMENTCOUNT("org.caffinitas.ohc.segmentCount"),
+TEST_RANDOM_SEED("cassandra.test.random.seed"),
TEST_READ_ITERATION_DELAY_MS("cassandra.test.read_iteration_delay_ms",
"0"),
TEST_REUSE_PREPARED("cassandra.test.reuse_prepared", "true"),
TEST_ROW_CACHE_SIZE("cassandra.test.row_cache_size"),
diff --git a/src/java/org/apache/cassandra/index/IndexStatusManager.java
b/src/java/org/apache/cassandra/index/IndexStatusManager.java
index ae1abf18d8..13a51831f9 100644
--- a/src/java/org/apache/cassandra/index/IndexStatusManager.java
+++ b/src/java/org/apache/cassandra/index/IndexStatusManager.java
@@ -31,7 +31,6 @@ import com.google.common.annotations.VisibleForTesting;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.cassandra.concurrent.ExecutorPlus;
import org.apache.cassandra.db.ConsistencyLevel;
import org.apache.cassandra.db.Keyspace;
@@ -52,9 +51,9 @@ import static
org.apache.cassandra.concurrent.ExecutorFactory.Global.executorFac
/**
*
[cassandra] branch trunk updated: Disable the deprecated keyspace/table thresholds and convert them to guardrails
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/trunk by this push: new aac070681b Disable the deprecated keyspace/table thresholds and convert them to guardrails aac070681b is described below commit aac070681bd01ba796814692c3fba04de103d8b4 Author: Daniel Jatnieks AuthorDate: Mon Jun 26 16:20:53 2023 -0700 Disable the deprecated keyspace/table thresholds and convert them to guardrails patch by Daniel Jatnieks; reviewed by Andrés de la Peña and Brandon Williams for CASSANDRA-18617 --- CHANGES.txt| 1 + NEWS.txt | 2 +- conf/cassandra.yaml| 7 -- .../org/apache/cassandra/auth/AuthKeyspace.java| 4 + src/java/org/apache/cassandra/config/Config.java | 7 +- .../org/apache/cassandra/config/Converters.java| 10 +- .../cassandra/config/DatabaseDescriptor.java | 24 - .../statements/schema/CreateKeyspaceStatement.java | 17 .../statements/schema/CreateTableStatement.java| 17 .../org/apache/cassandra/db/SystemKeyspace.java| 6 ++ .../apache/cassandra/schema/SchemaConstants.java | 27 ++ .../schema/SystemDistributedKeyspace.java | 3 + .../apache/cassandra/service/StorageService.java | 11 ++- .../apache/cassandra/tracing/TraceKeyspace.java| 3 + .../cql3/SystemKeyspaceTablesNamesTest.java| 103 + .../db/guardrails/GuardrailKeyspacesTest.java | 35 +++ .../db/guardrails/GuardrailTablesTest.java | 25 - .../schema/CreateTableValidationTest.java | 57 18 files changed, 222 insertions(+), 137 deletions(-) diff --git a/CHANGES.txt b/CHANGES.txt index 8198ece74c..996c938f00 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 5.0 + * Disable the deprecated keyspace/table thresholds and convert them to guardrails (CASSANDRA-18617) * Deprecate CloudstackSnitch and remove duplicate code in snitches (CASSANDRA-18438) * Add support for vectors in UDFs (CASSANDRA-18613) * Improve vector value validation errors (CASSANDRA-18652) diff --git a/NEWS.txt b/NEWS.txt index db77d36eba..195cefc25a 100644 --- a/NEWS.txt +++ b/NEWS.txt @@ -439,7 +439,7 @@ Deprecation No change of functionality in the new one, only name change for clarity in regards to units and to follow naming standartization. - The properties `keyspace_count_warn_threshold` and `table_count_warn_threshold` in cassandra.yaml have been - deprecated in favour of the new `guardrails.keyspaces` and `guardrails.tables` properties and will be removed + deprecated in favour of the new `keyspaces_warn_threshold` and `tables_warn_threshold` properties and will be removed in a subsequent major version. This also affects the setters and getters for those properties in the JMX MBean `org.apache.cassandra.db:type=StorageService`, which are equally deprecated in favour of the analogous methods in the JMX MBean `org.apache.cassandra.db:type=Guardrails`. See CASSANDRA-17195 for further details. diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml index 942812a7e1..92e092af75 100644 --- a/conf/cassandra.yaml +++ b/conf/cassandra.yaml @@ -1685,13 +1685,6 @@ repaired_data_tracking_for_partition_reads_enabled: false # mismatches are less actionable than confirmed ones. report_unconfirmed_repaired_data_mismatches: false -# Having many tables and/or keyspaces negatively affects performance of many operations in the -# cluster. When the number of tables/keyspaces in the cluster exceeds the following thresholds -# a client warning will be sent back to the user when creating a table or keyspace. -# As of cassandra 4.1, these properties are deprecated in favor of keyspaces_warn_threshold and tables_warn_threshold -# table_count_warn_threshold: 150 -# keyspace_count_warn_threshold: 40 - # configure the read and write consistency levels for modifications to auth tables # auth_read_consistency_level: LOCAL_QUORUM # auth_write_consistency_level: EACH_QUORUM diff --git a/src/java/org/apache/cassandra/auth/AuthKeyspace.java b/src/java/org/apache/cassandra/auth/AuthKeyspace.java index f0a3da891c..75d9871d03 100644 --- a/src/java/org/apache/cassandra/auth/AuthKeyspace.java +++ b/src/java/org/apache/cassandra/auth/AuthKeyspace.java @@ -17,8 +17,11 @@ */ package org.apache.cassandra.auth; +import java.util.Set; import java.util.concurrent.TimeUnit; +import com.google.common.collect.ImmutableSet; + import org.apache.cassandra.config.CassandraRelevantProperties; import org.apache.cassandra.cql3.statements.schema.CreateTableStatement; import org.apache.cassandra.config.DatabaseDescriptor; @@ -55,6 +58,7 @@ public final class AuthKeyspace
[cassandra] branch trunk updated: Add support for vectors in UDFs
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new ddbc52990f Add support for vectors in UDFs
ddbc52990f is described below
commit ddbc52990f90473db729e96f22d2914e51a957a6
Author: Andrés de la Peña
AuthorDate: Thu Jun 22 19:33:25 2023 +0100
Add support for vectors in UDFs
patch by Andrés de la Peña; reviewed by Berenguer Blasi and Maxwell Guo for
CASSANDRA-18613
---
CHANGES.txt| 1 +
.../types/AbstractAddressableByIndexData.java | 6 +
.../cql3/functions/types/AbstractData.java | 16 ++
.../types/AbstractGettableByIndexData.java | 22 +++
.../cql3/functions/types/AbstractGettableData.java | 18 ++
.../cql3/functions/types/CodecRegistry.java| 69 ---
.../cassandra/cql3/functions/types/DataType.java | 15 +-
.../functions/types/DataTypeClassNameParser.java | 14 ++
.../cql3/functions/types/GettableByIndexData.java | 44 +
.../cql3/functions/types/GettableByNameData.java | 44 +
.../cql3/functions/types/SettableByIndexData.java | 14 ++
.../cql3/functions/types/SettableByNameData.java | 14 ++
.../cassandra/cql3/functions/types/TypeCodec.java | 80
.../cassandra/cql3/functions/types/TypeTokens.java | 19 ++
.../cql3/functions/types/VectorCodec.java | 204 +
.../cassandra/cql3/functions/types/VectorType.java | 88 +
.../cassandra/cql3/selection/Selectable.java | 3 +-
.../apache/cassandra/utils/JavaDriverUtils.java| 12 --
.../cql3/validation/entities/UFSecurityTest.java | 8 +-
.../cql3/validation/operations/CQLVectorTest.java | 173 +++--
20 files changed, 807 insertions(+), 57 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index d79d78bad2..caea458186 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
5.0
+ * Add support for vectors in UDFs (CASSANDRA-18613)
* Improve vector value validation errors (CASSANDRA-18652)
* Upgrade Guava to 32.0.1 (CASSANDRA-18645)
* Add duration and count of partition keys to sstablemetadata
(CASSANDRA-18639)
diff --git
a/src/java/org/apache/cassandra/cql3/functions/types/AbstractAddressableByIndexData.java
b/src/java/org/apache/cassandra/cql3/functions/types/AbstractAddressableByIndexData.java
index 0e98c8948f..2b1fd84a9e 100644
---
a/src/java/org/apache/cassandra/cql3/functions/types/AbstractAddressableByIndexData.java
+++
b/src/java/org/apache/cassandra/cql3/functions/types/AbstractAddressableByIndexData.java
@@ -253,6 +253,12 @@ extends AbstractGettableByIndexData implements
SettableByIndexData
return setValue(i, codecFor(i,
TypeTokens.setOf(elementsType)).serialize(v, protocolVersion));
}
+@Override
+public T setVector(int i, List v)
+{
+return setValue(i, codecFor(i).serialize(v, protocolVersion));
+}
+
@Override
public T setUDTValue(int i, UDTValue v)
{
diff --git
a/src/java/org/apache/cassandra/cql3/functions/types/AbstractData.java
b/src/java/org/apache/cassandra/cql3/functions/types/AbstractData.java
index adb7c0e0bd..45ce309332 100644
--- a/src/java/org/apache/cassandra/cql3/functions/types/AbstractData.java
+++ b/src/java/org/apache/cassandra/cql3/functions/types/AbstractData.java
@@ -540,6 +540,22 @@ implements SettableData
return wrapped;
}
+@Override
+public T setVector(int i, List v)
+{
+return setValue(i, codecFor(i).serialize(v, protocolVersion));
+}
+
+@Override
+public T setVector(String name, List v)
+{
+for (int i : getAllIndexesOf(name))
+{
+setVector(i, v);
+}
+return wrapped;
+}
+
@Override
public T setUDTValue(int i, UDTValue v)
{
diff --git
a/src/java/org/apache/cassandra/cql3/functions/types/AbstractGettableByIndexData.java
b/src/java/org/apache/cassandra/cql3/functions/types/AbstractGettableByIndexData.java
index 1552309edb..ca8827b0d7 100644
---
a/src/java/org/apache/cassandra/cql3/functions/types/AbstractGettableByIndexData.java
+++
b/src/java/org/apache/cassandra/cql3/functions/types/AbstractGettableByIndexData.java
@@ -365,6 +365,28 @@ abstract class AbstractGettableByIndexData implements
GettableByIndexData
return codecFor(i, javaType).deserialize(value, protocolVersion);
}
+/**
+ * {@inheritDoc}
+ */
+@Override
+@SuppressWarnings("unchecked")
+public List getVector(int i, Class elementsClass)
+{
+return getList(i, TypeToken.of(elementsClass));
+}
+
+/**
+ * {@inheritDoc}
+ */
+@Override
+@SuppressWarnings("unchecked")
+public List getVector(int i, TypeToken elementsType)
+{
+ByteBuffer value = getValue(i);
+
[cassandra] branch trunk updated: Improve vector value validation errors
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new c95e5a5567 Improve vector value validation errors
c95e5a5567 is described below
commit c95e5a556709cf131ba074680585106559ff9511
Author: Andrés de la Peña
AuthorDate: Thu Jul 6 13:01:55 2023 +0100
Improve vector value validation errors
patch by Andrés de la Peña; reviewed by Ekaterina Dimitrova and Maxwell Guo
for CASSANDRA-18652
---
CHANGES.txt| 1 +
.../apache/cassandra/db/marshal/VectorType.java| 15 +++-
.../cql3/validation/operations/CQLVectorTest.java | 90 ++
3 files changed, 103 insertions(+), 3 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index bcfdcfee58..e3c591378b 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
5.0
+ * Improve vector value validation errors (CASSANDRA-18652)
* Upgrade Guava to 32.0.1 (CASSANDRA-18645)
* Add duration and count of partition keys to sstablemetadata
(CASSANDRA-18639)
* Remove deprecated compaction_tombstone_warning_threshold and
compaction_large_partition_warning_threshold from yaml (CASSANDRA-18626)
diff --git a/src/java/org/apache/cassandra/db/marshal/VectorType.java
b/src/java/org/apache/cassandra/db/marshal/VectorType.java
index a184be6bb6..631d990c6c 100644
--- a/src/java/org/apache/cassandra/db/marshal/VectorType.java
+++ b/src/java/org/apache/cassandra/db/marshal/VectorType.java
@@ -352,10 +352,11 @@ public final class VectorType extends
AbstractType>
}
}
-private static void checkConsumedFully(V buffer, ValueAccessor
accessor, int offset)
+private void checkConsumedFully(V buffer, ValueAccessor accessor,
int offset)
{
-if (!accessor.isEmptyFromOffset(buffer, offset))
-throw new MarshalException("Unexpected extraneous bytes after
vector value");
+int remaining = accessor.sizeFromOffset(buffer, offset);
+if (remaining > 0)
+throw new MarshalException("Unexpected " + remaining + "
extraneous bytes after " + asCQL3Type() + " value");
}
public abstract class VectorSerializer extends TypeSerializer>
@@ -494,6 +495,11 @@ public final class VectorType extends
AbstractType>
return;
int offset = 0;
int elementSize = elementType.valueLengthIfFixed();
+
+int expectedSize = elementSize * dimension;
+if (accessor.size(input) < expectedSize)
+throw new MarshalException("Not enough bytes to read a " +
asCQL3Type());
+
for (int i = 0; i < dimension; i++)
{
V bb = accessor.slice(input, offset, elementSize);
@@ -628,6 +634,9 @@ public final class VectorType extends
AbstractType>
int offset = 0;
for (int i = 0; i < dimension; i++)
{
+if (offset >= accessor.size(input))
+throw new MarshalException("Not enough bytes to read a " +
asCQL3Type());
+
V bb = readValue(input, accessor, offset);
offset += sizeOf(bb, accessor);
elementSerializer.validate(bb, accessor);
diff --git
a/test/unit/org/apache/cassandra/cql3/validation/operations/CQLVectorTest.java
b/test/unit/org/apache/cassandra/cql3/validation/operations/CQLVectorTest.java
index c1025659ca..ab39feb952 100644
---
a/test/unit/org/apache/cassandra/cql3/validation/operations/CQLVectorTest.java
+++
b/test/unit/org/apache/cassandra/cql3/validation/operations/CQLVectorTest.java
@@ -18,6 +18,7 @@
package org.apache.cassandra.cql3.validation.operations;
+import java.math.BigInteger;
import java.nio.ByteBuffer;
import org.junit.Assert;
@@ -139,6 +140,95 @@ public class CQLVectorTest extends CQLTester.InMemory
test.run();
}
+@Test
+public void invalidNumberOfDimensionsFixedWidth() throws Throwable
+{
+createTable("CREATE TABLE %s (pk int primary key, value vector)");
+
+// fewer values than expected, with literals and bind markers
+assertInvalidThrowMessage("Invalid vector literal for value of type
vector; expected 2 elements, but given 1",
+ InvalidRequestException.class,
+ "INSERT INTO %s (pk, value) VALUES (0,
[1])");
+assertInvalidThrowMessage("Not enough bytes to read a vector",
+ InvalidRequestException.class,
+ "INSERT INTO %s (pk, value) VALUES (0, ?)",
vector(1));
+
+// more values than expected, with literal
[cassandra-dtest] branch trunk updated: Align test_large_compaction_warning with CASSANDRA-18626
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra-dtest.git
The following commit(s) were added to refs/heads/trunk by this push:
new 862eeb79 Align test_large_compaction_warning with CASSANDRA-18626
862eeb79 is described below
commit 862eeb7980734a2f5e6189cab58592e565b60030
Author: Andrés de la Peña
AuthorDate: Wed Jun 28 11:08:47 2023 +0100
Align test_large_compaction_warning with CASSANDRA-18626
patch by Andrés de la Peña; reviewed by Brandon Williams for CASSANDRA-18626
---
compaction_test.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/compaction_test.py b/compaction_test.py
index abf64d4e..f7b27078 100644
--- a/compaction_test.py
+++ b/compaction_test.py
@@ -13,6 +13,7 @@ from dtest import Tester, create_ks
from tools.assertions import assert_length_equal, assert_none, assert_one
since = pytest.mark.since
+ported_to_in_jvm = pytest.mark.ported_to_in_jvm
logger = logging.getLogger(__name__)
strategies = ['LeveledCompactionStrategy', 'SizeTieredCompactionStrategy',
'DateTieredCompactionStrategy']
@@ -346,6 +347,7 @@ class TestCompaction(Tester):
time.sleep(5)
cluster.start()
+@ported_to_in_jvm('5.0') #
org.apache.cassandra.distributed.test.guardrails.GuardrailPartitionSizeTest
def test_large_compaction_warning(self):
"""
@jira_ticket CASSANDRA-9643
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Remove deprecated compaction_tombstone_warning_threshold and compaction_large_partition_warning_threshold from yaml
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new c579faa488 Remove deprecated compaction_tombstone_warning_threshold
and compaction_large_partition_warning_threshold from yaml
c579faa488 is described below
commit c579faa488ec156a59ed8e15dd6db55759b9c942
Author: Andrés de la Peña
AuthorDate: Tue Jun 27 16:58:16 2023 +0100
Remove deprecated compaction_tombstone_warning_threshold and
compaction_large_partition_warning_threshold from yaml
patch by Andrés de la Peña; reviewed by Brandon Williams for CASSANDRA-18626
---
CHANGES.txt| 1 +
NEWS.txt | 2 ++
conf/cassandra.yaml| 8
src/java/org/apache/cassandra/config/Config.java | 8 +++-
.../org/apache/cassandra/config/Converters.java| 6 ++
.../apache/cassandra/config/DataStorageSpec.java | 8
.../cassandra/config/DatabaseDescriptor.java | 18 -
.../io/sstable/format/SortedTableWriter.java | 23 --
.../apache/cassandra/service/StorageService.java | 5 +++--
.../distributed/test/TombstoneWarningTest.java | 3 ++-
.../guardrails/GuardrailPartitionSizeTest.java | 1 -
.../cassandra/config/DatabaseDescriptorTest.java | 3 ---
.../cassandra/config/ParseAndConvertUnitsTest.java | 1 -
.../config/YamlConfigurationLoaderTest.java| 21
14 files changed, 46 insertions(+), 62 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index cf87114f26..fc2d85f035 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
5.0
+ * Remove deprecated compaction_tombstone_warning_threshold and
compaction_large_partition_warning_threshold from yaml (CASSANDRA-18626)
* Enhance nodetool compactionstats with additional metrics (CASSANDRA-18305)
* Added support for type VECTOR (CASSANDRA-18504)
* Expose bootstrap and decommission state to nodetool info (CASSANDRA-18555)
diff --git a/NEWS.txt b/NEWS.txt
index 1ecd60ab29..0faea57dfa 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -218,9 +218,11 @@ Deprecation
- The config property `compaction_large_partition_warning_threshold` has
been deprecated in favour of the new
guardrail for partition size. That guardrail is based on the properties
`partition_size_warn_threshold` and
`partition_size_fail_threshold`. The warn threshold has a very similar
behaviour to the old config property.
+ The old property is still supported for backward compatibility, but now
it is disabled by default.
- The config property `compaction_tombstone_warning_threshold` has been
deprecated in favour of the new guardrail
for partition tombstones. That guardrail is based on the properties
`partition_tombstones_warn_threshold` and
`partition_tombstones_fail_threshold`. The warn threshold has a very
similar behaviour to the old config property.
+ The old property is still supported for backward compatibility, but now
it is disabled by default.
4.1
===
diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml
index dc884e2df9..b8aa59e2ca 100644
--- a/conf/cassandra.yaml
+++ b/conf/cassandra.yaml
@@ -1560,14 +1560,6 @@ batch_size_fail_threshold: 50KiB
# Log WARN on any batches not of type LOGGED than span across more partitions
than this limit
unlogged_batch_across_partitions_warn_threshold: 10
-# Log a warning when compacting partitions larger than this value.
-# As of Cassandra 5.0, this property is deprecated in favour of
partition_size_warn_threshold.
-compaction_large_partition_warning_threshold: 100MiB
-
-# Log a warning when writing more tombstones than this value to a partition.
-# As of Cassandra 5.0, this property is deprecated in favour of
partition_tombstones_warn_threshold.
-compaction_tombstone_warning_threshold: 10
-
# GC Pauses greater than 200 ms will be logged at INFO level
# This threshold can be adjusted to minimize logging if necessary
# Min unit: ms
diff --git a/src/java/org/apache/cassandra/config/Config.java
b/src/java/org/apache/cassandra/config/Config.java
index 37dce34d89..5651c4683a 100644
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@ -328,13 +328,8 @@ public class Config
public volatile Integer concurrent_compactors;
@Replaces(oldName = "compaction_throughput_mb_per_sec", converter =
Converters.MEBIBYTES_PER_SECOND_DATA_RATE, deprecated = true)
public volatile DataRateSpec.LongBytesPerSecondBound compaction_throughput
= new DataRateSpec.LongBytesPerSecondBound("64MiB/s");
-@Deprecated
-@Replaces(oldName = "compaction_large_partition_warning_threshold_mb",
converter = Converters.MEBIBYTES_DATA_
[cassandra] branch trunk updated: Fix SSTabledump errors when dumping data from index
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new cc3e742c73 Fix SSTabledump errors when dumping data from index
cc3e742c73 is described below
commit cc3e742c735daddc29d2fb90aac638c641463d39
Author: maxwellguo
AuthorDate: Mon Oct 24 21:53:50 2022 +0800
Fix SSTabledump errors when dumping data from index
patch by Maxwell Guo; reviewed by Andrés de la Peña and Branimir Lambov for
CASSANDRA-17698
---
CHANGES.txt| 1 +
.../db/marshal/AbstractCompositeType.java | 1 +
.../db/marshal/PartitionerDefinedOrder.java| 63 +--
.../apache/cassandra/db/marshal/TypeParser.java| 63 ++-
.../org/apache/cassandra/dht/IPartitioner.java | 13 ++
.../apache/cassandra/dht/Murmur3Partitioner.java | 7 +-
.../apache/cassandra/dht/RandomPartitioner.java| 7 +-
.../cassandra/index/internal/CassandraIndex.java | 10 +-
.../index/internal/CassandraIndexFunctions.java| 58 ++-
.../apache/cassandra/tools/JsonTransformer.java| 12 +-
.../org/apache/cassandra/tools/SSTableExport.java | 4 +-
src/java/org/apache/cassandra/tools/Util.java | 12 ++
.../cql3/SecondaryIndexSSTableExportTest.java | 181 +
.../db/marshal/PartitionerDefinedOrderTest.java| 59 +++
.../cassandra/db/marshal/TypeParserTest.java | 132 ++-
.../apache/cassandra/dht/LengthPartitioner.java| 5 +
.../apache/cassandra/utils/FBUtilitiesTest.java| 2 +-
.../bytecomparable/AbstractTypeByteSourceTest.java | 2 +-
18 files changed, 592 insertions(+), 40 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 9fa2bafff2..8014ca6f70 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
5.0
+ * Fix SSTabledump errors when dumping data from index (CASSANDRA-17698)
* Avoid unnecessary deserialization of terminal arguments when executing CQL
functions (CASSANDRA-18566)
* Remove dependency on pytz library for setting CQLSH timezones on Python
version >= 3.9 (CASSANDRA-17433)
* Extend maximum expiration date (CASSANDRA-14227)
diff --git
a/src/java/org/apache/cassandra/db/marshal/AbstractCompositeType.java
b/src/java/org/apache/cassandra/db/marshal/AbstractCompositeType.java
index 18007fb711..f120349785 100644
--- a/src/java/org/apache/cassandra/db/marshal/AbstractCompositeType.java
+++ b/src/java/org/apache/cassandra/db/marshal/AbstractCompositeType.java
@@ -266,6 +266,7 @@ public abstract class AbstractCompositeType extends
AbstractType
@Override
public String toJSONString(ByteBuffer buffer, ProtocolVersion
protocolVersion)
{
+// TODO: suport toJSONString (CASSANDRA-18177)
throw new UnsupportedOperationException();
}
diff --git
a/src/java/org/apache/cassandra/db/marshal/PartitionerDefinedOrder.java
b/src/java/org/apache/cassandra/db/marshal/PartitionerDefinedOrder.java
index be4b354ad0..57786edbb8 100644
--- a/src/java/org/apache/cassandra/db/marshal/PartitionerDefinedOrder.java
+++ b/src/java/org/apache/cassandra/db/marshal/PartitionerDefinedOrder.java
@@ -18,46 +18,54 @@
package org.apache.cassandra.db.marshal;
import java.nio.ByteBuffer;
-import java.util.Iterator;
+import java.util.Objects;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.cql3.Term;
import org.apache.cassandra.cql3.functions.ArgumentDeserializer;
import org.apache.cassandra.db.DecoratedKey;
import org.apache.cassandra.db.PartitionPosition;
-import org.apache.cassandra.serializers.TypeSerializer;
-import org.apache.cassandra.serializers.MarshalException;
import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.serializers.MarshalException;
+import org.apache.cassandra.serializers.TypeSerializer;
import org.apache.cassandra.transport.ProtocolVersion;
import org.apache.cassandra.utils.bytecomparable.ByteComparable;
import org.apache.cassandra.utils.bytecomparable.ByteComparable.Version;
import org.apache.cassandra.utils.bytecomparable.ByteSource;
-import org.apache.cassandra.utils.FBUtilities;
+
+import javax.annotation.Nullable;
/** for sorting columns representing row keys in the row ordering as
determined by a partitioner.
* Not intended for user-defined CFs, and will in fact error out if used with
such. */
public class PartitionerDefinedOrder extends AbstractType
{
private final IPartitioner partitioner;
-
+private final AbstractType partitionKeyType;
+
public PartitionerDefinedOrder(IPartitioner partitioner)
{
super(ComparisonType.CUSTOM);
this.partitioner = partitioner;
+this.partitionKeyType = null;
+}
+
+public PartitionerDefinedOrder(IPartitioner partitioner, AbstractType
partitionKeyType)
+{
+su
[cassandra] branch trunk updated: CircleCI: Add separate approval step for oa utests on separate workflows
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/trunk by this push: new 83a3316246 CircleCI: Add separate approval step for oa utests on separate workflows 83a3316246 is described below commit 83a331624674755ba90ad19bf206c351b5fcdff5 Author: Andrés de la Peña AuthorDate: Tue Jun 13 13:17:21 2023 +0100 CircleCI: Add separate approval step for oa utests on separate workflows patch by Andrés de la Peña; reviewed by Berenguer Blasi for CASSANDRA-18593 --- .circleci/config.yml | 24 +--- .circleci/config.yml.FREE | 24 +--- .circleci/config.yml.PAID | 24 +--- .circleci/config_11_and_17.yml | 24 +--- .circleci/config_11_and_17.yml.FREE| 24 +--- .circleci/config_11_and_17.yml.PAID| 24 +--- .circleci/config_template.yml | 66 -- .circleci/config_template.yml.PAID.patch | 62 ++-- .circleci/config_template_11_and_17.yml| 66 -- .circleci/config_template_11_and_17.yml.PAID.patch | 62 ++-- 10 files changed, 224 insertions(+), 176 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index df8682ec52..57355ca59c 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -10336,10 +10336,6 @@ workflows: requires: - start_j8_unit_tests - j8_build -- j8_utests_oa: -requires: -- start_j8_unit_tests -- j8_build - start_j8_jvm_dtests: type: approval - j8_jvm_dtests: @@ -10400,9 +10396,17 @@ workflows: requires: - start_j11_unit_tests - j8_build +- start_j8_utests_oa: +type: approval +- j8_utests_oa: +requires: +- start_j8_utests_oa +- j8_build +- start_j11_utests_oa: +type: approval - j11_utests_oa: requires: -- start_j11_unit_tests +- start_j11_utests_oa - j8_build - start_j8_utests_long: type: approval @@ -10864,10 +10868,6 @@ workflows: requires: - start_j11_unit_tests - j11_build -- j11_utests_oa: -requires: -- start_j11_unit_tests -- j11_build - start_j11_jvm_dtests: type: approval - j11_jvm_dtests: @@ -10968,6 +10968,12 @@ workflows: requires: - start_j11_cqlsh-dtests-offheap - j11_build +- start_j11_utests_oa: +type: approval +- j11_utests_oa: +requires: +- start_j11_utests_oa +- j11_build - start_j11_utests_long: type: approval - j11_utests_long: diff --git a/.circleci/config.yml.FREE b/.circleci/config.yml.FREE index df8682ec52..57355ca59c 100644 --- a/.circleci/config.yml.FREE +++ b/.circleci/config.yml.FREE @@ -10336,10 +10336,6 @@ workflows: requires: - start_j8_unit_tests - j8_build -- j8_utests_oa: -requires: -- start_j8_unit_tests -- j8_build - start_j8_jvm_dtests: type: approval - j8_jvm_dtests: @@ -10400,9 +10396,17 @@ workflows: requires: - start_j11_unit_tests - j8_build +- start_j8_utests_oa: +type: approval +- j8_utests_oa: +requires: +- start_j8_utests_oa +- j8_build +- start_j11_utests_oa: +type: approval - j11_utests_oa: requires: -- start_j11_unit_tests +- start_j11_utests_oa - j8_build - start_j8_utests_long: type: approval @@ -10864,10 +10868,6 @@ workflows: requires: - start_j11_unit_tests - j11_build -- j11_utests_oa: -requires: -- start_j11_unit_tests -- j11_build - start_j11_jvm_dtests: type: approval - j11_jvm_dtests: @@ -10968,6 +10968,12 @@ workflows: requires: - start_j11_cqlsh-dtests-offheap - j11_build +- start_j11_utests_oa: +type: approval +- j11_utests_oa: +requires: +- start_j11_utests_oa +- j11_build - start_j11_utests_long: type: approval - j11_utests_long: diff --git a/.circleci/config.yml.PAID b/.circleci/config.yml.PAID index f100c2274e..b690e4def1 100644 --- a/.circleci/config.yml.PAID +++ b/.circleci/config.yml.PAID @@ -10336,10 +10336,6 @@ workflows: requires: - start_j8_unit_tests - j8_build -- j8_utests_oa: -requires: -- start_j8_unit_tests -- j8_build - start_j8_jvm_dtests: type: approval - j8_jvm_dtests: @@ -10400,9 +10396,17 @@ workflows: requires: - start_j11_unit_tests
[cassandra] branch trunk updated (984f519bd9 -> e4f31b73c2)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from 984f519bd9 Merge branch 'cassandra-4.1' into trunk new 8d3c30dde4 Validate the comparison git branch when generating CircleCI configs new 3855c8a194 Merge branch 'cassandra-3.0' into cassandra-3.11 new 2393bb16f3 Merge branch 'cassandra-3.11' into cassandra-4.0 new 2b2d9f9b72 Merge branch 'cassandra-4.0' into cassandra-4.1 new e4f31b73c2 Merge branch 'cassandra-4.1' into trunk The 5 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .circleci/generate.sh | 19 --- .circleci/generate_11_and_17.sh | 19 --- 2 files changed, 32 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-3.11' into cassandra-4.0
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch cassandra-4.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 2393bb16f308d4d51335a97851646973aceb8066 Merge: f368b9dc1e 3855c8a194 Author: Andrés de la Peña AuthorDate: Wed Jun 7 11:04:52 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 .circleci/generate.sh | 19 --- 1 file changed, 16 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.11 updated (03da864bab -> 3855c8a194)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch cassandra-3.11 in repository https://gitbox.apache.org/repos/asf/cassandra.git from 03da864bab Add keyspace and table name to exception message during ColumnSubselection deserialization new 8d3c30dde4 Validate the comparison git branch when generating CircleCI configs new 3855c8a194 Merge branch 'cassandra-3.0' into cassandra-3.11 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .circleci/generate.sh | 19 --- 1 file changed, 16 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.0 updated: Validate the comparison git branch when generating CircleCI configs
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-3.0 by this push:
new 8d3c30dde4 Validate the comparison git branch when generating CircleCI
configs
8d3c30dde4 is described below
commit 8d3c30dde4b615835073089594003ca40a730992
Author: Derek Chen-Becker
AuthorDate: Mon Nov 7 13:32:01 2022 -0700
Validate the comparison git branch when generating CircleCI configs
Allow the user to use a flag to specify a non-default comparison git
branch when detecting changed unit tests to repeat.
Patch by Derek Chen-Becker; reviewed by Andrés de la Peña and Berenguer
Blasi for CASSANDRA-17997
---
.circleci/generate.sh | 19 ---
1 file changed, 16 insertions(+), 3 deletions(-)
diff --git a/.circleci/generate.sh b/.circleci/generate.sh
index ee62a9b9da..40bdd205db 100755
--- a/.circleci/generate.sh
+++ b/.circleci/generate.sh
@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
@@ -30,12 +30,15 @@ die ()
print_help()
{
- echo "Usage: $0 [-f|-p|-a|-e|-i]"
+ echo "Usage: $0 [-f|-p|-a|-e|-i|-b]"
echo " -a Generate the config.yml, config.yml.FREE and config.yml.PAID
expanded configuration"
echo " files from the main config_template.yml reusable configuration
file."
echo " Use this for permanent changes in config that will be committed
to the main repo."
echo " -f Generate config.yml for tests compatible with the CircleCI free
tier resources"
echo " -p Generate config.yml for tests compatible with the CircleCI paid
tier resources"
+ echo " -b Specify the base git branch for comparison when determining
changed tests to"
+ echo " repeat. Defaults to ${BASE_BRANCH}. Note that this option is not
used when"
+ echo " the '-a' option is specified."
echo " -e Environment variables to be used in the generated
config.yml, e.g.:"
echo " -e DTEST_BRANCH=CASSANDRA-8272"
echo " -e
DTEST_REPO=https://github.com/adelapena/cassandra-dtest.git;
@@ -71,7 +74,7 @@ paid=false
env_vars=""
has_env_vars=false
check_env_vars=true
-while getopts "e:afpi" opt; do
+while getopts "e:afpib:" opt; do
case $opt in
a ) all=true
;;
@@ -79,6 +82,8 @@ while getopts "e:afpi" opt; do
;;
p ) paid=true
;;
+ b ) BASE_BRANCH="$OPTARG"
+ ;;
e ) if (!($has_env_vars)); then
env_vars="$OPTARG"
else
@@ -169,6 +174,14 @@ fi
# add new or modified tests to the sets of tests to be repeated
if (!($all)); then
+ # Sanity check that the referenced branch exists
+ if ! git show ${BASE_BRANCH} -- >&/dev/null; then
+echo -e "\n\nUnknown base branch: ${BASE_BRANCH}. Unable to detect changed
tests.\n"
+echo"Please use the '-b' option to choose an existing branch name"
+echo"(e.g. origin/${BASE_BRANCH}, apache/${BASE_BRANCH}, etc.)."
+exit 2
+ fi
+
add_diff_tests ()
{
dir="${BASEDIR}/../${2}"
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-4.1 updated (867c074dda -> 2b2d9f9b72)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch cassandra-4.1 in repository https://gitbox.apache.org/repos/asf/cassandra.git from 867c074dda Merge branch 'cassandra-4.0' into cassandra-4.1 new 8d3c30dde4 Validate the comparison git branch when generating CircleCI configs new 3855c8a194 Merge branch 'cassandra-3.0' into cassandra-3.11 new 2393bb16f3 Merge branch 'cassandra-3.11' into cassandra-4.0 new 2b2d9f9b72 Merge branch 'cassandra-4.0' into cassandra-4.1 The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .circleci/generate.sh | 18 +++--- 1 file changed, 15 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-4.1' into trunk
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit e4f31b73c21b04966269c5ac2d3bd2562e5f6c63
Merge: 984f519bd9 2b2d9f9b72
Author: Andrés de la Peña
AuthorDate: Wed Jun 7 11:07:20 2023 +0100
Merge branch 'cassandra-4.1' into trunk
.circleci/generate.sh | 19 ---
.circleci/generate_11_and_17.sh | 19 ---
2 files changed, 32 insertions(+), 6 deletions(-)
diff --cc .circleci/generate.sh
index 300660f384,2046362370..0c8f4b8223
--- a/.circleci/generate.sh
+++ b/.circleci/generate.sh
@@@ -93,6 -98,6 +96,8 @@@ while getopts "e:afpib:" opt; d
fi
has_env_vars=true
;;
++ b ) BASE_BRANCH="$OPTARG"
++ ;;
i ) check_env_vars=false
;;
\?) die "Invalid option: -$OPTARG"
@@@ -183,6 -188,13 +188,14 @@@ f
# add new or modified tests to the sets of tests to be repeated
if (! ($all)); then
+ # Sanity check that the referenced branch exists
+ if ! git show ${BASE_BRANCH} -- >&/dev/null; then
+ echo -e "\n\nUnknown base branch: ${BASE_BRANCH}. Unable to detect
changed tests.\n"
+ echo"Please use the '-b' option to choose an existing branch name"
+ echo"(e.g. origin/${BASE_BRANCH}, apache/${BASE_BRANCH}, etc.)."
+ exit 2
+ fi
++
add_diff_tests ()
{
dir="${BASEDIR}/../${2}"
diff --cc .circleci/generate_11_and_17.sh
index 029d4a11cb,00..c9c190c4ef
mode 100755,00..100755
--- a/.circleci/generate_11_and_17.sh
+++ b/.circleci/generate_11_and_17.sh
@@@ -1,308 -1,0 +1,321 @@@
- #!/bin/sh
++#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+BASEDIR=`dirname $0`
+BASE_BRANCH=trunk
+set -e
+
+die ()
+{
+ echo "ERROR: $*"
+ print_help
+ exit 1
+}
+
+print_help()
+{
- echo "Usage: $0 [-f|-p|-a|-e|-i]"
++ echo "Usage: $0 [-f|-p|-a|-e|-i|-b]"
+ echo " -a Generate the config_11_and_17.yml, config_11_and_17.yml.FREE
and config_11_and_17.yml.PAID expanded configuration"
+ echo " files from the main config_template.yml reusable configuration
file."
+ echo " Use this for permanent changes in config_11_and_17.yml that
will be committed to the main repo."
+ echo " -f Generate config.yml for tests compatible with the CircleCI free
tier resources"
+ echo " -p Generate config.yml for tests compatible with the CircleCI paid
tier resources"
++ echo " -b Specify the base git branch for comparison when determining
changed tests to"
++ echo " repeat. Defaults to ${BASE_BRANCH}. Note that this option is
not used when"
++ echo " the '-a' option is specified."
+ echo " -e Environment variables to be used in the generated
config_11_and_17.yml, e.g.:"
+ echo " -e DTEST_BRANCH=CASSANDRA-8272"
+ echo " -e
DTEST_REPO=https://github.com/adelapena/cassandra-dtest.git;
+ echo " -e REPEATED_TESTS_STOP_ON_FAILURE=false"
+ echo " -e
REPEATED_UTESTS=org.apache.cassandra.cql3.ViewTest#testCountersTable"
+ echo " -e REPEATED_UTESTS_COUNT=500"
+ echo " -e
REPEATED_UTESTS_FQLTOOL=org.apache.cassandra.fqltool.FQLCompareTest"
+ echo " -e REPEATED_UTESTS_FQLTOOL_COUNT=500"
+ echo " -e
REPEATED_UTESTS_LONG=org.apache.cassandra.db.commitlog.CommitLogStressTest"
+ echo " -e REPEATED_UTESTS_LONG_COUNT=100"
+ echo " -e
REPEATED_UTESTS_STRESS=org.apache.cassandra.stress.generate.DistributionGaussianTest"
+ echo " -e REPEATED_UTESTS_STRESS_COUNT=500"
+ echo " -e
REPEATED_SIMULATOR_DTESTS=org.apache.cassandra.simulator.test.TrivialSimulationTest"
+ e
[cassandra] branch cassandra-4.0 updated (f368b9dc1e -> 2393bb16f3)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch cassandra-4.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git from f368b9dc1e Merge branch 'cassandra-3.11' into cassandra-4.0 new 8d3c30dde4 Validate the comparison git branch when generating CircleCI configs new 3855c8a194 Merge branch 'cassandra-3.0' into cassandra-3.11 new 2393bb16f3 Merge branch 'cassandra-3.11' into cassandra-4.0 The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .circleci/generate.sh | 19 --- 1 file changed, 16 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-3.0' into cassandra-3.11
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch cassandra-3.11 in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 3855c8a194d1d9ac904429407cb9586a6178e2f0 Merge: 03da864bab 8d3c30dde4 Author: Andrés de la Peña AuthorDate: Wed Jun 7 11:04:01 2023 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 .circleci/generate.sh | 19 --- 1 file changed, 16 insertions(+), 3 deletions(-) diff --cc .circleci/generate.sh index 64d443e211,40bdd205db..f6203d8255 --- a/.circleci/generate.sh +++ b/.circleci/generate.sh @@@ -88,6 -91,6 +91,8 @@@ while getopts "e:afpib:" opt; d fi has_env_vars=true ;; ++ b ) BASE_BRANCH="$OPTARG" ++ ;; i ) check_env_vars=false ;; \?) die "Invalid option: -$OPTARG" - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-4.0' into cassandra-4.1
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cassandra-4.1
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 2b2d9f9b725546c01017d688211931779565f39d
Merge: 867c074dda 2393bb16f3
Author: Andrés de la Peña
AuthorDate: Wed Jun 7 11:06:41 2023 +0100
Merge branch 'cassandra-4.0' into cassandra-4.1
.circleci/generate.sh | 18 +++---
1 file changed, 15 insertions(+), 3 deletions(-)
diff --cc .circleci/generate.sh
index 29f66a2151,6ae3a5d4b6..2046362370
--- a/.circleci/generate.sh
+++ b/.circleci/generate.sh
@@@ -86,7 -86,7 +89,9 @@@ while getopts "e:afpib:" opt; d
;;
p ) paid=true
;;
- e ) if (!($has_env_vars)); then
++ b ) BASE_BRANCH="$OPTARG"
++ ;;
+ e ) if (! ($has_env_vars)); then
env_vars="$OPTARG"
else
env_vars="$env_vars|$OPTARG"
@@@ -182,7 -181,15 +187,14 @@@ elif (! ($has_env_vars)); the
fi
# add new or modified tests to the sets of tests to be repeated
-if (!($all)); then
+if (! ($all)); then
+ # Sanity check that the referenced branch exists
+ if ! git show ${BASE_BRANCH} -- >&/dev/null; then
+ echo -e "\n\nUnknown base branch: ${BASE_BRANCH}. Unable to detect
changed tests.\n"
+ echo"Please use the '-b' option to choose an existing branch name"
+ echo"(e.g. origin/${BASE_BRANCH}, apache/${BASE_BRANCH}, etc.)."
+ exit 2
+ fi
-
add_diff_tests ()
{
dir="${BASEDIR}/../${2}"
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Add guardrail for partition tombstones and deprecate compaction_tombstone_warning_threshold
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/trunk by this push: new b72da02611 Add guardrail for partition tombstones and deprecate compaction_tombstone_warning_threshold b72da02611 is described below commit b72da02611b62436100fa3fd66537c68461bcaf5 Author: Andrés de la Peña AuthorDate: Wed May 24 19:20:35 2023 +0100 Add guardrail for partition tombstones and deprecate compaction_tombstone_warning_threshold patch by Andrés de la Peña; reviewed by Berenguer Blasi and Maxwell Guo for CASSANDRA-17194 --- CHANGES.txt| 1 + NEWS.txt | 4 + conf/cassandra.yaml| 12 +- src/java/org/apache/cassandra/config/Config.java | 3 + .../cassandra/config/DatabaseDescriptor.java | 2 + .../apache/cassandra/config/GuardrailsOptions.java | 26 + .../apache/cassandra/db/guardrails/Guardrails.java | 32 +- .../cassandra/db/guardrails/GuardrailsConfig.java | 10 ++ .../cassandra/db/guardrails/GuardrailsMBean.java | 26 - .../io/sstable/format/SortedTableWriter.java | 22 ++-- .../apache/cassandra/service/StorageService.java | 2 + .../cassandra/service/StorageServiceMBean.java | 6 +- .../guardrails/GuardrailPartitionSizeTest.java | 7 +- java => GuardrailPartitionTombstonesTest.java} | 88 ++ .../GuardrailPartitionTombstonesTest.java | 127 + 15 files changed, 298 insertions(+), 70 deletions(-) diff --git a/CHANGES.txt b/CHANGES.txt index 99a53404b0..0882bbaa77 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 5.0 + * Add guardrail for partition tombstones and deprecate compaction_tombstone_warning_threshold (CASSANDRA-17194) * Print header and statistics for cassandra-stress output with arbitrary frequency (CASSANDRA-12972) * CEP-25: Trie-indexed SSTable format (CASSANDRA-18398) * Make cassandra-stress able to read all credentials from a file (CASSANDRA-18544) diff --git a/NEWS.txt b/NEWS.txt index 2129ead915..294e57d1f8 100644 --- a/NEWS.txt +++ b/NEWS.txt @@ -83,6 +83,7 @@ New features - Whether DROP KEYSPACE commands are allowed. - Column value size - Partition size + - Partition tombstones - It is possible to list ephemeral snapshots by nodetool listsnaphots command when flag "-e" is specified. - Added a new flag to `nodetool profileload` and JMX endpoint to set up recurring profile load generation on specified intervals (see CASSANDRA-17821) @@ -193,6 +194,9 @@ Deprecation - The config property `compaction_large_partition_warning_threshold` has been deprecated in favour of the new guardrail for partition size. That guardrail is based on the properties `partition_size_warn_threshold` and `partition_size_fail_threshold`. The warn threshold has a very similar behaviour to the old config property. +- The config property `compaction_tombstone_warning_threshold` has been deprecated in favour of the new guardrail + for partition tombstones. That guardrail is based on the properties `partition_tombstones_warn_threshold` and + `partition_tombstones_fail_threshold`. The warn threshold has a very similar behaviour to the old config property. 4.1 === diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml index ca865b19da..ce39c019ee 100644 --- a/conf/cassandra.yaml +++ b/conf/cassandra.yaml @@ -1564,7 +1564,8 @@ unlogged_batch_across_partitions_warn_threshold: 10 # As of Cassandra 5.0, this property is deprecated in favour of partition_size_warn_threshold. compaction_large_partition_warning_threshold: 100MiB -# Log a warning when writing more tombstones than this value to a partition +# Log a warning when writing more tombstones than this value to a partition. +# As of Cassandra 5.0, this property is deprecated in favour of partition_tombstones_warn_threshold. compaction_tombstone_warning_threshold: 10 # GC Pauses greater than 200 ms will be logged at INFO level @@ -1842,6 +1843,15 @@ drop_compact_storage_enabled: false # partition_size_warn_threshold: # partition_size_fail_threshold: # +# Guardrail to warn or fail when writing partitions with more tombstones than threshold. +# The guardrail is only checked when writing sstables (flush and compaction), and exceeding the fail threshold on that +# moment will only log an error message, without interrupting the operation. +# This operates on a per-sstable basis, so it won't detect a large partition if it is spread across multiple sstables. +# The warning threshold replaces the deprecated config property compaction_tombstone_warning_threshold. +# The two thresholds default to -1 to disable. +# partition_tombstones_warn_
[cassandra] branch trunk updated: Add guardrail for partition size and deprecate compaction_large_partition_warning_threshold
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/trunk by this push: new b0aa44b27d Add guardrail for partition size and deprecate compaction_large_partition_warning_threshold b0aa44b27d is described below commit b0aa44b27da97b37345ee6fafbee16d66f3b384f Author: Andrés de la Peña AuthorDate: Tue May 9 12:07:29 2023 +0100 Add guardrail for partition size and deprecate compaction_large_partition_warning_threshold patch by Andrés de la Peña; reviewed by Berenguer Blasi and Maxwell-Guo for CASSANDRA-18500 --- CHANGES.txt| 1 + NEWS.txt | 4 + conf/cassandra.yaml| 12 +- src/java/org/apache/cassandra/config/Config.java | 3 + .../cassandra/config/DatabaseDescriptor.java | 6 +- .../apache/cassandra/config/GuardrailsOptions.java | 28 .../apache/cassandra/db/guardrails/Guardrails.java | 34 - .../cassandra/db/guardrails/GuardrailsConfig.java | 12 ++ .../cassandra/db/guardrails/GuardrailsMBean.java | 27 .../io/sstable/format/SortedTableWriter.java | 14 ++ .../guardrails/GuardrailPartitionSizeTest.java | 154 + .../db/guardrails/GuardrailPartitionSizeTest.java | 123 12 files changed, 415 insertions(+), 3 deletions(-) diff --git a/CHANGES.txt b/CHANGES.txt index 0e3834c05f..168e5b2bed 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 5.0 + * Add guardrail for partition size and deprecate compaction_large_partition_warning_threshold (CASSANDRA-18500) * Add HISTORY command for CQLSH (CASSANDRA-15046) * Fix sstable formats configuration (CASSANDRA-18441) * Add guardrail to bound timestamps (CASSANDRA-18352) diff --git a/NEWS.txt b/NEWS.txt index 4e0792aee6..5c3267e1e0 100644 --- a/NEWS.txt +++ b/NEWS.txt @@ -78,6 +78,7 @@ New features - Maximum replication factor - Whether DROP KEYSPACE commands are allowed. - Column value size + - Partition size - It is possible to list ephemeral snapshots by nodetool listsnaphots command when flag "-e" is specified. - Added a new flag to `nodetool profileload` and JMX endpoint to set up recurring profile load generation on specified intervals (see CASSANDRA-17821) @@ -183,6 +184,9 @@ Deprecation - All native CQL functions names that don't use the snake case names are deprecated in favour of equivalent names using snake casing. Thus, `totimestamp` is deprecated in favour of `to_timestamp`, `intasblob` in favour of `int_as_blob`, `castAsInt` in favour of `cast_as_int`, etc. +- The config property `compaction_large_partition_warning_threshold` has been deprecated in favour of the new + guardrail for partition size. That guardrail is based on the properties `partition_size_warn_threshold` and + `partition_size_fail_threshold`. The warn threshold has a very similar behaviour to the old config property. 4.1 === diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml index 72fe17fa63..322b84a6cf 100644 --- a/conf/cassandra.yaml +++ b/conf/cassandra.yaml @@ -1541,7 +1541,8 @@ batch_size_fail_threshold: 50KiB # Log WARN on any batches not of type LOGGED than span across more partitions than this limit unlogged_batch_across_partitions_warn_threshold: 10 -# Log a warning when compacting partitions larger than this value +# Log a warning when compacting partitions larger than this value. +# As of Cassandra 5.0, this property is deprecated in favour of partition_size_warn_threshold. compaction_large_partition_warning_threshold: 100MiB # Log a warning when writing more tombstones than this value to a partition @@ -1813,6 +1814,15 @@ drop_compact_storage_enabled: false # write_consistency_levels_warned: [] # write_consistency_levels_disallowed: [] # +# Guardrail to warn or fail when writing partitions larger than threshold, expressed as 100MiB, 1GiB, etc. +# The guardrail is only checked when writing sstables (flush and compaction), and exceeding the fail threshold on that +# moment will only log an error message, without interrupting the operation. +# This operates on a per-sstable basis, so it won't detect a large partition if it is spread across multiple sstables. +# The warning threshold replaces the deprecated config property compaction_large_partition_warning_threshold. +# The two thresholds default to null to disable. +# partition_size_warn_threshold: +# partition_size_fail_threshold: +# # Guardrail to warn or fail when writing column values larger than threshold. # This guardrail is only applied to the values of regular columns because both the serialized partitions keys and the # values of the components of the clustering key already have a fixed, relati
[cassandra] branch cep-7-sai updated: Unify CQLTester#waitForIndex and SAITester#waitForIndexQueryable
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cep-7-sai
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cep-7-sai by this push:
new 9970cf5684 Unify CQLTester#waitForIndex and
SAITester#waitForIndexQueryable
9970cf5684 is described below
commit 9970cf56848f0f6176560c0cf0ce704c9b9a8c05
Author: Andrés de la Peña
AuthorDate: Thu May 11 15:01:00 2023 +0100
Unify CQLTester#waitForIndex and SAITester#waitForIndexQueryable
patch by Andrés de la Peña; reviewed by Caleb Rackliffe for CASSANDRA-18521
---
.../cassandra/cql3/functions/types/ParseUtils.java | 4 +-
test/unit/org/apache/cassandra/cql3/CQLTester.java | 240 +++--
.../org/apache/cassandra/cql3/CQLTesterTest.java | 82 +++
.../org/apache/cassandra/cql3/KeyCacheCqlTest.java | 4 +-
.../entities/SecondaryIndexOnMapEntriesTest.java | 2 +-
.../validation/entities/SecondaryIndexTest.java| 38 ++--
.../operations/InsertUpdateIfConditionTest.java| 2 -
.../db/guardrails/GuardrailTablesTest.java | 2 +-
.../cassandra/index/SecondaryIndexManagerTest.java | 127 +--
.../index/internal/CassandraIndexTest.java | 9 +-
.../org/apache/cassandra/index/sai/SAITester.java | 41 +---
.../index/sai/cql/AllowFilteringTest.java | 6 -
.../cassandra/index/sai/cql/BaseDataModel.java | 29 ++-
.../cassandra/index/sai/cql/BooleanTypeTest.java | 1 -
.../index/sai/cql/CollectionIndexingTest.java | 3 -
.../index/sai/cql/DecimalLargeValueTest.java | 2 -
.../index/sai/cql/DuplicateRowIDTest.java | 1 -
.../sai/cql/MixedIndexImplementationsTest.java | 4 -
.../index/sai/cql/MultipleColumnIndexTest.java | 2 -
.../index/sai/cql/SingleNodeExecutor.java | 4 +-
.../index/sai/cql/StorageAttachedIndexDDLTest.java | 36 ++--
.../index/sai/cql/types/IndexingTypeSupport.java | 2 -
.../cassandra/index/sai/disk/NodeStartupTest.java | 5 +-
.../index/sai/disk/SelectiveIntersectionTest.java | 2 -
.../index/sai/disk/SingleNodeQueryFailureTest.java | 1 -
.../index/sai/functional/CompactionTest.java | 2 -
.../index/sai/functional/DiskSpaceTest.java| 1 -
.../index/sai/functional/DropTableTest.java| 1 -
.../index/sai/functional/FailureTest.java | 2 +-
.../index/sai/functional/GroupComponentsTest.java | 3 -
.../index/sai/functional/NodeRestartTest.java | 9 +-
.../index/sai/functional/SnapshotTest.java | 2 -
.../index/sai/metrics/QueryMetricsTest.java| 1 -
.../sai/metrics/SegmentFlushingFailureTester.java | 1 -
.../index/sai/metrics/StateMetricsTest.java| 6 +-
.../cassandra/index/sai/plan/OperationTest.java| 2 +-
.../index/sai/virtual/IndexesSystemViewTest.java | 4 +-
.../index/sai/virtual/SSTablesSystemViewTest.java | 2 -
.../index/sai/virtual/SegmentsSystemViewTest.java | 1 -
39 files changed, 383 insertions(+), 303 deletions(-)
diff --git a/src/java/org/apache/cassandra/cql3/functions/types/ParseUtils.java
b/src/java/org/apache/cassandra/cql3/functions/types/ParseUtils.java
index 8972beed15..8d0f29b45b 100644
--- a/src/java/org/apache/cassandra/cql3/functions/types/ParseUtils.java
+++ b/src/java/org/apache/cassandra/cql3/functions/types/ParseUtils.java
@@ -292,7 +292,7 @@ public abstract class ParseUtils
* @param value The string to un-double quote.
* @return The un-double quoted string.
*/
-static String unDoubleQuote(String value)
+public static String unDoubleQuote(String value)
{
return unquote(value, '"');
}
@@ -482,7 +482,7 @@ public abstract class ParseUtils
* @return {@code true} if the given string is surrounded by the quote
character, and {@code
* false} otherwise.
*/
-private static boolean isQuoted(String value, char quoteChar)
+public static boolean isQuoted(String value, char quoteChar)
{
return value != null
&& value.length() > 1
diff --git a/test/unit/org/apache/cassandra/cql3/CQLTester.java
b/test/unit/org/apache/cassandra/cql3/CQLTester.java
index bf2f137165..f81cdbf38e 100644
--- a/test/unit/org/apache/cassandra/cql3/CQLTester.java
+++ b/test/unit/org/apache/cassandra/cql3/CQLTester.java
@@ -35,6 +35,7 @@ import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
+import javax.annotation.Nullable;
import javax.management.MBeanServerConnection;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
@@ -71,9 +72,11 @@ import org.apache.cassandra.concurrent.ScheduledExecutors;
import org.apache.cassandra.concurrent.Stage;
import org.apache.cassandra.config.DataStorageSpec;
import org.apache.cassandra.config.EncryptionOptions;
+import org.apach
[cassandra] branch trunk updated: Hotfix: add sstablepartitions to redhat/noboolean/cassandra.spec
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new af07ed334a Hotfix: add sstablepartitions to
redhat/noboolean/cassandra.spec
af07ed334a is described below
commit af07ed334ad9371c58fa743c7ce15afad7484949
Author: Andrés de la Peña
AuthorDate: Fri May 5 09:34:20 2023 +0100
Hotfix: add sstablepartitions to redhat/noboolean/cassandra.spec
patch by Andrés de la Peña; reviewed by Brandon Williams for CASSANDRA-8720
---
redhat/noboolean/cassandra.spec | 1 +
1 file changed, 1 insertion(+)
diff --git a/redhat/noboolean/cassandra.spec b/redhat/noboolean/cassandra.spec
index cc3d630346..2b6cdd4d6f 100644
--- a/redhat/noboolean/cassandra.spec
+++ b/redhat/noboolean/cassandra.spec
@@ -190,6 +190,7 @@ Cassandra is a distributed (peer-to-peer) system for the
management and storage
This package contains extra tools for working with Cassandra clusters.
%files tools
+%attr(755,root,root) %{_bindir}/sstablepartitions
%attr(755,root,root) %{_bindir}/sstabledump
%attr(755,root,root) %{_bindir}/compaction-stress
%attr(755,root,root) %{_bindir}/sstableexpiredblockers
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Add sstablepartitions offline tool to find large partitions in sstables
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/trunk by this push: new e72ec4e828 Add sstablepartitions offline tool to find large partitions in sstables e72ec4e828 is described below commit e72ec4e82835cef80cf9e9d8c58b25801d4af8c8 Author: Andrés de la Peña AuthorDate: Fri Apr 21 18:31:32 2023 +0100 Add sstablepartitions offline tool to find large partitions in sstables patch by Andrés de la Peña and Robert Stupp; reviewed by Brandon Williams and Stefan Miklosovic for CASSANDRA-8720 Co-authored-by: Andrés de la Peña Co-authored-by: Robert Stupp --- CHANGES.txt| 1 + NEWS.txt | 1 + debian/cassandra.install | 1 + .../pages/managing/tools/sstable/index.adoc| 1 + .../managing/tools/sstable/sstablepartitions.adoc | 171 redhat/cassandra.spec | 1 + .../apache/cassandra/io/sstable/Descriptor.java| 43 +- src/java/org/apache/cassandra/schema/TableId.java | 25 + .../apache/cassandra/tools/SSTablePartitions.java | 889 + .../cassandra/tools/SSTablePartitionsTest.java | 654 +++ tools/bin/sstablepartitions| 49 ++ 11 files changed, 1823 insertions(+), 13 deletions(-) diff --git a/CHANGES.txt b/CHANGES.txt index 862755286a..970058c239 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 5.0 + * Add sstablepartitions offline tool to find large partitions in sstables (CASSANDRA-8720) * Replace usages of json-simple dependency by Jackson (CASSANDRA-16855) * When decommissioning should set Severity to limit traffic (CASSANDRA-18430) * For Java11 and Java17 remove -XX:-UseBiasedLocking as it is the default already (CASSANDRA-17869) diff --git a/NEWS.txt b/NEWS.txt index b18a5e3a48..4e0792aee6 100644 --- a/NEWS.txt +++ b/NEWS.txt @@ -140,6 +140,7 @@ New features - Added new SELECT_MASKED permission. It allows to run SELECT queries selecting the clear values of masked columns. Superusers have it by default, whereas regular users don't have it by default. - Added support for using UDFs as masking functions attached to table columns on the schema. +- Added `sstablepartitions` offline tool to find large partitions in sstables. Upgrading - diff --git a/debian/cassandra.install b/debian/cassandra.install index f54d1ad835..dced5a29e2 100644 --- a/debian/cassandra.install +++ b/debian/cassandra.install @@ -25,6 +25,7 @@ tools/bin/fqltool usr/bin tools/bin/auditlogviewer usr/bin tools/bin/jmxtool usr/bin tools/bin/hash_password usr/bin +tools/bin/sstablepartitions usr/bin lib/*.jar usr/share/cassandra/lib lib/*.zip usr/share/cassandra/lib lib/sigar-bin/* usr/share/cassandra/lib/sigar-bin diff --git a/doc/modules/cassandra/pages/managing/tools/sstable/index.adoc b/doc/modules/cassandra/pages/managing/tools/sstable/index.adoc index cb787ece21..cc7637c8dd 100644 --- a/doc/modules/cassandra/pages/managing/tools/sstable/index.adoc +++ b/doc/modules/cassandra/pages/managing/tools/sstable/index.adoc @@ -12,6 +12,7 @@ stopped. * xref:tools/sstable/sstableloader.adoc[sstableloader] * xref:tools/sstable/sstablemetadata.adoc[sstablemetadata] * xref:tools/sstable/sstableofflinerelevel.adoc[sstableofflinerelevel] +* xref:tools/sstable/sstablepartitions.adoc[sstablepartitions] * xref:tools/sstable/sstablerepairedset.adoc[sstablerepairdset] * xref:tools/sstable/sstablescrub.adoc[sstablescrub] * xref:tools/sstable/sstablesplit.adoc[sstablesplit] diff --git a/doc/modules/cassandra/pages/managing/tools/sstable/sstablepartitions.adoc b/doc/modules/cassandra/pages/managing/tools/sstable/sstablepartitions.adoc new file mode 100644 index 00..6684946b47 --- /dev/null +++ b/doc/modules/cassandra/pages/managing/tools/sstable/sstablepartitions.adoc @@ -0,0 +1,171 @@ += sstablepartitions + +Identifies large partitions of SSTables and outputs the partition size in bytes, row count, cell count, and tombstone count. + +You can supply any number of sstables file paths, or directories containing sstables. Each sstable will be analyzed separately. + +If a metrics threshold such as `--min-size`, `--min-rows`, `--min-cells` or `--min-tombstones` is provided, +then the partition keys exceeding of the threshold will be printed in the output. +It also prints a summary of metrics for the table. The percentiles in the metrics are estimates, +while the min/max/count metrics are accurate. + +The default output of this tool is meant to be read by human eyes. +Future versions might include small formatting changes or present new data that can fool scripts reading it. +Scripts or other automatic tools should use the `--csv` flag to produce machine
[cassandra-dtest] branch trunk updated: Skip tests marked with @ported_to_in_jvm independently of vnodes since 4.1
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra-dtest.git
The following commit(s) were added to refs/heads/trunk by this push:
new 5715e36a Skip tests marked with @ported_to_in_jvm independently of
vnodes since 4.1
5715e36a is described below
commit 5715e36a71e261408dfd22f74f1f4b8df3983659
Author: Andrés de la Peña
AuthorDate: Tue Apr 18 11:48:28 2023 +0100
Skip tests marked with @ported_to_in_jvm independently of vnodes since 4.1
patch by Andrés de la Peña; reviewed by Brandon Williams for CASSANDRA-18396
---
conftest.py | 27 +--
1 file changed, 17 insertions(+), 10 deletions(-)
diff --git a/conftest.py b/conftest.py
index e07aeac6..dd4d00c2 100644
--- a/conftest.py
+++ b/conftest.py
@@ -474,24 +474,26 @@ def fixture_ported_to_in_jvm(request,
fixture_dtest_setup):
"""
Adds a new mark called 'ported_to_in_jvm' which denotes that a test was
ported to an in-JVM dtest.
-In-JVM dtests do not currently support running with vnodes, so tests that
use this annotation will
-still be run around those configurations.
+In-JVM dtests only support running with vnodes since 4.1, so tests that
use this annotation will
+still be run around those configurations if they are testing an older
branch.
"""
marker = request.node.get_closest_marker('ported_to_in_jvm')
-if marker and not request.config.getoption("--use-vnodes"):
+if marker:
-if not marker.args:
-pytest.skip("ported to in-jvm")
-
-from_str = marker.args[0]
+from_str = marker.args[0] if marker.args else "2.2.13" # JVM dtests
were introduced on 2.2.13
ported_from_version = LooseVersion(from_str)
+use_vnodes = request.config.getoption("--use-vnodes")
-# For upgrade tests don't run the test if any of the involved versions
-# are excluded by the annotation
+# For upgrade tests don't run the test if any of the involved versions
are excluded by the annotation
if hasattr(request.cls, "UPGRADE_PATH"):
+
+# JVM upgrade dtests don't support vnodes, so we can't skip them
+if use_vnodes:
+return
+
upgrade_path = request.cls.UPGRADE_PATH
if hasattr(upgrade_path, 'upgrade_meta'):
-skip_msg =
_skip_msg(LooseVersion(upgrade_path.upgrade_meta.family), since, max_version)
+skip_msg =
_skip_ported_msg(LooseVersion(upgrade_path.upgrade_meta.family),
ported_from_version)
if skip_msg:
pytest.skip(skip_msg)
ccm_repo_cache_dir, _ =
ccmlib.repository.setup(upgrade_path.starting_meta.version)
@@ -510,6 +512,11 @@ def fixture_ported_to_in_jvm(request, fixture_dtest_setup):
# Use cassandra_version_from_build as it's guaranteed to be a
LooseVersion
# whereas cassandra_version may be a string if set in the cli
options
current_running_version =
fixture_dtest_setup.dtest_config.cassandra_version_from_build
+
+# vnodes weren't supported nor tested before 4.1, so we can't skip
them if the version is older than that
+if use_vnodes and loose_version_compare(current_running_version,
LooseVersion('4.1')) < 0:
+return
+
skip_msg = _skip_ported_msg(current_running_version,
ported_from_version)
if skip_msg:
pytest.skip(skip_msg)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-4.1 updated (bd49f6ff26 -> 2208235ce9)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch cassandra-4.1 in repository https://gitbox.apache.org/repos/asf/cassandra.git from bd49f6ff26 Allow keystore and truststore passwords to be nullable new 08b9471a1e Fix sstable_count metric missing from tablestats json/yaml output new 3f67827387 Merge branch 'cassandra-3.11' into cassandra-4.0 new 2208235ce9 Merge branch 'cassandra-4.0' into cassandra-4.1 The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + .../tools/nodetool/stats/TableStatsHolder.java | 2 + .../cassandra/tools/nodetool/TableStatsTest.java | 29 + .../nodetool/stats/TableStatsPrinterTest.java | 135 - 4 files changed, 165 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (531b4cde43 -> 976b8395f9)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git from 531b4cde43 Merge branch 'cassandra-4.1' into trunk new 08b9471a1e Fix sstable_count metric missing from tablestats json/yaml output new 3f67827387 Merge branch 'cassandra-3.11' into cassandra-4.0 new 2208235ce9 Merge branch 'cassandra-4.0' into cassandra-4.1 new 976b8395f9 Merge branch 'cassandra-4.1' into trunk The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + .../tools/nodetool/stats/TableStatsHolder.java | 2 + .../cassandra/tools/nodetool/TableStatsTest.java | 29 + .../nodetool/stats/TableStatsPrinterTest.java | 141 - 4 files changed, 171 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-4.0' into cassandra-4.1
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cassandra-4.1
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 2208235ce9928bbf7916e6c37616abe29f939214
Merge: bd49f6ff26 3f67827387
Author: Andrés de la Peña
AuthorDate: Mon Apr 17 14:50:18 2023 +0100
Merge branch 'cassandra-4.0' into cassandra-4.1
CHANGES.txt| 1 +
.../tools/nodetool/stats/TableStatsHolder.java | 2 +
.../cassandra/tools/nodetool/TableStatsTest.java | 29 +
.../nodetool/stats/TableStatsPrinterTest.java | 135 -
4 files changed, 165 insertions(+), 2 deletions(-)
diff --cc CHANGES.txt
index 79d3e8b287,cf208d7eb9..852e4f9c91
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,14 -1,16 +1,15 @@@
-4.0.10
+4.1.2
+ * Allow keystore and trustrore passwords to be nullable (CASSANDRA-18124)
+ * Return snapshots with dots in their name in nodetool listsnapshots
(CASSANDRA-18371)
+ * Fix NPE when loading snapshots and data directory is one directory from
root (CASSANDRA-18359)
+ * Do not submit hints when hinted_handoff_enabled=false (CASSANDRA-18304)
+ * Fix COPY ... TO STDOUT behavior in cqlsh (CASSANDRA-18353)
+ * Remove six and Py2SaferScanner merge cruft (CASSANDRA-18354)
+Merged from 4.0:
* Fix nested selection of reversed collections (CASSANDRA-17913)
-
-4.0.9
* Update zstd-jni library to version 1.5.5 (CASSANDRA-18429)
- * Backport CASSANDRA-17205 to 4.0 branch - Remove self-reference in
SSTableTidier (CASSANDRA-18332)
- * Avoid loading the preferred IP for BulkLoader streaming (CASSANDRA-18370)
- * Fix BufferPool incorrect memoryInUse when putUnusedPortion is used
(CASSANDRA-18311)
- * Improve memtable allocator accounting when updating AtomicBTreePartition
(CASSANDRA-18125)
- * Update zstd-jni to version 1.5.4-1 (CASSANDRA-18259)
- * Split and order IDEA workspace template VM_PARAMETERS (CASSANDRA-18242)
Merged from 3.11:
+ * Fix sstable_count metric missing from tablestats json/yaml output
(CASSANDRA-18448)
* Suppress CVE-2022-45688 (CASSANDRA-18389)
* Fix Splitter sometimes creating more splits than requested
(CASSANDRA-18013)
Merged from 3.0:
diff --cc
src/java/org/apache/cassandra/tools/nodetool/stats/TableStatsHolder.java
index c6b2301c2c,3cd2570185..60132d1fed
--- a/src/java/org/apache/cassandra/tools/nodetool/stats/TableStatsHolder.java
+++ b/src/java/org/apache/cassandra/tools/nodetool/stats/TableStatsHolder.java
@@@ -119,8 -117,9 +119,10 @@@ public class TableStatsHolder implement
private Map convertStatsTableToMap(StatsTable table)
{
Map mpTable = new HashMap<>();
+ mpTable.put("sstable_count", table.sstableCount);
+ mpTable.put("old_sstable_count", table.oldSSTableCount);
mpTable.put("sstables_in_each_level", table.sstablesInEachLevel);
+mpTable.put("sstable_bytes_in_each_level",
table.sstableBytesInEachLevel);
mpTable.put("space_used_live", table.spaceUsedLive);
mpTable.put("space_used_total", table.spaceUsedTotal);
mpTable.put("space_used_by_snapshots_total",
table.spaceUsedBySnapshotsTotal);
diff --cc test/unit/org/apache/cassandra/tools/nodetool/TableStatsTest.java
index 1ba30f0b80,00..578bd88618
mode 100644,00..100644
--- a/test/unit/org/apache/cassandra/tools/nodetool/TableStatsTest.java
+++ b/test/unit/org/apache/cassandra/tools/nodetool/TableStatsTest.java
@@@ -1,226 -1,0 +1,255 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.tools.nodetool;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.commons.lang3.StringUtils;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
++import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.cassandra.cql3.CQLTester;
+import org.apache.cassandra.tools.ToolRunner;
++import org.yaml.snakeyaml.Yaml;
+
+import static org.assert
[cassandra] branch cassandra-4.0 updated (2fce3025c4 -> 3f67827387)
This is an automated email from the ASF dual-hosted git repository. adelapena pushed a change to branch cassandra-4.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git from 2fce3025c4 Fix nested selection of reversed collections new 08b9471a1e Fix sstable_count metric missing from tablestats json/yaml output new 3f67827387 Merge branch 'cassandra-3.11' into cassandra-4.0 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + .../tools/nodetool/stats/TableStatsHolder.java | 2 + .../nodetool/stats/NodetoolTableStatsTest.java | 31 + .../nodetool/stats/TableStatsPrinterTest.java | 129 - 4 files changed, 161 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-4.1' into trunk
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 976b8395f981c5956ef617ea90b483e8e15d576d
Merge: 531b4cde43 2208235ce9
Author: Andrés de la Peña
AuthorDate: Mon Apr 17 14:52:40 2023 +0100
Merge branch 'cassandra-4.1' into trunk
CHANGES.txt| 1 +
.../tools/nodetool/stats/TableStatsHolder.java | 2 +
.../cassandra/tools/nodetool/TableStatsTest.java | 29 +
.../nodetool/stats/TableStatsPrinterTest.java | 141 -
4 files changed, 171 insertions(+), 2 deletions(-)
diff --cc
src/java/org/apache/cassandra/tools/nodetool/stats/TableStatsHolder.java
index d9eece9afa,60132d1fed..80660611d2
--- a/src/java/org/apache/cassandra/tools/nodetool/stats/TableStatsHolder.java
+++ b/src/java/org/apache/cassandra/tools/nodetool/stats/TableStatsHolder.java
@@@ -117,10 -119,10 +117,12 @@@ public class TableStatsHolder implement
private Map convertStatsTableToMap(StatsTable table)
{
Map mpTable = new HashMap<>();
+ mpTable.put("sstable_count", table.sstableCount);
+ mpTable.put("old_sstable_count", table.oldSSTableCount);
mpTable.put("sstables_in_each_level", table.sstablesInEachLevel);
mpTable.put("sstable_bytes_in_each_level",
table.sstableBytesInEachLevel);
+mpTable.put("max_sstable_size", table.maxSSTableSize);
+mpTable.put("twcs", table.twcs);
mpTable.put("space_used_live", table.spaceUsedLive);
mpTable.put("space_used_total", table.spaceUsedTotal);
mpTable.put("space_used_by_snapshots_total",
table.spaceUsedBySnapshotsTotal);
diff --cc
test/unit/org/apache/cassandra/tools/nodetool/stats/TableStatsPrinterTest.java
index 0bf4e1ce28,81687ba7c0..3ac83850a1
---
a/test/unit/org/apache/cassandra/tools/nodetool/stats/TableStatsPrinterTest.java
+++
b/test/unit/org/apache/cassandra/tools/nodetool/stats/TableStatsPrinterTest.java
@@@ -394,6 -382,136 +395,142 @@@ public class TableStatsPrinterTest exte
}
}
+ @Test
+ public void testJsonPrinter() throws Exception
+ {
+ TestTableStatsHolder holder = new
TestTableStatsHolder(testKeyspaces.subList(2, 3), "", 0); // kesypace3
+ StatsPrinter printer = TableStatsPrinter.from("json",
false);
+ try (ByteArrayOutputStream byteStream = new ByteArrayOutputStream())
+ {
+ printer.print(holder, new PrintStream(byteStream));
+ Assertions.assertThat(byteStream.toString())
+ .isEqualTo("{\n" +
+ " \"keyspace3\" : {\n" +
+ "\"write_latency_ms\" : \"NaN\",\n" +
+ "\"tables\" : {\n" +
+ " \"table6\" : {\n" +
+ "
\"average_tombstones_per_slice_last_five_minutes\" : 6.0,\n" +
+ "\"top_tombstone_partitions\" :
null,\n" +
+ "
\"bloom_filter_off_heap_memory_used\" : \"667408\",\n" +
++ "\"twcs\" : null,\n" +
+ "\"bytes_pending_repair\" : 0,\n" +
+ "\"memtable_switch_count\" : 6,\n" +
+ "
\"maximum_tombstones_per_slice_last_five_minutes\" : 6,\n" +
+ "\"memtable_cell_count\" : ,\n" +
+ "\"memtable_data_size\" :
\"100\",\n" +
+ "
\"average_live_cells_per_slice_last_five_minutes\" : 5.0,\n" +
+ "\"local_read_latency_ms\" :
\"1.000\",\n" +
+ "\"sstable_count\" : 1000,\n" +
+ "\"local_write_latency_ms\" :
\"0.500\",\n" +
+ "\"pending_flushes\" : 66,\n" +
+ "
\"compacted_partition_minimum_bytes\" : 6,\n" +
+ "\"local_read_count\" : 5,\n" +
+ &quo
