from:"blambov"

(cassandra) branch trunk updated: Test fixes for CASSANDRA-18753

2024-03-12 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/trunk by this push:
 new 06558a8f56 Test fixes for CASSANDRA-18753
06558a8f56 is described below

commit 06558a8f56c8bbb98c276cce83599aae1a50adfc
Author: Branimir Lambov 
AuthorDate: Thu Mar 7 14:55:58 2024 +0200

Test fixes for CASSANDRA-18753

Additional fix for NPE when initializing CFS (CASSANDRA-19169)
Change BounceIndexRebuildTest to work with SAI as default index

patch by Branimir Lambov; reviewed by Sam Tunnicliffe for CASSANDRA-18753
---
 .../cassandra/db/compaction/UnifiedCompactionStrategy.java  |  4 +++-
 .../distributed/test/log/BounceIndexRebuildTest.java| 13 -
 2 files changed, 15 insertions(+), 2 deletions(-)

diff --git 
a/src/java/org/apache/cassandra/db/compaction/UnifiedCompactionStrategy.java 
b/src/java/org/apache/cassandra/db/compaction/UnifiedCompactionStrategy.java
index 1626b8c3c0..38cbf61f0e 100644
--- a/src/java/org/apache/cassandra/db/compaction/UnifiedCompactionStrategy.java
+++ b/src/java/org/apache/cassandra/db/compaction/UnifiedCompactionStrategy.java
@@ -297,7 +297,9 @@ public class UnifiedCompactionStrategy extends 
AbstractCompactionStrategy
 private void maybeUpdateShardManager()
 {
 // TODO - modify ShardManager::isOutOfDate to take an Epoch
-if (shardManager != null && 
!shardManager.isOutOfDate(ClusterMetadata.current().epoch.getEpoch()))
+if (shardManager != null
+&& (cfs.localRangesWeighted().ringVersion == 
ColumnFamilyStore.RING_VERSION_IRRELEVANT
+|| 
!shardManager.isOutOfDate(ClusterMetadata.current().epoch.getEpoch(
 return; // the disk boundaries (and thus the local ranges too) 
have not changed since the last time we calculated
 
 synchronized (this)
diff --git 
a/test/distributed/org/apache/cassandra/distributed/test/log/BounceIndexRebuildTest.java
 
b/test/distributed/org/apache/cassandra/distributed/test/log/BounceIndexRebuildTest.java
index 3fb2178e12..acf19d8fbf 100644
--- 
a/test/distributed/org/apache/cassandra/distributed/test/log/BounceIndexRebuildTest.java
+++ 
b/test/distributed/org/apache/cassandra/distributed/test/log/BounceIndexRebuildTest.java
@@ -18,6 +18,7 @@
 
 package org.apache.cassandra.distributed.test.log;
 
+import org.junit.Assert;
 import org.junit.Test;
 
 import org.apache.cassandra.distributed.Cluster;
@@ -41,10 +42,20 @@ public class BounceIndexRebuildTest extends TestBaseImpl
 cluster.schemaChange(withKeyspace("create index idx on %s.tbl 
(x)"));
 Object[][] res = 
cluster.coordinator(1).execute(withKeyspace("select * from %s.tbl where x=5"), 
ConsistencyLevel.ALL);
 assert res.length > 0;
+String patternLegacyBuild = "Index build of idx complete";
+int preBounceLegacyBuilds = 
cluster.get(1).logs().grep(patternLegacyBuild).getResult().size();
+
+final String patternSaiValidation = "Validating per-column index 
components for distributed_test_keyspace.idx";
+int preBounceSaiValidations = 
cluster.get(1).logs().grep(patternSaiValidation).getResult().size();
 
 cluster.get(1).shutdown().get();
 cluster.get(1).startup();
-assertEquals(1, cluster.get(1).logs().grep("Index build of idx 
complete").getResult().size());
+// Make sure legacy index does not rebuild on restart
+assertEquals(preBounceLegacyBuilds, 
cluster.get(1).logs().grep(patternLegacyBuild).getResult().size());
+// If we are using SAI, we want the index to validate rather than 
build
+if (preBounceLegacyBuilds == 0)
+
Assert.assertTrue(cluster.get(1).logs().grep(patternSaiValidation).getResult().size()
 > preBounceSaiValidations);
+
 res = cluster.coordinator(1).execute(withKeyspace("select * from 
%s.tbl where x=5"), ConsistencyLevel.ALL);
 assert res.length > 0;
 }


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

(cassandra) branch trunk updated (bfb5c59342 -> 28efb63df5)

2024-03-07 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git


from bfb5c59342 Fix datetime_from_utc_to_local in cqlshlib
 add 06ed1afc34 Add an optimized default configuration to tests and make it 
available for new users
 new 28efb63df5 Merge branch 'cassandra-5.0' into trunk

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .build/README.md   |2 +-
 .build/docker/run-tests.sh |2 +-
 .build/run-tests.sh|4 +-
 .circleci/config.yml   | 2429 -
 .circleci/config.yml.FREE  | 2429 -
 .circleci/config.yml.PAID  | 2793 ++--
 .circleci/config_template.yml  |  374 ++-
 .circleci/config_template.yml.PAID.patch   |   66 +-
 .jenkins/Jenkinsfile   |   12 +-
 CHANGES.txt|3 +-
 NEWS.txt   |6 +
 build.xml  |   47 +-
 conf/cassandra.yaml|   52 +-
 conf/{cassandra.yaml => cassandra_latest.yaml} |  116 +-
 .../config/CassandraRelevantProperties.java|2 +
 .../cassandra/config/DatabaseDescriptor.java   |7 +-
 src/java/org/apache/cassandra/db/ReadCommand.java  |3 +
 .../db/compaction/UnifiedCompactionStrategy.java   |1 +
 .../db/compaction/unified/Controller.java  |5 +-
 test/conf/{trie_memtable.yaml => latest_diff.yaml} |   37 +-
 .../cassandra/distributed/impl/InstanceConfig.java |   39 +
 .../cassandra/distributed/test/CasWriteTest.java   |   33 +-
 .../test/PreviewRepairSnapshotTest.java|   21 +-
 .../test/SecondaryIndexCompactionTest.java |3 +
 .../distributed/test/SecondaryIndexTest.java   |2 +-
 .../distributed/test/UpgradeSSTablesTest.java  |6 +-
 .../test/sai/ConcurrencyFactorTest.java|4 +-
 test/unit/org/apache/cassandra/Util.java   |8 +
 .../cassandra/config/ParseAndConvertUnitsTest.java |3 +-
 .../cql3/statements/DescribeStatementTest.java |   20 +-
 .../validation/entities/FrozenCollectionsTest.java |3 +
 .../entities/SecondaryIndexOnMapEntriesTest.java   |   19 +-
 .../miscellaneous/CrcCheckChanceTest.java  |   61 +-
 .../operations/CompactStorageSplit1Test.java   |3 +
 .../InsertInvalidateSizedRecordsTest.java  |2 +
 .../operations/InsertUpdateIfConditionTest.java|2 +-
 .../validation/operations/SelectLimitTest.java |4 +-
 .../operations/SelectMultiColumnRelationTest.java  |4 +
 .../operations/SelectOrderedPartitionerTest.java   |6 +
 .../operations/SelectSingleColumnRelationTest.java |7 +-
 .../cql3/validation/operations/SelectTest.java |   52 +-
 .../apache/cassandra/db/SchemaCQLHelperTest.java   |   10 +-
 .../db/compaction/ActiveCompactionsTest.java   |4 +-
 .../db/compaction/CancelCompactionsTest.java   |7 +
 .../db/compaction/unified/ControllerTest.java  |8 +-
 .../compaction/unified/ShardedMultiWriterTest.java |   11 +-
 .../writers/CompactionAwareWriterTest.java |3 +-
 .../repair/AbstractPendingAntiCompactionTest.java  |5 +-
 .../db/repair/PendingAntiCompactionTest.java   |1 +
 .../apache/cassandra/index/CustomIndexTest.java|   15 +-
 .../cassandra/index/SecondaryIndexManagerTest.java |2 +
 .../index/internal/CassandraIndexTest.java |2 +
 .../sai/cql/MixedIndexImplementationsTest.java |3 +
 .../io/sstable/CQLSSTableWriterClientTest.java |3 +-
 .../cassandra/io/sstable/LegacySSTableTest.java|2 +-
 .../apache/cassandra/service/StorageProxyTest.java |1 +
 .../cassandra/tools/CompactionStressTest.java  |4 +-
 57 files changed, 4644 insertions(+), 4129 deletions(-)
 copy conf/{cassandra.yaml => cassandra_latest.yaml} (97%)
 rename test/conf/{trie_memtable.yaml => latest_diff.yaml} (54%)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

(cassandra) 01/01: Merge branch 'cassandra-5.0' into trunk

2024-03-07 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 28efb63df52bafaf51cd458da021f6050900017a
Merge: bfb5c59342 06ed1afc34
Author: Branimir Lambov 
AuthorDate: Thu Mar 7 11:22:48 2024 +0200

Merge branch 'cassandra-5.0' into trunk

 .build/README.md   |2 +-
 .build/docker/run-tests.sh |2 +-
 .build/run-tests.sh|4 +-
 .circleci/config.yml   | 2429 -
 .circleci/config.yml.FREE  | 2429 -
 .circleci/config.yml.PAID  | 2793 ++--
 .circleci/config_template.yml  |  374 ++-
 .circleci/config_template.yml.PAID.patch   |   66 +-
 .jenkins/Jenkinsfile   |   12 +-
 CHANGES.txt|3 +-
 NEWS.txt   |6 +
 build.xml  |   47 +-
 conf/cassandra.yaml|   52 +-
 conf/{cassandra.yaml => cassandra_latest.yaml} |  116 +-
 .../config/CassandraRelevantProperties.java|2 +
 .../cassandra/config/DatabaseDescriptor.java   |7 +-
 src/java/org/apache/cassandra/db/ReadCommand.java  |3 +
 .../db/compaction/UnifiedCompactionStrategy.java   |1 +
 .../db/compaction/unified/Controller.java  |5 +-
 test/conf/{trie_memtable.yaml => latest_diff.yaml} |   37 +-
 .../cassandra/distributed/impl/InstanceConfig.java |   39 +
 .../cassandra/distributed/test/CasWriteTest.java   |   33 +-
 .../test/PreviewRepairSnapshotTest.java|   21 +-
 .../test/SecondaryIndexCompactionTest.java |3 +
 .../distributed/test/SecondaryIndexTest.java   |2 +-
 .../distributed/test/UpgradeSSTablesTest.java  |6 +-
 .../test/sai/ConcurrencyFactorTest.java|4 +-
 test/unit/org/apache/cassandra/Util.java   |8 +
 .../cassandra/config/ParseAndConvertUnitsTest.java |3 +-
 .../cql3/statements/DescribeStatementTest.java |   20 +-
 .../validation/entities/FrozenCollectionsTest.java |3 +
 .../entities/SecondaryIndexOnMapEntriesTest.java   |   19 +-
 .../miscellaneous/CrcCheckChanceTest.java  |   61 +-
 .../operations/CompactStorageSplit1Test.java   |3 +
 .../InsertInvalidateSizedRecordsTest.java  |2 +
 .../operations/InsertUpdateIfConditionTest.java|2 +-
 .../validation/operations/SelectLimitTest.java |4 +-
 .../operations/SelectMultiColumnRelationTest.java  |4 +
 .../operations/SelectOrderedPartitionerTest.java   |6 +
 .../operations/SelectSingleColumnRelationTest.java |7 +-
 .../cql3/validation/operations/SelectTest.java |   52 +-
 .../apache/cassandra/db/SchemaCQLHelperTest.java   |   10 +-
 .../db/compaction/ActiveCompactionsTest.java   |4 +-
 .../db/compaction/CancelCompactionsTest.java   |7 +
 .../db/compaction/unified/ControllerTest.java  |8 +-
 .../compaction/unified/ShardedMultiWriterTest.java |   11 +-
 .../writers/CompactionAwareWriterTest.java |3 +-
 .../repair/AbstractPendingAntiCompactionTest.java  |5 +-
 .../db/repair/PendingAntiCompactionTest.java   |1 +
 .../apache/cassandra/index/CustomIndexTest.java|   15 +-
 .../cassandra/index/SecondaryIndexManagerTest.java |2 +
 .../index/internal/CassandraIndexTest.java |2 +
 .../sai/cql/MixedIndexImplementationsTest.java |3 +
 .../io/sstable/CQLSSTableWriterClientTest.java |3 +-
 .../cassandra/io/sstable/LegacySSTableTest.java|2 +-
 .../apache/cassandra/service/StorageProxyTest.java |1 +
 .../cassandra/tools/CompactionStressTest.java  |4 +-
 57 files changed, 4644 insertions(+), 4129 deletions(-)

diff --cc CHANGES.txt
index 43b128db24,4a005b4aab..06df50e173
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,26 -1,5 +1,27 @@@
 -5.0-beta2
 +5.1
 + * Modernize CQLSH datetime conversions (CASSANDRA-18879)
 + * Harry model and in-JVM tests for partition-restricted 2i queries 
(CASSANDRA-18275)
 + * Refactor cqlshmain global constants (CASSANDRA-19201)
 + * Remove native_transport_port_ssl (CASSANDRA-19397)
 + * Make nodetool reconfigurecms sync by default and add --cancel to be able 
to cancel ongoing reconfigurations (CASSANDRA-19216)
 + * Expose auth mode in system_views.clients, nodetool clientstats, metrics 
(CASSANDRA-19366)
 + * Remove sealed_periods and last_sealed_period tables (CASSANDRA-19189)
 + * Improve setup and initialisation of LocalLog/LogSpec (CASSANDRA-19271)
 + * Refactor structure of caching metrics and expose auth cache metrics via 
JMX (CASSANDRA-17062)
 + * Allow CQL client certificate authentication to work without sending an 
AUTHENTICATE request (CASSANDRA-18857)
-  *

(cassandra-dtest) 01/02: Add support for specifying the name of the file to use as cassandra YAML.

2023-12-20 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra-dtest.git

commit f41ee663f4e79d4adc5ae5e02d39b75478908115
Author: Branimir Lambov 
AuthorDate: Wed Nov 22 12:57:52 2023 +0200

Add support for specifying the name of the file to use as cassandra YAML.

patch by Branimir Lambov; reviewed by Mick Semb Wever for CASSANDRA-18753
---
 conftest.py | 2 ++
 dtest_config.py | 2 ++
 dtest_setup.py  | 2 ++
 run_dtests.py   | 1 +
 4 files changed, 7 insertions(+)

diff --git a/conftest.py b/conftest.py
index ef3a9d47..620857a3 100644
--- a/conftest.py
+++ b/conftest.py
@@ -50,6 +50,8 @@ def pytest_addoption(parser):
  help="Determines wither or not to setup clusters using 
vnodes for tests")
 parser.addoption("--use-off-heap-memtables", action="store_true", 
default=False,
  help="Enable Off Heap Memtables when creating test 
clusters for tests")
+parser.addoption("--configuration-yaml", action="store", default=None,
+ help="The name of the cassandra configuration YAML (e.g. 
cassandra_latest.yaml)")
 parser.addoption("--num-tokens", action="store", default=256,
  help="Number of tokens to set num_tokens yaml setting to 
when creating instances "
   "with vnodes enabled")
diff --git a/dtest_config.py b/dtest_config.py
index 86e8c96b..5d7cc0e5 100644
--- a/dtest_config.py
+++ b/dtest_config.py
@@ -12,6 +12,7 @@ logger = logging.getLogger(__name__)
 class DTestConfig:
 def __init__(self):
 self.use_vnodes = True
+self.configuration_yaml = None
 self.use_off_heap_memtables = False
 self.num_tokens = -1
 self.data_dir_count = -1
@@ -42,6 +43,7 @@ class DTestConfig:
 return
 
 self.use_vnodes = config.getoption("--use-vnodes")
+self.configuration_yaml = config.getoption("--configuration-yaml")
 self.use_off_heap_memtables = 
config.getoption("--use-off-heap-memtables")
 self.num_tokens = config.getoption("--num-tokens")
 self.data_dir_count = config.getoption("--data-dir-count-per-instance")
diff --git a/dtest_setup.py b/dtest_setup.py
index 2512f132..3980259b 100644
--- a/dtest_setup.py
+++ b/dtest_setup.py
@@ -474,6 +474,8 @@ class DTestSetup(object):
 
 if self.dtest_config.use_off_heap_memtables:
 
self.cluster.set_configuration_options(values={'memtable_allocation_type': 
'offheap_objects'})
+if self.dtest_config.configuration_yaml is not None:
+
self.cluster.set_configuration_yaml(self.dtest_config.configuration_yaml)
 
 self.cluster.set_configuration_options(values)
 logger.debug("Done setting configuration options:\n" + 
pprint.pformat(self.cluster._config_options, indent=4))
diff --git a/run_dtests.py b/run_dtests.py
index b8eed9c7..0f24443d 100755
--- a/run_dtests.py
+++ b/run_dtests.py
@@ -28,6 +28,7 @@ optional arguments:
   --dtest-print-tests-output=DTEST_PRINT_TESTS_OUTPUTPath to file 
where the output of --dtest-print-tests-only should be written to (default: 
False)
   --pytest-options=PYTEST_OPTIONSAdditional 
command line arguments to proxy directly thru when invoking pytest. (default: 
None)
   --dtest-tests=DTEST_TESTS  Comma separated 
list of test files, test classes, or test methods to execute. (default: None)
+  --configuration-yaml=CONFIG_FILE   The name of the 
cassandra configuration YAML (e.g. cassandra_latest.yaml) (default: None)
 """
 import subprocess
 import sys


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

(cassandra-dtest) 02/02: Test generalizations for BTI format and SAI index defaults

2023-12-20 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra-dtest.git

commit b3c4d981d465f59de601425689015bcb323d32ad
Author: Branimir Lambov 
AuthorDate: Fri Dec 1 17:45:50 2023 +0200

Test generalizations for BTI format and SAI index defaults

patch by Branimir Lambov; reviewed by Mick Semb Wever for CASSANDRA-18753
---
 offline_tools_test.py  |  4 +--
 schema_metadata_test.py| 17 --
 scrub_test.py  | 11 ++
 secondary_indexes_test.py  | 12 +++
 sstable_generation_loading_test.py | 68 ++
 5 files changed, 79 insertions(+), 33 deletions(-)

diff --git a/offline_tools_test.py b/offline_tools_test.py
index 26e0bf17..e69786ea 100644
--- a/offline_tools_test.py
+++ b/offline_tools_test.py
@@ -277,9 +277,9 @@ class TestOfflineTools(Tester):
 hashcomputed = False
 for line in outlines:
 if sstable in line:
-if "Verifying BigTableReader" in line:
+if re.search(re.compile(r"Verifying \w+TableReader"), 
line):
 verified = True
-elif "Checking computed hash of BigTableReader" in line:
+elif re.search(re.compile(r"Checking computed hash of 
\w+TableReader"), line):
 hashcomputed = True
 else:
 logger.debug(line)
diff --git a/schema_metadata_test.py b/schema_metadata_test.py
index e859ae21..6f901094 100644
--- a/schema_metadata_test.py
+++ b/schema_metadata_test.py
@@ -73,7 +73,11 @@ def verify_indexes_table(created_on_version, 
current_version, keyspace, session,
 
 assert 1 == len(meta.indexes)
 
-assert {'target': 'd'} == meta.indexes[index_name].index_options
+opts = dict(meta.indexes[index_name].index_options)
+# we do not want the class name (set when SAI is the default index class) 
to break this
+opts = { k : opts[k] for k in set(opts) - {'class_name'} }
+
+assert {'target': 'd'} == opts
 assert 3 == len(meta.primary_key)
 assert 'a' == meta.primary_key[0].name
 assert 'b' == meta.primary_key[1].name
@@ -558,8 +562,15 @@ class TestSchemaMetadata(Tester):
 ix_meta = self._keyspace_meta().indexes['ix_born_to_die_name']
 assert 'ix_born_to_die_name' == ix_meta.name
 
-assert {'target': 'name'} == ix_meta.index_options
-assert 'COMPOSITES' == ix_meta.kind
+opts = dict(ix_meta.index_options)
+# When SAI is the default index class, we get a somewhat different 
configuration.
+if 'class_name' in opts:
+opts = { k : opts[k] for k in set(opts) - {'class_name'} }
+assert 'CUSTOM' == ix_meta.kind
+else:
+assert 'COMPOSITES' == ix_meta.kind
+
+assert {'target': 'name'} == opts
 
 self.session.execute("drop table born_to_die")
 assert self._keyspace_meta().tables.get('born_to_die') is None
diff --git a/scrub_test.py b/scrub_test.py
index 3d50d70c..69e77bde 100644
--- a/scrub_test.py
+++ b/scrub_test.py
@@ -10,6 +10,8 @@ import logging
 
 from ccmlib import common
 
+from tools.misc import ImmutableMapping
+from dtest_setup_overrides import DTestSetupOverrides
 from dtest import Tester, create_ks, create_cf
 from tools.assertions import assert_length_equal, assert_stderr_clean
 
@@ -190,8 +192,17 @@ class TestHelper(Tester):
 class TestScrubIndexes(TestHelper):
 """
 Test that we scrub indexes as well as their parent tables
+Only valid for legacy secondary indexes
 """
 
+@pytest.fixture(scope='function', autouse=True)
+def fixture_dtest_setup_overrides(self, dtest_config):
+dtest_setup_overrides = DTestSetupOverrides()
+
+if dtest_config.cassandra_version_from_build >= '5.0':
+dtest_setup_overrides.cluster_options = 
ImmutableMapping({'default_secondary_index': 'legacy_local_table'})
+return dtest_setup_overrides
+
 def create_users(self, session):
 columns = {"password": "varchar", "gender": "varchar", 
"session_token": "varchar", "state": "varchar", "birth_year": "bigint"}
 create_cf(session, 'users', columns=columns)
diff --git a/secondary_indexes_test.py b/secondary_indexes_test.py
index 97f40483..1e14d636 100644
--- a/secondary_indexes_test.py
+++ b/secondary_indexes_test.py
@@ -17,6 +17,8 @@ from cassandra.protocol import ConfigurationException
 from cassandra.query import BatchStatement, SimpleStatement
 
 from bootstrap_test import BootstrapTester
+from tools.misc import ImmutableMapping
+from dtest_setup_overrides import DTestSetupOverrides
 from dtest import

(cassandra-dtest) branch trunk updated (24d6b291 -> b3c4d981)

2023-12-20 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra-dtest.git


from 24d6b291 Release 5.0-beta1
 new f41ee663 Add support for specifying the name of the file to use as 
cassandra YAML.
 new b3c4d981 Test generalizations for BTI format and SAI index defaults

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 conftest.py|  2 ++
 dtest_config.py|  2 ++
 dtest_setup.py |  2 ++
 offline_tools_test.py  |  4 +--
 run_dtests.py  |  1 +
 schema_metadata_test.py| 17 --
 scrub_test.py  | 11 ++
 secondary_indexes_test.py  | 12 +++
 sstable_generation_loading_test.py | 68 ++
 9 files changed, 86 insertions(+), 33 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch cassandra-4.1 updated (1acab45e4f -> db6641fbb6)

2023-09-21 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a change to branch cassandra-4.1
in repository https://gitbox.apache.org/repos/asf/cassandra.git


from 1acab45e4f Merge branch 'cassandra-4.0' into cassandra-4.1
 new 87c2af85c1 Fix delayed SSTable release with 
unsafe_aggressive_sstable_expiration
 new c6385ac3dd Merge branch 'cassandra-3.11' into cassandra-4.0
 new db6641fbb6 Merge branch 'cassandra-4.0' into cassandra-4.1

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|   1 +
 .../db/compaction/CompactionController.java|  12 +-
 .../db/compaction/CompactionControllerTest.java| 140 +
 3 files changed, 146 insertions(+), 7 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 01/01: Merge branch 'cassandra-5.0' into trunk

2023-09-21 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 970ec2d1db5770c13a42e1f2862ea398317d0f15
Merge: e1fc195558 a23f4c0b15
Author: Branimir Lambov 
AuthorDate: Thu Sep 21 17:20:29 2023 +0300

Merge branch 'cassandra-5.0' into trunk

 CHANGES.txt|   1 +
 .../db/compaction/CompactionController.java|  12 +-
 .../db/compaction/CompactionControllerTest.java| 141 +
 3 files changed, 147 insertions(+), 7 deletions(-)

diff --cc CHANGES.txt
index a998a12c81,f4c57d0444..f0be56e5e4
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -230,8 -210,12 +230,9 @@@ Merged from 4.1
  Merged from 4.0:
   * Partial compaction can resurrect deleted data (CASSANDRA-18507)
  Merged from 3.11:
+  * Fix delayed SSTable release with unsafe_aggressive_sstable_expiration 
(CASSANDRA-18756)
 - * Revert CASSANDRA-18543 (CASSANDRA-18854)
   * Fix NPE when using udfContext in UDF after a restart of a node 
(CASSANDRA-18739)
  Merged from 3.0:
 - * Refactor validation logic in StorageService.rebuild (CASSANDRA-18803)
 - * Make alternation of a user type validate the same way as creation of a 
user type does (CASSANDRA-18585)
   * CQLSH emits a warning when the server version doesn't match 
(CASSANDRA-18745)
   * Fix missing speculative retries in tablestats (CASSANDRA-18767)
   * Fix Requires for Java for RPM package (CASSANDRA-18751)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 01/01: Merge branch 'cassandra-4.0' into cassandra-4.1

2023-09-21 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch cassandra-4.1
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit db6641fbb6fd0c439e14f94caecdeee999311c62
Merge: 1acab45e4f c6385ac3dd
Author: Branimir Lambov 
AuthorDate: Thu Sep 21 16:03:41 2023 +0300

Merge branch 'cassandra-4.0' into cassandra-4.1

 CHANGES.txt|   1 +
 .../db/compaction/CompactionController.java|  12 +-
 .../db/compaction/CompactionControllerTest.java| 140 +
 3 files changed, 146 insertions(+), 7 deletions(-)

diff --cc CHANGES.txt
index bae7f3c6d8,6c4e0ef6b0..bb5cc728c9
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -9,8 -4,10 +9,9 @@@ Merged from 4.0
   * Fix BulkLoader ignoring cipher suites options (CASSANDRA-18582)
   * Migrate Python optparse to argparse (CASSANDRA-17914)
  Merged from 3.11:
+  * Fix delayed SSTable release with unsafe_aggressive_sstable_expiration 
(CASSANDRA-18756)
   * Revert CASSANDRA-18543 (CASSANDRA-18854)
   * Fix NPE when using udfContext in UDF after a restart of a node 
(CASSANDRA-18739)
 - * Moved jflex from runtime to build dependencies (CASSANDRA-18664)
  Merged from 3.0:
   * Add cqlshrc.sample and credentials.sample into Debian package 
(CASSANDRA-18818)
   * Refactor validation logic in StorageService.rebuild (CASSANDRA-18803)
diff --cc 
test/unit/org/apache/cassandra/db/compaction/CompactionControllerTest.java
index 9d81b61ed3,86546bb9f6..d1a8d01b92
--- a/test/unit/org/apache/cassandra/db/compaction/CompactionControllerTest.java
+++ b/test/unit/org/apache/cassandra/db/compaction/CompactionControllerTest.java
@@@ -182,6 -199,124 +199,124 @@@ public class CompactionControllerTest e
  assertEquals(1, expired.size());
  }
  
+ @Test
+ @BMRules(rules = {
+ @BMRule(name = "Pause compaction",
+ targetClass = "CompactionTask",
+ targetMethod = "runMayThrow",
+ targetLocation = "INVOKE getCompactionAwareWriter",
+ condition = "Thread.currentThread().getName().equals(\"compaction1\")",
+ action = 
"org.apache.cassandra.db.compaction.CompactionControllerTest.createCompactionControllerLatch.countDown();"
 +
+  
"com.google.common.util.concurrent.Uninterruptibles.awaitUninterruptibly" +
+  
"(org.apache.cassandra.db.compaction.CompactionControllerTest.compaction2FinishLatch);"),
+ @BMRule(name = "Check overlaps",
+ targetClass = "CompactionTask",
+ targetMethod = "runMayThrow",
+ targetLocation = "INVOKE finish",
+ condition = "Thread.currentThread().getName().equals(\"compaction1\")",
+ action = 
"org.apache.cassandra.db.compaction.CompactionControllerTest.compaction1RefreshLatch.countDown();"
 +
+  
"com.google.common.util.concurrent.Uninterruptibles.awaitUninterruptibly" +
+  
"(org.apache.cassandra.db.compaction.CompactionControllerTest.refreshCheckLatch);"),
+ @BMRule(name = "Increment overlap refresh counter",
+ targetClass = "ColumnFamilyStore",
+ targetMethod = "getAndReferenceOverlappingLiveSSTables",
+ condition = "Thread.currentThread().getName().equals(\"compaction1\")",
+ action = 
"org.apache.cassandra.db.compaction.CompactionControllerTest.incrementOverlapRefreshCounter();")
+ })
+ public void testIgnoreOverlaps() throws Exception
+ {
+ testOverlapIterator(true);
+ overlapRefreshCounter = 0;
+ compaction2FinishLatch = new CountDownLatch(1);
+ createCompactionControllerLatch = new CountDownLatch(1);
+ compaction1RefreshLatch = new CountDownLatch(1);
+ refreshCheckLatch = new CountDownLatch(1);
+ testOverlapIterator(false);
+ }
+ 
+ public void testOverlapIterator(boolean ignoreOverlaps) throws Exception
+ {
+ 
+ Keyspace keyspace = Keyspace.open(KEYSPACE);
+ ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(CF1);
+ cfs.truncateBlocking();
+ cfs.disableAutoCompaction();
+ 
+ //create 2 overlapping sstables
+ DecoratedKey key = Util.dk("k1");
+ long timestamp1 = FBUtilities.timestampMicros();
+ long timestamp2 = timestamp1 - 5;
+ applyMutation(cfs.metadata(), key, timestamp1);
 -cfs.forceBlockingFlush();
++cfs.forceBlockingFlush(ColumnFamilyStore.FlushReason.UNIT_TESTS);
+ assertEquals(cfs.getLiveSSTables().size(), 1);
+ Set sstables = cfs.getLiveSSTables();
+ 
+ applyMutation(cfs.metadata(), key, timestamp2);
 -cfs.forceBlockingFlush();
++cfs.forceBlockingFlush(ColumnFamilyStore.FlushReason.UNIT_TESTS);
+ assertEquals(cfs.getLiveSSTables().size(),

[cassandra] branch cassandra-4.0 updated (31aede3275 -> c6385ac3dd)

2023-09-21 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a change to branch cassandra-4.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git


from 31aede3275 Enable 3rd party JDK installations for Debian package
 new 87c2af85c1 Fix delayed SSTable release with 
unsafe_aggressive_sstable_expiration
 new c6385ac3dd Merge branch 'cassandra-3.11' into cassandra-4.0

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|   1 +
 .../db/compaction/CompactionController.java|  12 +-
 .../db/compaction/CompactionControllerTest.java| 141 -
 3 files changed, 146 insertions(+), 8 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 01/01: Merge branch 'cassandra-3.11' into cassandra-4.0

2023-09-21 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch cassandra-4.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit c6385ac3ddccabdc7cb650b090fa69c0523274e8
Merge: 31aede3275 87c2af85c1
Author: Branimir Lambov 
AuthorDate: Thu Sep 21 16:02:01 2023 +0300

Merge branch 'cassandra-3.11' into cassandra-4.0

 CHANGES.txt|   1 +
 .../db/compaction/CompactionController.java|  12 +-
 .../db/compaction/CompactionControllerTest.java| 141 -
 3 files changed, 146 insertions(+), 8 deletions(-)

diff --cc CHANGES.txt
index 13de2ab602,74755be6e7..6c4e0ef6b0
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,12 -1,7 +1,13 @@@
 -3.11.17
 +4.0.12
 + * Enable 3rd party JDK installations for Debian package (CASSANDRA-18844)
 + * Fix NTS log message when an unrecognized strategy option is passed 
(CASSANDRA-18679)
 + * Fix BulkLoader ignoring cipher suites options (CASSANDRA-18582)
 + * Migrate Python optparse to argparse (CASSANDRA-17914)
 +Merged from 3.11:
+  * Fix delayed SSTable release with unsafe_aggressive_sstable_expiration 
(CASSANDRA-18756)
   * Revert CASSANDRA-18543 (CASSANDRA-18854)
   * Fix NPE when using udfContext in UDF after a restart of a node 
(CASSANDRA-18739)
 + * Moved jflex from runtime to build dependencies (CASSANDRA-18664)
  Merged from 3.0:
   * Add cqlshrc.sample and credentials.sample into Debian package 
(CASSANDRA-18818)
   * Refactor validation logic in StorageService.rebuild (CASSANDRA-18803)
diff --cc src/java/org/apache/cassandra/db/compaction/CompactionController.java
index cee2b58f75,06272a1075..0c520d9491
--- a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
@@@ -73,7 -81,11 +73,9 @@@ public class CompactionController exten
  
  public CompactionController(ColumnFamilyStore cfs, Set 
compacting, int gcBefore, RateLimiter limiter, TombstoneOption tombstoneOption)
  {
+ //When making changes to the method, be aware that some of the state 
of the controller may still be uninitialized
+ //(e.g. TWCS sets up the value of ignoreOverlaps() after this 
completes)
 -assert cfs != null;
 -this.cfs = cfs;
 -this.gcBefore = gcBefore;
 +super(cfs, gcBefore, tombstoneOption);
  this.compacting = compacting;
  this.limiter = limiter;
  compactingRepaired = compacting != null && 
compacting.stream().allMatch(SSTableReader::isRepaired);
@@@ -94,18 -107,6 +96,12 @@@
  return;
  }
  
- if (ignoreOverlaps())
- {
- logger.debug("not refreshing overlaps - running with 
ignoreOverlaps activated");
- return;
- }
- 
 +if (cfs.getNeverPurgeTombstones())
 +{
 +logger.debug("not refreshing overlaps for {}.{} - 
neverPurgeTombstones is enabled", cfs.keyspace.getName(), cfs.getTableName());
 +return;
 +}
 +
  for (SSTableReader reader : overlappingSSTables)
  {
  if (reader.isMarkedCompacted())
diff --cc 
test/unit/org/apache/cassandra/db/compaction/CompactionControllerTest.java
index 500a88179f,aa95ba56fb..86546bb9f6
--- a/test/unit/org/apache/cassandra/db/compaction/CompactionControllerTest.java
+++ b/test/unit/org/apache/cassandra/db/compaction/CompactionControllerTest.java
@@@ -19,13 -19,19 +19,19 @@@
  package org.apache.cassandra.db.compaction;
  
  import java.nio.ByteBuffer;
+ import java.util.HashMap;
+ import java.util.Map;
  import java.util.Set;
+ import java.util.concurrent.CountDownLatch;
+ import java.util.concurrent.TimeUnit;
 -import java.util.function.Predicate;
 +import java.util.function.LongPredicate;
- import java.util.function.Predicate;
  
+ import com.google.common.collect.Iterables;
  import com.google.common.collect.Sets;
+ import com.google.common.util.concurrent.Uninterruptibles;
  import org.junit.BeforeClass;
  import org.junit.Test;
+ import org.junit.runner.RunWith;
  
  import org.apache.cassandra.SchemaLoader;
  import org.apache.cassandra.Util;
@@@ -183,7 -201,125 +199,125 @@@ public class CompactionControllerTest e
  assertEquals(1, expired.size());
  }
  
+ @Test
+ @BMRules(rules = {
+ @BMRule(name = "Pause compaction",
+ targetClass = "CompactionTask",
+ targetMethod = "runMayThrow",
+ targetLocation = "INVOKE getCompactionAwareWriter",
+ condition = "Thread.currentThread().getName().equals(\"compaction1\")",
+ action = 
"org.apache.cassandra.db.compaction.CompactionControllerTest.createCompactionControllerLatch.countDown();"
 +
+  
"com.google.common.util.concurrent.Uninterruptibles.awaitUninterruptibly" +
+  
"(org.apache.cassandra.db.compact

[cassandra] branch cassandra-5.0 updated (07768674ac -> a23f4c0b15)

2023-09-21 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a change to branch cassandra-5.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git


from 07768674ac Merge branch 'cassandra-4.1' into cassandra-5.0
 new 87c2af85c1 Fix delayed SSTable release with 
unsafe_aggressive_sstable_expiration
 new c6385ac3dd Merge branch 'cassandra-3.11' into cassandra-4.0
 new db6641fbb6 Merge branch 'cassandra-4.0' into cassandra-4.1
 new a23f4c0b15 Merge branch 'cassandra-4.1' into cassandra-5.0

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|   1 +
 .../db/compaction/CompactionController.java|  12 +-
 .../db/compaction/CompactionControllerTest.java| 141 +
 3 files changed, 147 insertions(+), 7 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch trunk updated (e1fc195558 -> 970ec2d1db)

2023-09-21 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git


from e1fc195558 Fix invalid unit test check for CreateTableValidationTest
 new 87c2af85c1 Fix delayed SSTable release with 
unsafe_aggressive_sstable_expiration
 new c6385ac3dd Merge branch 'cassandra-3.11' into cassandra-4.0
 new db6641fbb6 Merge branch 'cassandra-4.0' into cassandra-4.1
 new a23f4c0b15 Merge branch 'cassandra-4.1' into cassandra-5.0
 new 970ec2d1db Merge branch 'cassandra-5.0' into trunk

The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|   1 +
 .../db/compaction/CompactionController.java|  12 +-
 .../db/compaction/CompactionControllerTest.java| 141 +
 3 files changed, 147 insertions(+), 7 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 01/01: Merge branch 'cassandra-4.1' into cassandra-5.0

2023-09-21 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch cassandra-5.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit a23f4c0b15c684240ef0bcd55875610e8bd7179b
Merge: 07768674ac db6641fbb6
Author: Branimir Lambov 
AuthorDate: Thu Sep 21 17:01:16 2023 +0300

Merge branch 'cassandra-4.1' into cassandra-5.0

 CHANGES.txt|   1 +
 .../db/compaction/CompactionController.java|  12 +-
 .../db/compaction/CompactionControllerTest.java| 141 +
 3 files changed, 147 insertions(+), 7 deletions(-)

diff --cc CHANGES.txt
index 3f0fecbb6b,bb5cc728c9..f4c57d0444
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -208,8 -4,12 +208,9 @@@ Merged from 4.1
   * Allow empty keystore_password in encryption_options (CASSANDRA-18778)
   * Skip ColumnFamilyStore#topPartitions initialization when client or tool 
mode (CASSANDRA-18697)
  Merged from 4.0:
 - * Enable 3rd party JDK installations for Debian package (CASSANDRA-18844)
 - * Fix NTS log message when an unrecognized strategy option is passed 
(CASSANDRA-18679)
 - * Fix BulkLoader ignoring cipher suites options (CASSANDRA-18582)
 - * Migrate Python optparse to argparse (CASSANDRA-17914)
 + * Partial compaction can resurrect deleted data (CASSANDRA-18507)
  Merged from 3.11:
+  * Fix delayed SSTable release with unsafe_aggressive_sstable_expiration 
(CASSANDRA-18756)
   * Revert CASSANDRA-18543 (CASSANDRA-18854)
   * Fix NPE when using udfContext in UDF after a restart of a node 
(CASSANDRA-18739)
  Merged from 3.0:
diff --cc src/java/org/apache/cassandra/db/compaction/CompactionController.java
index eadd85c3fd,279a1c4e6f..c01da7000d
--- a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
@@@ -79,8 -70,10 +79,10 @@@ public class CompactionController exten
   
cfs.getCompactionStrategyManager().getCompactionParams().tombstoneOption());
  }
  
 -public CompactionController(ColumnFamilyStore cfs, Set 
compacting, int gcBefore, RateLimiter limiter, TombstoneOption tombstoneOption)
 +public CompactionController(ColumnFamilyStore cfs, Set 
compacting, long gcBefore, RateLimiter limiter, TombstoneOption tombstoneOption)
  {
+ //When making changes to the method, be aware that some of the state 
of the controller may still be uninitialized
+ //(e.g. TWCS sets up the value of ignoreOverlaps() after this 
completes)
  super(cfs, gcBefore, tombstoneOption);
  this.compacting = compacting;
  this.limiter = limiter;
@@@ -101,15 -95,9 +103,9 @@@
  return;
  }
  
- if (ignoreOverlaps())
- {
- logger.debug("not refreshing overlaps - running with 
ignoreOverlaps activated");
- return;
- }
- 
  if (cfs.getNeverPurgeTombstones())
  {
 -logger.debug("not refreshing overlaps for {}.{} - 
neverPurgeTombstones is enabled", cfs.keyspace.getName(), cfs.getTableName());
 +logger.debug("not refreshing overlaps for {}.{} - 
neverPurgeTombstones is enabled", cfs.getKeyspaceName(), cfs.getTableName());
  return;
  }
  
diff --cc 
test/unit/org/apache/cassandra/db/compaction/CompactionControllerTest.java
index 3154a16ac3,d1a8d01b92..ff76ce2575
--- a/test/unit/org/apache/cassandra/db/compaction/CompactionControllerTest.java
+++ b/test/unit/org/apache/cassandra/db/compaction/CompactionControllerTest.java
@@@ -28,6 -35,6 +35,7 @@@ import org.junit.runner.RunWith
  
  import org.apache.cassandra.SchemaLoader;
  import org.apache.cassandra.Util;
++import org.apache.cassandra.config.CassandraRelevantProperties;
  import org.apache.cassandra.schema.TableMetadata;
  import org.apache.cassandra.db.ColumnFamilyStore;
  import org.apache.cassandra.db.DecoratedKey;
@@@ -182,6 -199,124 +200,124 @@@ public class CompactionControllerTest e
  assertEquals(1, expired.size());
  }
  
+ @Test
+ @BMRules(rules = {
+ @BMRule(name = "Pause compaction",
+ targetClass = "CompactionTask",
+ targetMethod = "runMayThrow",
+ targetLocation = "INVOKE getCompactionAwareWriter",
+ condition = "Thread.currentThread().getName().equals(\"compaction1\")",
+ action = 
"org.apache.cassandra.db.compaction.CompactionControllerTest.createCompactionControllerLatch.countDown();"
 +
+  
"com.google.common.util.concurrent.Uninterruptibles.awaitUninterruptibly" +
+  
"(org.apache.cassandra.db.compaction.CompactionControllerTest.compaction2FinishLatch);"),
+ @BMRule(name = "Check overlaps",
+ targetClass = "CompactionTask",
+ targetMethod = "

[cassandra] branch cassandra-3.11 updated: Fix delayed SSTable release with unsafe_aggressive_sstable_expiration

2023-09-21 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/cassandra-3.11 by this push:
 new 87c2af85c1 Fix delayed SSTable release with 
unsafe_aggressive_sstable_expiration
87c2af85c1 is described below

commit 87c2af85c1305c130af7d66f83dec03a1c4a8bb2
Author: Ethan Brown 
AuthorDate: Fri Aug 18 13:02:15 2023 -0700

Fix delayed SSTable release with unsafe_aggressive_sstable_expiration

patch by Ethan Brown; reviewed by Branimir Lambov and Mick Semb Wever for 
CASSANDRA-18756
---
 CHANGES.txt|   1 +
 .../db/compaction/CompactionController.java|  12 +-
 .../db/compaction/CompactionControllerTest.java| 140 +
 3 files changed, 146 insertions(+), 7 deletions(-)

diff --git a/CHANGES.txt b/CHANGES.txt
index 2d9e2059e1..74755be6e7 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.11.17
+ * Fix delayed SSTable release with unsafe_aggressive_sstable_expiration 
(CASSANDRA-18756)
  * Revert CASSANDRA-18543 (CASSANDRA-18854)
  * Fix NPE when using udfContext in UDF after a restart of a node 
(CASSANDRA-18739)
 Merged from 3.0:
diff --git 
a/src/java/org/apache/cassandra/db/compaction/CompactionController.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
index 19318ff1a9..06272a1075 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
@@ -81,6 +81,8 @@ public class CompactionController implements AutoCloseable
 
 public CompactionController(ColumnFamilyStore cfs, Set 
compacting, int gcBefore, RateLimiter limiter, TombstoneOption tombstoneOption)
 {
+//When making changes to the method, be aware that some of the state 
of the controller may still be uninitialized
+//(e.g. TWCS sets up the value of ignoreOverlaps() after this 
completes)
 assert cfs != null;
 this.cfs = cfs;
 this.gcBefore = gcBefore;
@@ -105,12 +107,6 @@ public class CompactionController implements AutoCloseable
 return;
 }
 
-if (ignoreOverlaps())
-{
-logger.debug("not refreshing overlaps - running with 
ignoreOverlaps activated");
-return;
-}
-
 for (SSTableReader reader : overlappingSSTables)
 {
 if (reader.isMarkedCompacted())
@@ -129,7 +125,7 @@ public class CompactionController implements AutoCloseable
 if (this.overlappingSSTables != null)
 close();
 
-if (compacting == null || ignoreOverlaps())
+if (compacting == null)
 overlappingSSTables = 
Refs.tryRef(Collections.emptyList());
 else
 overlappingSSTables = 
cfs.getAndReferenceOverlappingLiveSSTables(compacting);
@@ -358,6 +354,8 @@ public class CompactionController implements AutoCloseable
  * This strategy can retain for a long time a lot of sstables on disk (see 
CASSANDRA-13418) so this option
  * control whether or not this check should be ignored.
  *
+ * Do NOT call this method in the CompactionController constructor
+ *
  * @return false by default
  */
 protected boolean ignoreOverlaps()
diff --git 
a/test/unit/org/apache/cassandra/db/compaction/CompactionControllerTest.java 
b/test/unit/org/apache/cassandra/db/compaction/CompactionControllerTest.java
index 052206e685..aa95ba56fb 100644
--- a/test/unit/org/apache/cassandra/db/compaction/CompactionControllerTest.java
+++ b/test/unit/org/apache/cassandra/db/compaction/CompactionControllerTest.java
@@ -19,12 +19,19 @@
 package org.apache.cassandra.db.compaction;
 
 import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.Map;
 import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
 import java.util.function.Predicate;
 
+import com.google.common.collect.Iterables;
 import com.google.common.collect.Sets;
+import com.google.common.util.concurrent.Uninterruptibles;
 import org.junit.BeforeClass;
 import org.junit.Test;
+import org.junit.runner.RunWith;
 
 import org.apache.cassandra.SchemaLoader;
 import org.apache.cassandra.Util;
@@ -41,17 +48,27 @@ import org.apache.cassandra.io.sstable.format.SSTableReader;
 import org.apache.cassandra.schema.KeyspaceParams;
 import org.apache.cassandra.utils.ByteBufferUtil;
 import org.apache.cassandra.utils.FBUtilities;
+import org.jboss.byteman.contrib.bmunit.BMRule;
+import org.jboss.byteman.contrib.bmunit.BMRules;
+import org.jboss.byteman.contrib.bmunit.BMUnitRunner;
 
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertFalse;
 import static org.junit.Assert.assertTrue;
 import static org.junit.Assert.assertNotNull;
 
+@RunWith(BMUnitRunner.clas

[cassandra] branch trunk updated (d3a7387836 -> 724c616f58)

2023-08-14 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git


from d3a7387836 Merge branch 'cassandra-5.0' into trunk
 new 9f187fafbd Revert checks for negative timestamp
 new 724c616f58 Merge branch 'cassandra-5.0' into trunk

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 src/java/org/apache/cassandra/db/DeletionTime.java |  7 ---
 src/java/org/apache/cassandra/db/rows/Cell.java|  2 --
 .../org/apache/cassandra/db/rows/UnfilteredSerializer.java |  1 -
 test/unit/org/apache/cassandra/db/ReadCommandTest.java | 10 +-
 4 files changed, 9 insertions(+), 11 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 01/01: Merge branch 'cassandra-5.0' into trunk

2023-08-14 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 724c616f58b79d09769d00adb23687be3f50dcdd
Merge: d3a7387836 9f187fafbd
Author: Branimir Lambov 
AuthorDate: Fri Aug 11 00:31:39 2023 +0300

Merge branch 'cassandra-5.0' into trunk

 src/java/org/apache/cassandra/db/DeletionTime.java |  7 ---
 src/java/org/apache/cassandra/db/rows/Cell.java|  2 --
 .../org/apache/cassandra/db/rows/UnfilteredSerializer.java |  1 -
 test/unit/org/apache/cassandra/db/ReadCommandTest.java | 10 +-
 4 files changed, 9 insertions(+), 11 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch cassandra-5.0 updated: Revert checks for negative timestamp

2023-08-14 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch cassandra-5.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/cassandra-5.0 by this push:
 new 9f187fafbd Revert checks for negative timestamp
9f187fafbd is described below

commit 9f187fafbd47a62be80cae21796178641b5b3627
Author: Branimir Lambov 
AuthorDate: Wed Aug 9 11:40:04 2023 +0300

Revert checks for negative timestamp

Contrary to expectations, we do permit negative timestamps.
This reverts checks that treat such timestamps as corruption
introduced in CASSANDRA-18676.

patch by Branimir Lambov; reviewed by Ekaterina Dimitrova for 
CASSANDRA-18735
---
 src/java/org/apache/cassandra/db/DeletionTime.java |  7 ---
 src/java/org/apache/cassandra/db/rows/Cell.java|  2 --
 .../org/apache/cassandra/db/rows/UnfilteredSerializer.java |  1 -
 test/unit/org/apache/cassandra/db/ReadCommandTest.java | 10 +-
 4 files changed, 9 insertions(+), 11 deletions(-)

diff --git a/src/java/org/apache/cassandra/db/DeletionTime.java 
b/src/java/org/apache/cassandra/db/DeletionTime.java
index 45a5bf841a..5970fbb042 100644
--- a/src/java/org/apache/cassandra/db/DeletionTime.java
+++ b/src/java/org/apache/cassandra/db/DeletionTime.java
@@ -115,13 +115,14 @@ public class DeletionTime implements 
Comparable, IMeasurableMemory
 }
 
 /**
- * Check if this deletion time is valid - markedForDeleteAt can only 
negative if the deletion is LIVE.
- * localDeletionTime is not checked as it is stored as an unsigned int and 
cannot be negative.
+ * Check if this deletion time is valid. This is always true, because
+ * - as we permit negative timestamps, markedForDeleteAt can be negative.
+ * - localDeletionTime is stored as an unsigned int and cannot be negative.
  * @return true if it is valid
  */
 public boolean validate()
 {
-return markedForDeleteAt >= 0 || isLive();
+return true;
 }
 
 @Override
diff --git a/src/java/org/apache/cassandra/db/rows/Cell.java 
b/src/java/org/apache/cassandra/db/rows/Cell.java
index fffcca821a..d60fdda5a0 100644
--- a/src/java/org/apache/cassandra/db/rows/Cell.java
+++ b/src/java/org/apache/cassandra/db/rows/Cell.java
@@ -342,8 +342,6 @@ public abstract class Cell extends ColumnData
 }
 }
 
-if (timestamp < 0)
-throw new IOException("Invalid negative timestamp: " + 
timestamp);
 if (ttl < 0)
 throw new IOException("Invalid TTL: " + ttl);
 localDeletionTime = decodeLocalDeletionTime(localDeletionTime, 
ttl, helper);
diff --git a/src/java/org/apache/cassandra/db/rows/UnfilteredSerializer.java 
b/src/java/org/apache/cassandra/db/rows/UnfilteredSerializer.java
index cfbcad177c..e31426bf68 100644
--- a/src/java/org/apache/cassandra/db/rows/UnfilteredSerializer.java
+++ b/src/java/org/apache/cassandra/db/rows/UnfilteredSerializer.java
@@ -589,7 +589,6 @@ public class UnfilteredSerializer
 if (hasTimestamp)
 {
 long timestamp = header.readTimestamp(in);
-assert timestamp >= 0 : "Invalid negative timestamp " + 
timestamp;
 int ttl = hasTTL ? header.readTTL(in) : LivenessInfo.NO_TTL;
 assert ttl >= 0 : "Invalid TTL " + ttl;
 long localDeletionTime = hasTTL ? 
header.readLocalDeletionTime(in) : LivenessInfo.NO_EXPIRATION_TIME;
diff --git a/test/unit/org/apache/cassandra/db/ReadCommandTest.java 
b/test/unit/org/apache/cassandra/db/ReadCommandTest.java
index 16de3f061c..f1efe97722 100644
--- a/test/unit/org/apache/cassandra/db/ReadCommandTest.java
+++ b/test/unit/org/apache/cassandra/db/ReadCommandTest.java
@@ -798,21 +798,21 @@ public class ReadCommandTest
 long nowInSec = FBUtilities.nowInSeconds();
 
 // A simple tombstone
-new RowUpdateBuilder(cfs.metadata(), 100, 
keys[0]).clustering("cc").delete("a").build().apply();
+new RowUpdateBuilder(cfs.metadata(), 0, 
keys[0]).clustering("cc").delete("a").build().apply();
 
 // Collection with an associated complex deletion
-PartitionUpdate.SimpleBuilder builder = 
PartitionUpdate.simpleBuilder(cfs.metadata(), keys[1]).timestamp(100);
+PartitionUpdate.SimpleBuilder builder = 
PartitionUpdate.simpleBuilder(cfs.metadata(), keys[1]).timestamp(0);
 builder.row("cc").add("c", ImmutableSet.of("element1", "element2"));
 builder.buildAsMutation().apply();
 
 // RangeTombstone and a row (not covered by the RT). The row contains 
a regular tombstone which will not be
 // purged. This is to prevent the partition from being fully purged 
and rem

[cassandra] branch trunk updated: Fix data corruption handling issues

2023-07-26 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/trunk by this push:
 new a565711056 Fix data corruption handling issues
a565711056 is described below

commit a565711056c859398d0b26081b46e71d2076de1d
Author: Branimir Lambov 
AuthorDate: Mon Jul 24 12:23:27 2023 +0300

Fix data corruption handling issues

Treat AssertionError as corruption, assert positive deletion timestamps and
TTLs and treat localDeletionTime < TTL as invalid.

patch by Branimir Lambov; reviewed by Berenguer Blasi for CASSANDRA-18676
---
 src/java/org/apache/cassandra/db/DeletionTime.java |  8 ++---
 src/java/org/apache/cassandra/db/rows/Cell.java| 41 +-
 .../cassandra/db/rows/UnfilteredSerializer.java|  8 ++---
 .../io/sstable/SSTableIdentityIterator.java|  4 +--
 .../org/apache/cassandra/db/ReadCommandTest.java   | 10 +++---
 .../CorruptedSSTablesCompactionsTest.java  |  2 +-
 6 files changed, 49 insertions(+), 24 deletions(-)

diff --git a/src/java/org/apache/cassandra/db/DeletionTime.java 
b/src/java/org/apache/cassandra/db/DeletionTime.java
index a0450a0fd1..45a5bf841a 100644
--- a/src/java/org/apache/cassandra/db/DeletionTime.java
+++ b/src/java/org/apache/cassandra/db/DeletionTime.java
@@ -70,8 +70,7 @@ public class DeletionTime implements 
Comparable, IMeasurableMemory
 
 private DeletionTime(long markedForDeleteAt, long localDeletionTime)
 {
-this.markedForDeleteAt = markedForDeleteAt;
-this.localDeletionTimeUnsignedInteger = 
Cell.deletionTimeLongToUnsignedInteger(localDeletionTime);
+this(markedForDeleteAt, 
Cell.deletionTimeLongToUnsignedInteger(localDeletionTime));
 }
 
 private DeletionTime(long markedForDeleteAt, int 
localDeletionTimeUnsignedInteger)
@@ -116,12 +115,13 @@ public class DeletionTime implements 
Comparable, IMeasurableMemory
 }
 
 /**
- * check if this deletion time is valid - localDeletionTime can never be 
negative
+ * Check if this deletion time is valid - markedForDeleteAt can only 
negative if the deletion is LIVE.
+ * localDeletionTime is not checked as it is stored as an unsigned int and 
cannot be negative.
  * @return true if it is valid
  */
 public boolean validate()
 {
-return true;
+return markedForDeleteAt >= 0 || isLive();
 }
 
 @Override
diff --git a/src/java/org/apache/cassandra/db/rows/Cell.java 
b/src/java/org/apache/cassandra/db/rows/Cell.java
index 9850d08af3..fffcca821a 100644
--- a/src/java/org/apache/cassandra/db/rows/Cell.java
+++ b/src/java/org/apache/cassandra/db/rows/Cell.java
@@ -21,12 +21,14 @@ import java.io.IOException;
 import java.nio.ByteBuffer;
 import java.util.Comparator;
 
-import org.apache.cassandra.config.*;
-import org.apache.cassandra.db.*;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.DeletionPurger;
+import org.apache.cassandra.db.LivenessInfo;
+import org.apache.cassandra.db.SerializationHeader;
 import org.apache.cassandra.db.marshal.ValueAccessor;
+import org.apache.cassandra.io.util.DataInputPlus;
 import org.apache.cassandra.io.util.DataOutputPlus;
 import org.apache.cassandra.net.MessagingService;
-import org.apache.cassandra.io.util.DataInputPlus;
 import org.apache.cassandra.schema.ColumnMetadata;
 import org.apache.cassandra.utils.CassandraUInt;
 import org.apache.cassandra.utils.memory.ByteBufferCloner;
@@ -213,6 +215,29 @@ public abstract class Cell extends ColumnData
 
 protected abstract int localDeletionTimeAsUnsignedInt();
 
+/**
+ * Handle unsigned encoding and potentially invalid localDeletionTime.
+ */
+public static long decodeLocalDeletionTime(long localDeletionTime, int 
ttl, DeserializationHelper helper)
+{
+if (localDeletionTime >= ttl)
+return localDeletionTime;   // fast path, positive and valid 
signed 32-bit integer
+
+if (localDeletionTime < 0)
+{
+// Overflown signed int, decode to long. The result is guaranteed 
> ttl (and any signed int)
+return helper.version < MessagingService.VERSION_50
+   ? INVALID_DELETION_TIME
+   : deletionTimeUnsignedIntegerToLong((int) 
localDeletionTime);
+}
+
+if (ttl == LivenessInfo.EXPIRED_LIVENESS_TTL)
+return localDeletionTime;   // ttl is already expired, 
localDeletionTime is valid
+else
+return INVALID_DELETION_TIME;  // Invalid as it can't occur 
without corruption and would cause negative
+   // timestamp on expiry.
+}
+
 /**
  * The serialization format for cell is:
  * [ flags ][ timestamp ][ deletion time ][ttl][ path size ][ 
path ][ value size ][ value ]

[cassandra-dtest] branch trunk updated: Permit B as a bytes unit in compaction_test

2023-07-20 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra-dtest.git


The following commit(s) were added to refs/heads/trunk by this push:
 new 6bffd00e Permit B as a bytes unit in compaction_test
6bffd00e is described below

commit 6bffd00ea003677a590b8d6c8dee39a548a20cb9
Author: Branimir Lambov 
AuthorDate: Tue Jul 18 14:34:59 2023 +0300

Permit B as a bytes unit in compaction_test

patch by Branimir Lambov; reviewed by Brandon Williams for CASSANDRA-18397
---
 compaction_test.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/compaction_test.py b/compaction_test.py
index f7b27078..c5d90560 100644
--- a/compaction_test.py
+++ b/compaction_test.py
@@ -286,10 +286,11 @@ class TestCompaction(Tester):
 "MB": 1,
 "MiB": 1,
 "KiB": 1. / 1024,
-"GiB": 1024
+"GiB": 1024,
+"B": 1. / (1024 * 1024),
 }
 
-units = ['MB'] if cluster.version() < LooseVersion('3.6') else ['KiB', 
'MiB', 'GiB']
+units = ['MB'] if cluster.version() < LooseVersion('3.6') else ['B', 
'KiB', 'MiB', 'GiB']
 assert found_units in units
 
 logger.debug(avgthroughput)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch trunk updated: Fix incorrect value of preload flag for opening BTI partition indexes

2023-07-20 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/trunk by this push:
 new e9198d6a66 Fix incorrect value of preload flag for opening BTI 
partition indexes
e9198d6a66 is described below

commit e9198d6a660c96c21fe1c5bf0bbf19fbfc619189
Author: Branimir Lambov 
AuthorDate: Tue Jul 18 23:45:08 2023 +0300

Fix incorrect value of preload flag for opening BTI partition indexes

patch by Branimir Lambov; reviewed by Jacek Lewandowski for CASSANDRA-18677
---
 .../format/bti/BtiTableReaderLoadingBuilder.java   |  2 +-
 .../io/sstable/format/bti/LoadingBuilderTest.java  | 96 ++
 2 files changed, 97 insertions(+), 1 deletion(-)

diff --git 
a/src/java/org/apache/cassandra/io/sstable/format/bti/BtiTableReaderLoadingBuilder.java
 
b/src/java/org/apache/cassandra/io/sstable/format/bti/BtiTableReaderLoadingBuilder.java
index 0e4a74de5a..3128f06a24 100644
--- 
a/src/java/org/apache/cassandra/io/sstable/format/bti/BtiTableReaderLoadingBuilder.java
+++ 
b/src/java/org/apache/cassandra/io/sstable/format/bti/BtiTableReaderLoadingBuilder.java
@@ -122,7 +122,7 @@ public class BtiTableReaderLoadingBuilder extends 
SortedTableReaderLoadingBuilde
 
 if (builder.getComponents().contains(Components.PARTITION_INDEX))
 {
-
builder.setPartitionIndex(openPartitionIndex(builder.getFilter().isInformative()));
+
builder.setPartitionIndex(openPartitionIndex(!builder.getFilter().isInformative()));
 if (builder.getFirst() == null || builder.getLast() == null)
 {
 builder.setFirst(builder.getPartitionIndex().firstKey());
diff --git 
a/test/unit/org/apache/cassandra/io/sstable/format/bti/LoadingBuilderTest.java 
b/test/unit/org/apache/cassandra/io/sstable/format/bti/LoadingBuilderTest.java
new file mode 100644
index 00..a56ca8e537
--- /dev/null
+++ 
b/test/unit/org/apache/cassandra/io/sstable/format/bti/LoadingBuilderTest.java
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.io.sstable.format.bti;
+
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.junit.Assume;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+
+import org.apache.cassandra.cql3.CQLTester;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.io.sstable.Descriptor;
+import org.apache.cassandra.io.sstable.format.SSTableReader;
+import org.apache.cassandra.io.util.File;
+import org.jboss.byteman.contrib.bmunit.BMRule;
+import org.jboss.byteman.contrib.bmunit.BMUnitRunner;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+@RunWith(BMUnitRunner.class)
+public class LoadingBuilderTest extends CQLTester
+{
+public static Map preloadsMap = new ConcurrentHashMap<>();
+
+@Test
+@BMRule(name = "Save preload argument of PartitionIndex initialization",
+targetClass = 
"org.apache.cassandra.io.sstable.format.bti.PartitionIndex",
+targetMethod = "load(org.apache.cassandra.io.util.FileHandle, 
org.apache.cassandra.dht.IPartitioner, boolean)",
+action = 
"org.apache.cassandra.io.sstable.format.bti.LoadingBuilderTest.preloadsMap.put($1.path(),
 $3)")
+public void testPreloadFlag()
+{
+Assume.assumeTrue(BtiFormat.isSelected());
+testPreloadFlag(false);
+testPreloadFlag(true);
+}
+
+private void testPreloadFlag(boolean disableBloomFilter)
+{
+createTable("CREATE TABLE %s (k int PRIMARY KEY, v int) WITH 
bloom_filter_fp_chance = " +
+(disableBloomFilter ? "1" : "0.01"));
+for (int i = 0; i < 100; ++i)
+execute("INSERT INTO %s (k, v) VALUES (?, ?)", i, i);
+flush();
+
+ColumnFamilyStore cfs = getCur

[cassandra] 02/02: Rename the byte-comparable translation version to OSS50

2023-05-29 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit fad1f7457032544ab6a7b40c5d38ecb8b25899bb
Author: Branimir Lambov 
AuthorDate: Fri Apr 21 11:52:10 2023 +0300

Rename the byte-comparable translation version to OSS50

Also fix some minor issues in ByteComparable.md

patch by Branimir Lambov and Jacek Lewandowski; reviewed by Caleb Rackliffe 
and Maxwell Guo for CASSANDRA-18398
---
 .../apache/cassandra/db/ClusteringComparator.java  |   6 +-
 .../org/apache/cassandra/db/marshal/TupleType.java |   4 +-
 .../apache/cassandra/db/memtable/TrieMemtable.java |   2 +-
 src/java/org/apache/cassandra/db/tries/Trie.java   |   2 +-
 src/java/org/apache/cassandra/io/tries/Walker.java |   2 +-
 .../utils/bytecomparable/ByteComparable.java   |   2 +-
 .../utils/bytecomparable/ByteComparable.md | 370 +++--
 .../AbstractTypeByteSourceDecodingBench.java   |   4 +-
 .../test/microbench/tries/ComparisonReadBench.java |  14 +-
 .../apache/cassandra/db/tries/SlicedTrieTest.java  |  12 +-
 .../bytecomparable/AbstractTypeByteSourceTest.java |   8 +-
 .../bytecomparable/ByteSourceComparisonTest.java   |  30 +-
 .../bytecomparable/ByteSourceConversionTest.java   |   2 +-
 .../bytecomparable/ByteSourceInverseTest.java  |   4 +-
 .../bytecomparable/ByteSourceSequenceTest.java |   2 +-
 .../bytecomparable/DecoratedKeyByteSourceTest.java |   2 +-
 16 files changed, 242 insertions(+), 224 deletions(-)

diff --git a/src/java/org/apache/cassandra/db/ClusteringComparator.java 
b/src/java/org/apache/cassandra/db/ClusteringComparator.java
index 316e6f71cb..d007c5d360 100644
--- a/src/java/org/apache/cassandra/db/ClusteringComparator.java
+++ b/src/java/org/apache/cassandra/db/ClusteringComparator.java
@@ -351,7 +351,7 @@ public class ClusteringComparator implements 
Comparator
  */
 public  Clustering clusteringFromByteComparable(ValueAccessor 
accessor, ByteComparable comparable)
 {
-ByteComparable.Version version = ByteComparable.Version.OSS42;
+ByteComparable.Version version = ByteComparable.Version.OSS50;
 ByteSource.Peekable orderedBytes = 
ByteSource.peekable(comparable.asComparableBytes(version));
 
 // First check for special cases (partition key only, static 
clustering) that can do without buffers.
@@ -415,7 +415,7 @@ public class ClusteringComparator implements 
Comparator
   ByteComparable 
comparable,
   boolean isEnd)
 {
-ByteComparable.Version version = ByteComparable.Version.OSS42;
+ByteComparable.Version version = ByteComparable.Version.OSS50;
 ByteSource.Peekable orderedBytes = 
ByteSource.peekable(comparable.asComparableBytes(version));
 
 int sep = orderedBytes.next();
@@ -472,7 +472,7 @@ public class ClusteringComparator implements 
Comparator
  */
 public  ClusteringBoundary 
boundaryFromByteComparable(ValueAccessor accessor, ByteComparable comparable)
 {
-ByteComparable.Version version = ByteComparable.Version.OSS42;
+ByteComparable.Version version = ByteComparable.Version.OSS50;
 ByteSource.Peekable orderedBytes = 
ByteSource.peekable(comparable.asComparableBytes(version));
 
 int sep = orderedBytes.next();
diff --git a/src/java/org/apache/cassandra/db/marshal/TupleType.java 
b/src/java/org/apache/cassandra/db/marshal/TupleType.java
index 328afb482a..79f921920d 100644
--- a/src/java/org/apache/cassandra/db/marshal/TupleType.java
+++ b/src/java/org/apache/cassandra/db/marshal/TupleType.java
@@ -209,7 +209,7 @@ public class TupleType extends AbstractType
 {
 case LEGACY:
 return asComparableBytesLegacy(accessor, data);
-case OSS42:
+case OSS50:
 return asComparableBytesNew(accessor, data, version);
 default:
 throw new AssertionError();
@@ -255,7 +255,7 @@ public class TupleType extends AbstractType
 @Override
 public  V fromComparableBytes(ValueAccessor accessor, 
ByteSource.Peekable comparableBytes, ByteComparable.Version version)
 {
-assert version == ByteComparable.Version.OSS42; // Reverse translation 
is not supported for the legacy version.
+assert version == ByteComparable.Version.OSS50; // Reverse translation 
is not supported for the legacy version.
 if (comparableBytes == null)
 return accessor.empty();
 
diff --git a/src/java/org/apache/cassandra/db/memtable/TrieMemtable.java 
b/src/java/org/apache/cassandra/db/memtable/TrieMemtable.java
index f1401f76d8..e8d597825e 100644
--- a/src/java/org/apache/cassandra/db/memtable/TrieMemtable.java
+++ b/src/java/org/apache/cassandra/db/memtable/TrieMemtable.java
@@ -115,7 +115,7 @@ public class

[cassandra] branch trunk updated (f0abf77e64 -> fad1f74570)

2023-05-29 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git


from f0abf77e64 Merge branch 'cassandra-4.1' into trunk
 new f16fb6765b Implementation of the trie-indexed SSTable format (BTI) as 
described in CEP-25.
 new fad1f74570 Rename the byte-comparable translation version to OSS50

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|1 +
 NEWS.txt   |6 +-
 conf/cassandra.yaml|   49 +-
 src/java/org/apache/cassandra/config/Config.java   |2 +-
 .../cassandra/config/DatabaseDescriptor.java   |   13 +-
 .../apache/cassandra/db/ClusteringComparator.java  |6 +-
 src/java/org/apache/cassandra/db/DecoratedKey.java |2 +-
 src/java/org/apache/cassandra/db/DeletionTime.java |   10 +
 .../org/apache/cassandra/db/lifecycle/LogFile.java |6 +-
 .../db/marshal/PartitionerDefinedOrder.java|2 +-
 .../org/apache/cassandra/db/marshal/TupleType.java |4 +-
 .../apache/cassandra/db/memtable/TrieMemtable.java |2 +-
 .../LazilyInitializedUnfilteredRowIterator.java|   25 +-
 src/java/org/apache/cassandra/db/tries/Trie.java   |2 +-
 .../io/sstable/AbstractRowIndexEntry.java  |4 +-
 .../cassandra/io/sstable/SSTableReadsListener.java |2 +-
 .../org/apache/cassandra/io/sstable/SSTable_API.md |2 +-
 .../io/sstable/format/AbstractSSTableFormat.java   |6 +
 .../cassandra/io/sstable/format/SSTableFormat.java |4 -
 .../cassandra/io/sstable/format/SSTableReader.java |   18 +-
 .../io/sstable/format/SSTableReaderWithFilter.java |2 +-
 .../io/sstable/format/SSTableScanner.java  |  299 ++
 .../format/SortedTableReaderLoadingBuilder.java|   69 ++
 .../io/sstable/format/SortedTableVerifier.java |  172 +++-
 .../cassandra/io/sstable/format/big/BigFormat.java |   31 +-
 .../format/big/BigFormatPartitionWriter.java   |6 +-
 .../format/big/BigSSTableReaderLoadingBuilder.java |   34 +-
 .../io/sstable/format/big/BigTableKeyReader.java   |4 +-
 .../io/sstable/format/big/BigTableReader.java  |   12 +-
 .../io/sstable/format/big/BigTableScanner.java |  307 ++
 .../io/sstable/format/big/BigTableVerifier.java|  244 +
 .../io/sstable/format/big/IndexState.java  |   14 +-
 .../io/sstable/format/big/RowIndexEntry.java   |   10 +-
 .../cassandra/io/sstable/format/bti/BtiFormat.java |  453 +
 .../cassandra/io/sstable/format/bti/BtiFormat.md   | 1010 
 .../format/bti/BtiFormatPartitionWriter.java   |  125 +++
 .../io/sstable/format/bti/BtiTableReader.java  |  534 +++
 .../format/bti/BtiTableReaderLoadingBuilder.java   |  208 
 .../io/sstable/format/bti/BtiTableScanner.java |  140 +++
 .../io/sstable/format/bti/BtiTableScrubber.java|  315 ++
 .../io/sstable/format/bti/BtiTableVerifier.java|   39 +
 .../io/sstable/format/bti/BtiTableWriter.java  |  428 +
 .../io/sstable/format/bti/PartitionIndex.java  |  454 +
 .../sstable/format/bti/PartitionIndexBuilder.java  |  234 +
 .../io/sstable/format/bti/PartitionIndexEarly.java |   50 +
 .../io/sstable/format/bti/PartitionIterator.java   |  254 +
 .../io/sstable/format/bti/RowIndexReader.java  |  193 
 .../format/bti/RowIndexReverseIterator.java|   66 ++
 .../io/sstable/format/bti/RowIndexWriter.java  |  123 +++
 .../io/sstable/format/bti/SSTableIterator.java |  112 +++
 .../format/bti/SSTableReversedIterator.java|  295 ++
 .../io/sstable/format/bti/ScrubIterator.java   |   92 ++
 .../sstable/format/bti/ScrubPartitionIterator.java |   54 ++
 .../io/sstable/format/bti/TrieIndexEntry.java  |  119 +++
 .../io/sstable/metadata/StatsMetadata.java |   52 +-
 .../tries/IncrementalDeepTrieWriterPageAware.java  |  411 
 .../cassandra/io/tries/IncrementalTrieWriter.java  |   82 ++
 .../io/tries/IncrementalTrieWriterBase.java|  266 ++
 .../io/tries/IncrementalTrieWriterPageAware.java   |  467 +
 .../io/tries/IncrementalTrieWriterSimple.java  |  130 +++
 .../cassandra/io/tries/ReverseValueIterator.java   |  221 +
 .../cassandra/io/tries/SerializationNode.java  |   55 ++
 .../org/apache/cassandra/io/tries/TrieNode.java|  987 +++
 .../apache/cassandra/io/tries/TrieSerializer.java  |   28 +
 .../apache/cassandra/io/tries/ValueIterator.java   |  230 +
 src/java/org/apache/cassandra/io/tries/Walker.java |  392 
 .../io/util/BufferedDataOutputStreamPlus.java  |4 +-
 .../apache/cassandra/io/util

[cassandra] branch trunk updated (36e16ee3c9 -> 930f141fa0)

2022-11-16 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git


from 36e16ee3c9 Adding endpoint verification option to 
client_encryption_options
 new e08c7a6295 Avoid schema mismatch problems on memtable API 
misconfiguration
 new 9f58d76f38 Avoid schema mismatch problems on memtable API 
misconfiguration
 new 930f141fa0 Merge branch 'cassandra-4.1' into trunk

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/cassandra/db/memtable/Memtable_API.md   |  7 +-
 .../apache/cassandra/schema/MemtableParams.java| 18 ++
 .../apache/cassandra/schema/SchemaKeyspace.java|  4 +-
 .../cassandra/distributed/test/AlterTest.java  | 75 ++
 4 files changed, 100 insertions(+), 4 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch cassandra-4.1 updated: Avoid schema mismatch problems on memtable API misconfiguration

2022-11-16 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch cassandra-4.1
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/cassandra-4.1 by this push:
 new 9f58d76f38 Avoid schema mismatch problems on memtable API 
misconfiguration
9f58d76f38 is described below

commit 9f58d76f3841864be11f5b9c4534027451328569
Author: Branimir Lambov 
AuthorDate: Mon Nov 14 13:59:05 2022 +0200

Avoid schema mismatch problems on memtable API misconfiguration

patch by Branimir Lambov; reviewed by Caleb Rackliffe for CASSANDRA-18040
---
 .../apache/cassandra/db/memtable/Memtable_API.md   |  7 +-
 .../apache/cassandra/schema/MemtableParams.java| 18 +
 .../apache/cassandra/schema/SchemaKeyspace.java|  4 +-
 .../cassandra/distributed/test/AlterTest.java  | 82 ++
 4 files changed, 107 insertions(+), 4 deletions(-)

diff --git a/src/java/org/apache/cassandra/db/memtable/Memtable_API.md 
b/src/java/org/apache/cassandra/db/memtable/Memtable_API.md
index 39f9b201af..8a582e3ad9 100644
--- a/src/java/org/apache/cassandra/db/memtable/Memtable_API.md
+++ b/src/java/org/apache/cassandra/db/memtable/Memtable_API.md
@@ -72,9 +72,10 @@ ALTER TABLE ... WITH memtable = 'default';
 ```
 
 The memtable configuration selection is per table, i.e. it will be propagated 
to all nodes in the cluster. If some nodes
-do not have a definition for that configuration, cannot instantiate the class, 
or are still on a version of Cassandra
-before 4.1, they will reject the schema change. We therefore recommend using a 
separate `ALTER` statement to change a
-table's memtable implementation; upgrading all nodes to 4.1 or later is 
required to use the API.
+do not have a definition for that configuration or cannot instantiate the 
class, they will log an error and fall 
+back to the default memtable configuration to avoid schema disagreements. 
However, if some nodes are still on a version 
+of Cassandra before 4.1, they will reject the schema change. We therefore 
recommend using a separate `ALTER` statement 
+to change a table's memtable implementation; upgrading all nodes to 4.1 or 
later is required to use the API.
 
 As additional safety when first deploying an alternative implementation to a 
production cluster, one may consider
 first deploying a remapped `default` configuration to all nodes in the 
cluster, switching the schema to reference
diff --git a/src/java/org/apache/cassandra/schema/MemtableParams.java 
b/src/java/org/apache/cassandra/schema/MemtableParams.java
index a3f1bb2323..3470b7ac8d 100644
--- a/src/java/org/apache/cassandra/schema/MemtableParams.java
+++ b/src/java/org/apache/cassandra/schema/MemtableParams.java
@@ -28,6 +28,8 @@ import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Objects;
 import com.google.common.collect.ImmutableMap;
 
+import org.slf4j.LoggerFactory;
+
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.config.InheritingClass;
 import org.apache.cassandra.config.ParameterizedClass;
@@ -110,6 +112,22 @@ public final class MemtableParams
 }
 }
 
+public static MemtableParams getWithFallback(String key)
+{
+try
+{
+return get(key);
+}
+catch (ConfigurationException e)
+{
+LoggerFactory.getLogger(MemtableParams.class).error("Invalid 
memtable configuration \"" + key + "\" in schema. " +
+"Falling back 
to default to avoid schema mismatch.\n" +
+"Please ensure 
the correct definition is given in cassandra.yaml.",
+e);
+return new MemtableParams(DEFAULT.factory(), key);
+}
+}
+
 @VisibleForTesting
 static Map expandDefinitions(Map memtableConfigurations)
 {
diff --git a/src/java/org/apache/cassandra/schema/SchemaKeyspace.java 
b/src/java/org/apache/cassandra/schema/SchemaKeyspace.java
index 33d2b7d6c8..b4f27830ef 100644
--- a/src/java/org/apache/cassandra/schema/SchemaKeyspace.java
+++ b/src/java/org/apache/cassandra/schema/SchemaKeyspace.java
@@ -960,7 +960,9 @@ public final class SchemaKeyspace
   .comment(row.getString("comment"))
   
.compaction(CompactionParams.fromMap(row.getFrozenTextMap("compaction")))
   
.compression(CompressionParams.fromMap(row.getFrozenTextMap("compression")))
-  .memtable(MemtableParams.get(row.has("memtable") ? 
row.getString("memtable") : null)) // memtable column was introduced in 4.1
+  
.memtable(MemtableParams.getWithFallback(row.

[cassandra] 02/02: Merge branch 'cassandra-4.1' into trunk

2022-11-16 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 930f141fa0f203df020d14903548e72aae2743f2
Merge: e08c7a6295 9f58d76f38
Author: Branimir Lambov 
AuthorDate: Wed Nov 16 10:30:39 2022 +0200

Merge branch 'cassandra-4.1' into trunk



-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 01/02: Avoid schema mismatch problems on memtable API misconfiguration

2022-11-16 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit e08c7a6295eed716e9973fbd809dfca32d68a3e8
Author: Branimir Lambov 
AuthorDate: Mon Nov 14 13:59:05 2022 +0200

Avoid schema mismatch problems on memtable API misconfiguration

patch by Branimir Lambov; reviewed by Caleb Rackliffe for CASSANDRA-18040
---
 .../apache/cassandra/db/memtable/Memtable_API.md   |  7 +-
 .../apache/cassandra/schema/MemtableParams.java| 18 ++
 .../apache/cassandra/schema/SchemaKeyspace.java|  4 +-
 .../cassandra/distributed/test/AlterTest.java  | 75 ++
 4 files changed, 100 insertions(+), 4 deletions(-)

diff --git a/src/java/org/apache/cassandra/db/memtable/Memtable_API.md 
b/src/java/org/apache/cassandra/db/memtable/Memtable_API.md
index e5399641c3..70f8f0b605 100644
--- a/src/java/org/apache/cassandra/db/memtable/Memtable_API.md
+++ b/src/java/org/apache/cassandra/db/memtable/Memtable_API.md
@@ -91,9 +91,10 @@ ALTER TABLE ... WITH memtable = 'default';
 ```
 
 The memtable configuration selection is per table, i.e. it will be propagated 
to all nodes in the cluster. If some nodes
-do not have a definition for that configuration, cannot instantiate the class, 
or are still on a version of Cassandra
-before 4.1, they will reject the schema change. We therefore recommend using a 
separate `ALTER` statement to change a
-table's memtable implementation; upgrading all nodes to 4.1 or later is 
required to use the API.
+do not have a definition for that configuration or cannot instantiate the 
class, they will log an error and fall 
+back to the default memtable configuration to avoid schema disagreements. 
However, if some nodes are still on a version 
+of Cassandra before 4.1, they will reject the schema change. We therefore 
recommend using a separate `ALTER` statement 
+to change a table's memtable implementation; upgrading all nodes to 4.1 or 
later is required to use the API.
 
 As additional safety when first deploying an alternative implementation to a 
production cluster, one may consider
 first deploying a remapped `default` configuration to all nodes in the 
cluster, switching the schema to reference
diff --git a/src/java/org/apache/cassandra/schema/MemtableParams.java 
b/src/java/org/apache/cassandra/schema/MemtableParams.java
index a3f1bb2323..3470b7ac8d 100644
--- a/src/java/org/apache/cassandra/schema/MemtableParams.java
+++ b/src/java/org/apache/cassandra/schema/MemtableParams.java
@@ -28,6 +28,8 @@ import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Objects;
 import com.google.common.collect.ImmutableMap;
 
+import org.slf4j.LoggerFactory;
+
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.config.InheritingClass;
 import org.apache.cassandra.config.ParameterizedClass;
@@ -110,6 +112,22 @@ public final class MemtableParams
 }
 }
 
+public static MemtableParams getWithFallback(String key)
+{
+try
+{
+return get(key);
+}
+catch (ConfigurationException e)
+{
+LoggerFactory.getLogger(MemtableParams.class).error("Invalid 
memtable configuration \"" + key + "\" in schema. " +
+"Falling back 
to default to avoid schema mismatch.\n" +
+"Please ensure 
the correct definition is given in cassandra.yaml.",
+e);
+return new MemtableParams(DEFAULT.factory(), key);
+}
+}
+
 @VisibleForTesting
 static Map expandDefinitions(Map memtableConfigurations)
 {
diff --git a/src/java/org/apache/cassandra/schema/SchemaKeyspace.java 
b/src/java/org/apache/cassandra/schema/SchemaKeyspace.java
index cbcba70caa..0604fc1ffd 100644
--- a/src/java/org/apache/cassandra/schema/SchemaKeyspace.java
+++ b/src/java/org/apache/cassandra/schema/SchemaKeyspace.java
@@ -967,7 +967,9 @@ public final class SchemaKeyspace
  
.comment(row.getString("comment"))
  
.compaction(CompactionParams.fromMap(row.getFrozenTextMap("compaction")))
  
.compression(CompressionParams.fromMap(row.getFrozenTextMap("compression")))
- 
.memtable(MemtableParams.get(row.has("memtable") ? row.getString("memtable") : 
null)) // memtable column was introduced in 4.1
+ 
.memtable(MemtableParams.getWithFallback(row.has("memtable")
+

[cassandra] branch cassandra-4.1 updated: Bug fix for WriteTimeoutException when using Paxos v2 with a single datacenter

2022-11-07 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch cassandra-4.1
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/cassandra-4.1 by this push:
 new acda6fad68 Bug fix for WriteTimeoutException when using Paxos v2 with 
a single datacenter
acda6fad68 is described below

commit acda6fad688e95e0a9a34de28e3b61a775e59736
Author: Marianne Lyne Manaog 
AuthorDate: Mon Nov 7 10:33:02 2022 +

Bug fix for WriteTimeoutException when using Paxos v2 with a single 
datacenter

patch by Marianne Lyne Manaog; reviewed by Branimir Lambov and Benedict 
Elliott Smith for CASSANDRA-17999
---
 .../cassandra/service/paxos/PaxosCommit.java   |  2 +-
 .../apache/cassandra/distributed/test/CASTest.java | 25 ++
 2 files changed, 26 insertions(+), 1 deletion(-)

diff --git a/src/java/org/apache/cassandra/service/paxos/PaxosCommit.java 
b/src/java/org/apache/cassandra/service/paxos/PaxosCommit.java
index 246fed7799..4321fc918f 100644
--- a/src/java/org/apache/cassandra/service/paxos/PaxosCommit.java
+++ b/src/java/org/apache/cassandra/service/paxos/PaxosCommit.java
@@ -260,7 +260,7 @@ public class PaxosCommit> ex
  */
 private void response(boolean success, InetAddressAndPort from)
 {
-if (consistencyForCommit.isDatacenterLocal() && 
InOurDc.endpoints().test(from))
+if (consistencyForCommit.isDatacenterLocal() && 
!InOurDc.endpoints().test(from))
 return;
 
 long responses = responsesUpdater.addAndGet(this, success ? 0x1L : 
0x1L);
diff --git 
a/test/distributed/org/apache/cassandra/distributed/test/CASTest.java 
b/test/distributed/org/apache/cassandra/distributed/test/CASTest.java
index 2007f10b2f..8e2b8ac901 100644
--- a/test/distributed/org/apache/cassandra/distributed/test/CASTest.java
+++ b/test/distributed/org/apache/cassandra/distributed/test/CASTest.java
@@ -41,6 +41,7 @@ import org.apache.cassandra.utils.FBUtilities;
 
 import static org.apache.cassandra.distributed.api.ConsistencyLevel.ANY;
 import static org.apache.cassandra.distributed.api.ConsistencyLevel.ONE;
+import static 
org.apache.cassandra.distributed.api.ConsistencyLevel.LOCAL_QUORUM;
 import static org.apache.cassandra.distributed.api.ConsistencyLevel.QUORUM;
 import static org.apache.cassandra.distributed.api.ConsistencyLevel.SERIAL;
 import static org.apache.cassandra.distributed.shared.AssertUtils.assertRows;
@@ -755,4 +756,28 @@ public class CASTest extends CASCommonTestCases
 {
 return THREE_NODES;
 }
+
+/**
+ * Regression test for a bug (CASSANDRA-17999) where a 
WriteTimeoutException is encountered when using Paxos v2 in
+ * an LWT performance test that only has a single datacenter because Paxos 
was still waiting for a response from
+ * another datacenter during the Commit/Acknowledge phase even though we 
were running with LOCAL_SERIAL.
+ *
+ *
+ * This specifically test for the inconsistency described/fixed by 
CASSANDRA-17999.
+ */
+@Test
+public void testWriteTimeoutExceptionUsingPaxosInLwtPerformaceTest() 
throws IOException
+{
+
+THREE_NODES.schemaChange(String.format("CREATE KEYSPACE IF NOT EXISTS 
%s WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'}", 
KEYSPACE));
+
+String tableName = tableName("t");
+String table = KEYSPACE + "." + tableName;
+THREE_NODES.schemaChange("CREATE TABLE " + table + " (k int PRIMARY 
KEY, v int)");
+
+THREE_NODES.coordinator(1).execute("INSERT INTO " + table + " (k, v) 
VALUES (5, 5) IF NOT EXISTS", LOCAL_QUORUM);
+THREE_NODES.coordinator(1).execute("UPDATE " + table + " SET v = 123 
WHERE k = 5 IF EXISTS", LOCAL_QUORUM);
+
+}
+
 }


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 01/01: Merge branch 'cassandra-4.1' into trunk

2022-11-07 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit f236f40fc526dd75db648f4c6cc34d7f33fbd8f3
Merge: 4469f017c0 acda6fad68
Author: Branimir Lambov 
AuthorDate: Mon Nov 7 15:05:16 2022 +0200

Merge branch 'cassandra-4.1' into trunk

 .../cassandra/service/paxos/PaxosCommit.java   |  2 +-
 .../apache/cassandra/distributed/test/CASTest.java | 25 ++
 2 files changed, 26 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch trunk updated (4469f017c0 -> f236f40fc5)

2022-11-07 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git


from 4469f017c0 Merge branch 'cassandra-4.1' into trunk
 new acda6fad68 Bug fix for WriteTimeoutException when using Paxos v2 with 
a single datacenter
 new f236f40fc5 Merge branch 'cassandra-4.1' into trunk

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../cassandra/service/paxos/PaxosCommit.java   |  2 +-
 .../apache/cassandra/distributed/test/CASTest.java | 25 ++
 2 files changed, 26 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 05/05: Renames MemtableTrie to InMemoryTrie

2022-10-21 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 9074ee7ef8e041e1b15116373be0df80b985e3d9
Author: Branimir Lambov 
AuthorDate: Thu Oct 20 12:32:07 2022 +0300

Renames MemtableTrie to InMemoryTrie

patch by Branimir Lambov; reviewed by Andres de la Peña and Caleb Rackliffe 
for CASSANDRA-17240
---
 .../apache/cassandra/db/memtable/TrieMemtable.java | 14 +++
 ...MemtableReadTrie.java => InMemoryReadTrie.java} | 13 +++---
 .../tries/{MemtableTrie.java => InMemoryTrie.java} | 23 ++-
 .../db/tries/{MemtableTrie.md => InMemoryTrie.md}  | 46 +++---
 ...mtableTrie.md.a1.svg => InMemoryTrie.md.a1.svg} |  0
 ...mtableTrie.md.g1.svg => InMemoryTrie.md.g1.svg} |  0
 ...mtableTrie.md.g2.svg => InMemoryTrie.md.g2.svg} |  0
 ...mtableTrie.md.g3.svg => InMemoryTrie.md.g3.svg} |  0
 ...mtableTrie.md.g4.svg => InMemoryTrie.md.g4.svg} |  0
 ...mtableTrie.md.m1.svg => InMemoryTrie.md.m1.svg} |  0
 ...mtableTrie.md.m2.svg => InMemoryTrie.md.m2.svg} |  0
 ...mtableTrie.md.m3.svg => InMemoryTrie.md.m3.svg} |  0
 ...mtableTrie.md.p1.svg => InMemoryTrie.md.p1.svg} |  0
 ...mtableTrie.md.w1.svg => InMemoryTrie.md.w1.svg} |  0
 ...mtableTrie.md.w2.svg => InMemoryTrie.md.w2.svg} |  0
 ...mtableTrie.md.w3.svg => InMemoryTrie.md.w3.svg} |  0
 ...ableTrie.md.wc1.svg => InMemoryTrie.md.wc1.svg} |  0
 ...ableTrie.md.wc2.svg => InMemoryTrie.md.wc2.svg} |  0
 src/java/org/apache/cassandra/db/tries/Trie.md | 14 +++
 .../test/microbench/tries/ComparisonReadBench.java | 10 ++---
 ...ieReadBench.java => InMemoryTrieReadBench.java} | 10 ++---
 ...UnionBench.java => InMemoryTrieUnionBench.java} | 16 
 ...WriteBench.java => InMemoryTrieWriteBench.java} | 22 +--
 .../db/tries/CollectionMergeTrieTest.java  | 22 +--
 ...ieApplyTest.java => InMemoryTrieApplyTest.java} |  2 +-
 ...leTriePutTest.java => InMemoryTriePutTest.java} | 22 +--
 ...TrieTestBase.java => InMemoryTrieTestBase.java} | 34 
 ...adedTest.java => InMemoryTrieThreadedTest.java} |  8 ++--
 .../apache/cassandra/db/tries/MergeTrieTest.java   | 18 -
 .../apache/cassandra/db/tries/SlicedTrieTest.java  | 22 +--
 .../apache/cassandra/db/tries/TrieToDotTest.java   |  4 +-
 .../cassandra/metrics/TrieMemtableMetricsTest.java |  2 +-
 32 files changed, 150 insertions(+), 152 deletions(-)

diff --git a/src/java/org/apache/cassandra/db/memtable/TrieMemtable.java 
b/src/java/org/apache/cassandra/db/memtable/TrieMemtable.java
index 4c59bbba6f..ae3d2e8ec3 100644
--- a/src/java/org/apache/cassandra/db/memtable/TrieMemtable.java
+++ b/src/java/org/apache/cassandra/db/memtable/TrieMemtable.java
@@ -55,7 +55,7 @@ import 
org.apache.cassandra.db.partitions.UnfilteredPartitionIterator;
 import org.apache.cassandra.db.rows.EncodingStats;
 import org.apache.cassandra.db.rows.Row;
 import org.apache.cassandra.db.rows.UnfilteredRowIterator;
-import org.apache.cassandra.db.tries.MemtableTrie;
+import org.apache.cassandra.db.tries.InMemoryTrie;
 import org.apache.cassandra.db.tries.Trie;
 import org.apache.cassandra.dht.AbstractBounds;
 import org.apache.cassandra.dht.Bounds;
@@ -130,7 +130,7 @@ public class TrieMemtable extends AbstractShardedMemtable
 
 /**
  * A merged view of the memtable map. Used for partition range queries and 
flush.
- * For efficiency we serve single partition requests off the shard which 
offers more direct MemtableTrie methods.
+ * For efficiency we serve single partition requests off the shard which 
offers more direct InMemoryTrie methods.
  */
 private final Trie mergedTrie;
 
@@ -214,9 +214,9 @@ public class TrieMemtable extends AbstractShardedMemtable
 
 return colUpdateTimeDelta;
 }
-catch (MemtableTrie.SpaceExhaustedException e)
+catch (InMemoryTrie.SpaceExhaustedException e)
 {
-// This should never happen as {@link 
MemtableTrie#reachedAllocatedSizeThreshold} should become
+// This should never happen as {@link 
InMemoryTrie#reachedAllocatedSizeThreshold} should become
 // true and trigger a memtable switch long before this limit is 
reached.
 throw new IllegalStateException(e);
 }
@@ -443,7 +443,7 @@ public class TrieMemtable extends AbstractShardedMemtable
 // unsafely, meaning that the memtable will not be discarded as long 
as the data is used, or whether the data
 // should be copied on heap for off-heap allocators.
 @VisibleForTesting
-final MemtableTrie data;
+final InMemoryTrie data;
 
 private final ColumnsCollector columnsCollector;
 
@@ -458,14 +458,14 @@ public class TrieMemtable extends AbstractShardedMemtable
 @VisibleForTesting
 MemtableShard(TableMetadataRef

[cassandra] 03/05: MemtableTrie using multiple buffers

2022-10-21 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 562cb26010659830dd1192939ac815a0f6cb3502
Author: Branimir Lambov 
AuthorDate: Thu Nov 11 15:39:21 2021 +0200

MemtableTrie using multiple buffers

The replaces the size doubling and copying required to grow the trie
with an allocation of a new buffer. This improves the cost of expansion
at the expense of increasing individual read and write costs.

patch by Branimir Lambov; reviewed by Jason Rutherglen, Jacek Lewandowski, 
Andres de la Peña and Caleb Rackliffe for CASSANDRA-17240
---
 .../cassandra/db/tries/MemtableReadTrie.java   |  94 --
 .../apache/cassandra/db/tries/MemtableTrie.java| 192 -
 .../cassandra/db/tries/MemtableTrieTestBase.java   |   2 +-
 3 files changed, 192 insertions(+), 96 deletions(-)

diff --git a/src/java/org/apache/cassandra/db/tries/MemtableReadTrie.java 
b/src/java/org/apache/cassandra/db/tries/MemtableReadTrie.java
index 073a99e16e..e9c1e150ec 100644
--- a/src/java/org/apache/cassandra/db/tries/MemtableReadTrie.java
+++ b/src/java/org/apache/cassandra/db/tries/MemtableReadTrie.java
@@ -188,20 +188,67 @@ public class MemtableReadTrie extends Trie
 
 volatile int root;
 
-final UnsafeBuffer buffer;
+/*
+ EXPANDABLE DATA STORAGE
+
+ The tries will need more and more space in buffers and content lists as 
they grow. Instead of using ArrayList-like
+ reallocation with copying, which may be prohibitively expensive for large 
buffers, we use a sequence of
+ buffers/content arrays that double in size on every expansion.
+
+ For a given address x the index of the buffer can be found with the 
following calculation:
+index_of_most_significant_set_bit(x / min_size + 1)
+ (relying on sum (2^i) for i in [0, n-1] == 2^n - 1) which can be 
performed quickly on modern hardware.
+
+ Finding the offset within the buffer is then
+x + min - (min << buffer_index)
+
+ The allocated space starts 256 bytes for the buffer and 16 entries for 
the content list.
+
+ Note that a buffer is not allowed to split 32-byte blocks (code assumes 
same buffer can be used for all bytes
+ inside the block).
+ */
+
+static final int BUF_START_SHIFT = 8;
+static final int BUF_START_SIZE = 1 << BUF_START_SHIFT;
+
+static final int CONTENTS_START_SHIFT = 4;
+static final int CONTENTS_START_SIZE = 1 << CONTENTS_START_SHIFT;
 
-volatile AtomicReferenceArray contentArray;
+final UnsafeBuffer[] buffers;
+final AtomicReferenceArray[] contentArrays;
 
-MemtableReadTrie(UnsafeBuffer buffer, AtomicReferenceArray 
contentArray, int root)
+MemtableReadTrie(UnsafeBuffer[] buffers, AtomicReferenceArray[] 
contentArrays, int root)
 {
-this.buffer = buffer;
-this.contentArray = contentArray;
+this.buffers = buffers;
+this.contentArrays = contentArrays;
 this.root = root;
 }
 
 /*
  Buffer, content list and block management
  */
+int getChunkIdx(int pos, int minChunkShift, int minChunkSize)
+{
+return 31 - minChunkShift - Integer.numberOfLeadingZeros(pos + 
minChunkSize);
+}
+
+int inChunkPointer(int pos, int chunkIndex, int minChunkSize)
+{
+return pos + minChunkSize - (minChunkSize << chunkIndex);
+}
+
+UnsafeBuffer getChunk(int pos)
+{
+int leadBit = getChunkIdx(pos, BUF_START_SHIFT, BUF_START_SIZE);
+return buffers[leadBit];
+}
+
+int inChunkPointer(int pos)
+{
+int leadBit = getChunkIdx(pos, BUF_START_SHIFT, BUF_START_SIZE);
+return inChunkPointer(pos, leadBit, BUF_START_SIZE);
+}
+
 
 /**
  * Pointer offset for a node pointer.
@@ -213,19 +260,25 @@ public class MemtableReadTrie extends Trie
 
 final int getUnsignedByte(int pos)
 {
-return buffer.getByte(pos) & 0xFF;
+return getChunk(pos).getByte(inChunkPointer(pos)) & 0xFF;
 }
 
 final int getUnsignedShort(int pos)
 {
-return buffer.getShort(pos) & 0x;
+return getChunk(pos).getShort(inChunkPointer(pos)) & 0x;
 }
 
-final int getInt(int pos) { return buffer.getInt(pos); }
+final int getInt(int pos)
+{
+return getChunk(pos).getInt(inChunkPointer(pos));
+}
 
 T getContent(int index)
 {
-return contentArray.get(index);
+int leadBit = getChunkIdx(index, CONTENTS_START_SHIFT, 
CONTENTS_START_SIZE);
+int ofs = inChunkPointer(index, leadBit, CONTENTS_START_SIZE);
+AtomicReferenceArray array = contentArrays[leadBit];
+return array.get(ofs);
 }
 
 /*
@@ -512,17 +565,19 @@ public class MemtableReadTrie extends Trie
 return advance();
 
 // Jump directly to the chain'

[cassandra] 01/05: Further improves precision of memtable heap tracking

2022-10-21 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 49e0c61107005b1a83799f7f1e6c0a855d159c29
Author: Branimir Lambov 
AuthorDate: Tue Nov 9 13:59:48 2021 +0200

Further improves precision of memtable heap tracking

Partition key ByteBuffer and columns btree were not taken
into account and some ByteBuffers were not measured correctly.

Also fixes flakes in MemtableSizeTest caused by including
allocator pool in measurements and updates it to test all
memtable allocation types.

patch by Branimir Lambov; reviewed by Ekaterina Dimitrova, Andres de la 
Peña and Caleb Rackliffe for CASSANDRA-17240
---
 .../org/apache/cassandra/db/ArrayClustering.java   |   5 +
 .../org/apache/cassandra/db/BufferClustering.java  |   4 +
 src/java/org/apache/cassandra/db/Columns.java  |   2 +-
 .../db/memtable/AbstractAllocatorMemtable.java |   8 +-
 .../cassandra/db/memtable/AbstractMemtable.java|   2 +
 .../cassandra/db/memtable/SkipListMemtable.java|  14 +-
 .../db/partitions/AtomicBTreePartition.java|   6 +-
 .../org/apache/cassandra/utils/ObjectSizes.java|  13 ++
 .../org/apache/cassandra/utils/btree/BTree.java|   3 +
 .../apache/cassandra/cql3/MemtableSizeTest.java| 157 
 .../cassandra/db/ClusteringHeapSizeTest.java   |  16 ++
 .../{cql3 => db/memtable}/MemtableQuickTest.java   |   4 +-
 .../db/memtable/MemtableSizeHeapBuffersTest.java   |  48 +
 .../memtable/MemtableSizeOffheapBuffersTest.java   |  49 +
 .../memtable/MemtableSizeOffheapObjectsTest.java   |  47 +
 .../db/memtable/MemtableSizeTestBase.java  | 199 +
 .../db/memtable/MemtableSizeUnslabbedTest.java |  47 +
 17 files changed, 459 insertions(+), 165 deletions(-)

diff --git a/src/java/org/apache/cassandra/db/ArrayClustering.java 
b/src/java/org/apache/cassandra/db/ArrayClustering.java
index 53d45e7474..b04910c434 100644
--- a/src/java/org/apache/cassandra/db/ArrayClustering.java
+++ b/src/java/org/apache/cassandra/db/ArrayClustering.java
@@ -18,6 +18,7 @@
 
 package org.apache.cassandra.db;
 
+import org.apache.cassandra.db.marshal.ByteArrayAccessor;
 import org.apache.cassandra.utils.ObjectSizes;
 
 public class ArrayClustering extends AbstractArrayClusteringPrefix implements 
Clustering
@@ -31,6 +32,8 @@ public class ArrayClustering extends 
AbstractArrayClusteringPrefix implements Cl
 
 public long unsharedHeapSize()
 {
+if (this == ByteArrayAccessor.factory.clustering() || this == 
ByteArrayAccessor.factory.staticClustering())
+return 0;
 long arrayRefSize = ObjectSizes.sizeOfArray(values);
 long elementsSize = 0;
 for (int i = 0; i < values.length; i++)
@@ -40,6 +43,8 @@ public class ArrayClustering extends 
AbstractArrayClusteringPrefix implements Cl
 
 public long unsharedHeapSizeExcludingData()
 {
+if (this == ByteArrayAccessor.factory.clustering() || this == 
ByteArrayAccessor.factory.staticClustering())
+return 0;
 return EMPTY_SIZE + ObjectSizes.sizeOfArray(values);
 }
 
diff --git a/src/java/org/apache/cassandra/db/BufferClustering.java 
b/src/java/org/apache/cassandra/db/BufferClustering.java
index e3592e1d6d..ed6d61c82c 100644
--- a/src/java/org/apache/cassandra/db/BufferClustering.java
+++ b/src/java/org/apache/cassandra/db/BufferClustering.java
@@ -43,11 +43,15 @@ public class BufferClustering extends 
AbstractBufferClusteringPrefix implements
 
 public long unsharedHeapSize()
 {
+if (this == Clustering.EMPTY || this == Clustering.STATIC_CLUSTERING)
+return 0;
 return EMPTY_SIZE + ObjectSizes.sizeOnHeapOf(values);
 }
 
 public long unsharedHeapSizeExcludingData()
 {
+if (this == Clustering.EMPTY || this == Clustering.STATIC_CLUSTERING)
+return 0;
 return EMPTY_SIZE + ObjectSizes.sizeOnHeapExcludingData(values);
 }
 
diff --git a/src/java/org/apache/cassandra/db/Columns.java 
b/src/java/org/apache/cassandra/db/Columns.java
index 88185abf7b..8adce341c9 100644
--- a/src/java/org/apache/cassandra/db/Columns.java
+++ b/src/java/org/apache/cassandra/db/Columns.java
@@ -436,7 +436,7 @@ public class Columns extends 
AbstractCollection implements Colle
 if(this == NONE)
 return 0;
 
-return EMPTY_SIZE;
+return EMPTY_SIZE + BTree.sizeOfStructureOnHeap(columns);
 }
 
 @Override
diff --git 
a/src/java/org/apache/cassandra/db/memtable/AbstractAllocatorMemtable.java 
b/src/java/org/apache/cassandra/db/memtable/AbstractAllocatorMemtable.java
index 227cc6dcc2..66556a16e4 100644
--- a/src/java/org/apache/cassandra/db/memtable/AbstractAllocatorMemtable.java
+++ b/src/java/org/apache/cassandra/db/memtable/AbstractAllocatorMemtable.java
@@ -21,6 +21,7 @@ p

[cassandra] branch trunk updated (5f3e60cfd4 -> 9074ee7ef8)

2022-10-21 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git


from 5f3e60cfd4 Merge branch 'cassandra-4.1' into trunk
 new 49e0c61107 Further improves precision of memtable heap tracking
 new 30641ea7b6 Provides the Trie interface with MemtableTrie implementation
 new 562cb26010 MemtableTrie using multiple buffers
 new 7c55c73825 Adds a trie-based memtable implementation
 new 9074ee7ef8 Renames MemtableTrie to InMemoryTrie

The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .build/cassandra-deps-template.xml |4 +
 .build/parent-pom-template.xml |5 +
 .circleci/config-2_1.yml   |   51 +
 .circleci/config-2_1.yml.high_res.patch|8 +-
 .circleci/config-2_1.yml.mid_res.patch |   36 +-
 .circleci/config.yml   |  246 -
 .circleci/config.yml.HIGHRES   |  256 -
 .circleci/config.yml.LOWRES|  256 -
 .circleci/config.yml.MIDRES|  256 -
 .circleci/generate.sh  |1 +
 CHANGES.txt|2 +
 NEWS.txt   |3 +
 build.xml  |   37 +-
 .../config/CassandraRelevantProperties.java|1 +
 .../db/AbstractArrayClusteringPrefix.java  |3 +-
 .../db/AbstractBufferClusteringPrefix.java |   13 +-
 .../org/apache/cassandra/db/ArrayClustering.java   |5 +
 .../org/apache/cassandra/db/BufferClustering.java  |4 +
 .../apache/cassandra/db/BufferDecoratedKey.java|6 +
 .../org/apache/cassandra/db/ClusteringPrefix.java  |8 +-
 .../org/apache/cassandra/db/ColumnFamilyStore.java |   52 +-
 src/java/org/apache/cassandra/db/Columns.java  |2 +-
 src/java/org/apache/cassandra/db/DecoratedKey.java |1 +
 .../org/apache/cassandra/db/NativeClustering.java  |   22 +-
 .../apache/cassandra/db/NativeDecoratedKey.java|6 +
 .../db/memtable/AbstractAllocatorMemtable.java |   16 +-
 .../cassandra/db/memtable/AbstractMemtable.java|2 +
 .../db/memtable/AbstractShardedMemtable.java   |  103 ++
 .../apache/cassandra/db/memtable/Memtable_API.md   |   21 +-
 .../cassandra/db/memtable/ShardBoundaries.java |3 +-
 .../db/memtable/ShardedMemtableConfigMXBean.java   |   30 +-
 .../db/memtable/ShardedSkipListMemtable.java   |   58 +-
 .../cassandra/db/memtable/SkipListMemtable.java|   31 +-
 .../apache/cassandra/db/memtable/TrieMemtable.java |  722 ++
 .../db/partitions/AbstractBTreePartition.java  |   78 +-
 .../db/partitions/AtomicBTreePartition.java|  261 ++---
 .../db/partitions/BTreePartitionData.java  |  101 ++
 .../db/partitions/BTreePartitionUpdater.java   |  184 
 .../db/partitions/CachedBTreePartition.java|6 +-
 .../db/partitions/ImmutableBTreePartition.java |8 +-
 .../cassandra/db/partitions/PartitionUpdate.java   |   32 +-
 .../cassandra/db/tries/CollectionMergeTrie.java|  363 +++
 .../cassandra/db/tries/InMemoryReadTrie.java   |  920 ++
 .../apache/cassandra/db/tries/InMemoryTrie.java| 1028 
 .../org/apache/cassandra/db/tries/InMemoryTrie.md  |  753 ++
 .../cassandra/db/tries/InMemoryTrie.md.a1.svg  |  599 
 .../cassandra/db/tries/InMemoryTrie.md.g1.svg  |   76 ++
 .../cassandra/db/tries/InMemoryTrie.md.g2.svg  |  116 +++
 .../cassandra/db/tries/InMemoryTrie.md.g3.svg  |  253 +
 .../cassandra/db/tries/InMemoryTrie.md.g4.svg  |  290 ++
 .../cassandra/db/tries/InMemoryTrie.md.m1.svg  |  349 +++
 .../cassandra/db/tries/InMemoryTrie.md.m2.svg  |  430 
 .../cassandra/db/tries/InMemoryTrie.md.m3.svg  |  500 ++
 .../cassandra/db/tries/InMemoryTrie.md.p1.svg  |  405 
 .../cassandra/db/tries/InMemoryTrie.md.w1.svg  |  226 +
 .../cassandra/db/tries/InMemoryTrie.md.w2.svg  |  326 +++
 .../cassandra/db/tries/InMemoryTrie.md.w3.svg  |  269 +
 .../cassandra/db/tries/InMemoryTrie.md.wc1.svg |  349 +++
 .../cassandra/db/tries/InMemoryTrie.md.wc2.svg |  314 ++
 .../org/apache/cassandra/db/tries/MergeTrie.java   |  166 
 .../apache/cassandra/db/tries/SingletonTrie.java   |  109 +++
 .../org/apache/cassandra/db/tries/SlicedTrie.java  |  242 +
 src/java/org/apache/cassandra/db/tries/Trie.java   |  622 
 src/java/org/apache/cassandra/db/tries/Trie.md |  252 +
 .../org/apache/cassandra/db/tries/Tri

[cassandra] 01/01: Merge branch 'cassandra-4.1' into trunk

2022-06-29 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 4e8c83118963c1e142bd29eb997be73308a942c7
Merge: 8ffaf4af62 60b54425ed
Author: Branimir Lambov 
AuthorDate: Wed Jun 29 12:12:02 2022 +0300

Merge branch 'cassandra-4.1' into trunk

 CHANGES.txt|  1 +
 .../org/apache/cassandra/db/ColumnFamilyStore.java |  2 +-
 .../cassandra/db/memtable/AbstractMemtable.java|  5 +-
 .../db/memtable/AbstractMemtableWithCommitlog.java |  8 ++-
 .../org/apache/cassandra/db/memtable/Flushing.java |  2 +-
 .../org/apache/cassandra/db/memtable/Memtable.java |  2 +-
 .../cassandra/db/commitlog/CommitLogCQLTest.java   | 69 ++
 7 files changed, 80 insertions(+), 9 deletions(-)

diff --cc CHANGES.txt
index ded42eb5a5,a8def999c5..6a7eae004d
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,17 -1,5 +1,18 @@@
 -4.1-alpha2
 +4.2
 + * Add guardrail for maximum replication factor (CASSANDRA-17500)
 + * Increment CQLSH to version 6.2.0 for release 4.2 (CASSANDRA-17646)
 + * Adding support to perform certificate based internode authentication 
(CASSANDRA-17661)
 + * Option to disable CDC writes of repaired data (CASSANDRA-17666)
 + * When a node is bootstrapping it gets the whole gossip state but applies in 
random order causing some cases where StorageService will fail causing an 
instance to not show up in TokenMetadata (CASSANDRA-17676)
 + * Add CQLSH command SHOW REPLICAS (CASSANDRA-17577)
 + * Add guardrail to allow disabling of SimpleStrategy (CASSANDRA-17647)
 + * Change default directory permission to 750 in packaging (CASSANDRA-17470)
 + * Adding support for TLS client authentication for internode communication 
(CASSANDRA-17513)
 + * Add new CQL function maxWritetime (CASSANDRA-17425)
 + * Add guardrail for ALTER TABLE ADD / DROP / REMOVE column operations 
(CASSANDRA-17495)
 + * Rename DisableFlag class to EnableFlag on guardrails (CASSANDRA-17544)
 +Merged from 4.1:
+  * Fix commitLogUpperBound initialization in AbstractMemtableWithCommitlog 
(CASSANDRA-17587)
   * Fix widening to long in getBatchSizeFailThreshold (CASSANDRA-17650)
   * Fix widening from mebibytes to bytes in IntMebibytesBound (CASSANDRA-17716)
   * Revert breaking change in nodetool clientstats and expose cient options 
through nodetool clientstats --client-options. (CASSANDRA-17715)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch cassandra-4.1 updated: Fix commitLogUpperBound initialization in AbstractMemtableWithCommitlog so that it is always available when writeBarrier gets initialized.

2022-06-29 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch cassandra-4.1
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/cassandra-4.1 by this push:
 new 60b54425ed Fix commitLogUpperBound initialization in 
AbstractMemtableWithCommitlog so that it is always available when writeBarrier 
gets initialized.
60b54425ed is described below

commit 60b54425edc0a328bc1baf00c2e5bf111d4b9da8
Author: Jakub Żytka 
AuthorDate: Tue May 11 10:01:12 2021 +0200

Fix commitLogUpperBound initialization in AbstractMemtableWithCommitlog
so that it is always available when writeBarrier gets initialized.

Harden Memtable API so that it is apparent that getting commitLogUpperBound
is valid only after it is fully established.

patch by Jakub Żytka and Dan Jatnieks;
reviewed by Dan Jatnieks, Jeremiah D Jordan and Caleb Rackliffe for 
CASSANDRA-17587
---
 CHANGES.txt|  1 +
 .../org/apache/cassandra/db/ColumnFamilyStore.java |  2 +-
 .../cassandra/db/memtable/AbstractMemtable.java|  5 +-
 .../db/memtable/AbstractMemtableWithCommitlog.java |  8 ++-
 .../org/apache/cassandra/db/memtable/Flushing.java |  2 +-
 .../org/apache/cassandra/db/memtable/Memtable.java |  2 +-
 .../cassandra/db/commitlog/CommitLogCQLTest.java   | 69 ++
 7 files changed, 80 insertions(+), 9 deletions(-)

diff --git a/CHANGES.txt b/CHANGES.txt
index 96bebe774d..a8def999c5 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 4.1-alpha2
+ * Fix commitLogUpperBound initialization in AbstractMemtableWithCommitlog 
(CASSANDRA-17587)
  * Fix widening to long in getBatchSizeFailThreshold (CASSANDRA-17650)
  * Fix widening from mebibytes to bytes in IntMebibytesBound (CASSANDRA-17716)
  * Revert breaking change in nodetool clientstats and expose cient options 
through nodetool clientstats --client-options. (CASSANDRA-17715)
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index bf43f327e3..a40e5c7ad1 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -1103,7 +1103,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean, Memtable.Owner
 // If a flush errored out but the error was ignored, make sure we 
don't discard the commit log.
 if (flushFailure == null && mainMemtable != null)
 {
-commitLogUpperBound = mainMemtable.getCommitLogUpperBound();
+commitLogUpperBound = 
mainMemtable.getFinalCommitLogUpperBound();
 CommitLog.instance.discardCompletedSegments(metadata.id, 
mainMemtable.getCommitLogLowerBound(), commitLogUpperBound);
 }
 
diff --git a/src/java/org/apache/cassandra/db/memtable/AbstractMemtable.java 
b/src/java/org/apache/cassandra/db/memtable/AbstractMemtable.java
index 1d683db910..0ac7482e4a 100644
--- a/src/java/org/apache/cassandra/db/memtable/AbstractMemtable.java
+++ b/src/java/org/apache/cassandra/db/memtable/AbstractMemtable.java
@@ -28,7 +28,6 @@ import java.util.concurrent.atomic.AtomicReference;
 
 import org.apache.cassandra.db.RegularAndStaticColumns;
 import org.apache.cassandra.db.commitlog.CommitLogPosition;
-import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
 import org.apache.cassandra.db.partitions.Partition;
 import org.apache.cassandra.db.rows.EncodingStats;
 import org.apache.cassandra.schema.ColumnMetadata;
@@ -204,9 +203,9 @@ public abstract class AbstractMemtable implements Memtable
 return AbstractMemtable.this.getCommitLogLowerBound();
 }
 
-public CommitLogPosition commitLogUpperBound()
+public LastCommitLogPosition commitLogUpperBound()
 {
-return AbstractMemtable.this.getCommitLogUpperBound();
+return AbstractMemtable.this.getFinalCommitLogUpperBound();
 }
 
 public EncodingStats encodingStats()
diff --git 
a/src/java/org/apache/cassandra/db/memtable/AbstractMemtableWithCommitlog.java 
b/src/java/org/apache/cassandra/db/memtable/AbstractMemtableWithCommitlog.java
index d60fe866ab..4fe39a10ca 100644
--- 
a/src/java/org/apache/cassandra/db/memtable/AbstractMemtableWithCommitlog.java
+++ 
b/src/java/org/apache/cassandra/db/memtable/AbstractMemtableWithCommitlog.java
@@ -57,8 +57,8 @@ public abstract class AbstractMemtableWithCommitlog extends 
AbstractMemtable
 // This can prepare the memtable data for deletion; it will still be 
used while the flush is proceeding.
 // A setDiscarded call will follow.
 assert this.writeBarrier == null;
-this.writeBarrier = writeBarrier;
 this.commitLogUpperBound = commitLogUpperBound;
+this.writeBarrier = writeBarrier;
 }
 
 public void discard()
@@ -113,9

[cassandra] branch trunk updated (8ffaf4af62 -> 4e8c831189)

2022-06-29 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git


from 8ffaf4af62 Merge branch 'cassandra-4.1' into trunk
 new 60b54425ed Fix commitLogUpperBound initialization in 
AbstractMemtableWithCommitlog so that it is always available when writeBarrier 
gets initialized.
 new 4e8c831189 Merge branch 'cassandra-4.1' into trunk

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|  1 +
 .../org/apache/cassandra/db/ColumnFamilyStore.java |  2 +-
 .../cassandra/db/memtable/AbstractMemtable.java|  5 +-
 .../db/memtable/AbstractMemtableWithCommitlog.java |  8 ++-
 .../org/apache/cassandra/db/memtable/Flushing.java |  2 +-
 .../org/apache/cassandra/db/memtable/Memtable.java |  2 +-
 .../cassandra/db/commitlog/CommitLogCQLTest.java   | 69 ++
 7 files changed, 80 insertions(+), 9 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra-dtest] branch trunk updated: Fix test expectations for Memtable API (CEP-11)

2022-04-29 Thread blambov

This is an automated email from the ASF dual-hosted git repository.

blambov pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra-dtest.git


The following commit(s) were added to refs/heads/trunk by this push:
 new bd5e29c7 Fix test expectations for Memtable API (CEP-11)
bd5e29c7 is described below

commit bd5e29c7ca8e0d6987ba9d180d97766cb30eb0fa
Author: Branimir Lambov 
AuthorDate: Wed Apr 20 10:57:47 2022 +0300

Fix test expectations for Memtable API (CEP-11)

patch by Branimir Lambov; reviewed by Andrés de la Peña and Caleb Rackliffe 
for CASSANDRA-17034
---
 cqlsh_tests/test_cqlsh.py | 77 +--
 snapshot_test.py  | 22 +-
 2 files changed, 88 insertions(+), 11 deletions(-)

diff --git a/cqlsh_tests/test_cqlsh.py b/cqlsh_tests/test_cqlsh.py
index 9ee4ea98..69ca0a93 100644
--- a/cqlsh_tests/test_cqlsh.py
+++ b/cqlsh_tests/test_cqlsh.py
@@ -1130,7 +1130,28 @@ CREATE TYPE test.address_type (
 PRIMARY KEY (id, col)
 """
 
-if self.cluster.version() >= LooseVersion('4.0'):
+if self.cluster.version() >= LooseVersion('4.1'):
+create_table += """
+) WITH CLUSTERING ORDER BY (col ASC)
+AND additional_write_policy = '99p'
+AND bloom_filter_fp_chance = 0.01
+AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
+AND cdc = false
+AND comment = ''
+AND compaction = {'class': 
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
'max_threshold': '32', 'min_threshold': '4'}
+AND compression = {'chunk_length_in_kb': '16', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}
+AND memtable = 'default'
+AND crc_check_chance = 1.0
+AND default_time_to_live = 0
+AND extensions = {}
+AND gc_grace_seconds = 864000
+AND max_index_interval = 2048
+AND memtable_flush_period_in_ms = 0
+AND min_index_interval = 128
+AND read_repair = 'BLOCKING'
+AND speculative_retry = '99p';
+"""
+elif self.cluster.version() >= LooseVersion('4.0'):
 create_table += """
 ) WITH CLUSTERING ORDER BY (col ASC)
 AND additional_write_policy = '99p'
@@ -1215,7 +1236,32 @@ CREATE TYPE test.address_type (
 myindex_output = self.get_index_output('myindex', 'test', 'users', 
'age')
 create_table = None
 
-if self.cluster.version() >= LooseVersion('4.0'):
+if self.cluster.version() >= LooseVersion('4.1'):
+create_table = """
+CREATE TABLE test.users (
+userid text PRIMARY KEY,
+age int,
+firstname text,
+lastname text
+) WITH additional_write_policy = '99p'
+AND bloom_filter_fp_chance = 0.01
+AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
+AND cdc = false
+AND comment = ''
+AND compaction = {'class': 
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
'max_threshold': '32', 'min_threshold': '4'}
+AND compression = {'chunk_length_in_kb': '16', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}
+AND memtable = 'default'
+AND crc_check_chance = 1.0
+AND default_time_to_live = 0
+AND extensions = {}
+AND gc_grace_seconds = 864000
+AND max_index_interval = 2048
+AND memtable_flush_period_in_ms = 0
+AND min_index_interval = 128
+AND read_repair = 'BLOCKING'
+AND speculative_retry = '99p';
+"""
+elif self.cluster.version() >= LooseVersion('4.0'):
 create_table = """
 CREATE TABLE test.users (
 userid text PRIMARY KEY,
@@ -1320,7 +1366,32 @@ CREATE TYPE test.address_type (
 return "CREATE INDEX {} ON {}.{} ({});".format(index, ks, table, col)
 
 def get_users_by_state_mv_output(self):
-if self.cluster.version() >= LooseVersion('4.0'):
+if self.cluster.version() >= LooseVersion('4.1'):
+return """
+CREATE MATERIALIZED VIEW test.users_by_state AS
+SELECT *
+FROM test.users
+WHERE state IS NOT NULL AND username IS NOT NULL
+PRIMARY KEY (state, username)
+WITH CLUSTERING ORDER BY (username ASC)
+AND additional_write_policy = '99p'
+AND bloom_filter_fp_chance = 0.01
+AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
+AND cdc = false
+AND comment = ''
+

cassandra git commit: Pad uncompressed chunks when they would be interpreted as compressed

2018-11-29 Thread blambov

Repository: cassandra
Updated Branches:
  refs/heads/trunk c5dee08df -> 9a7db292c


Pad uncompressed chunks when they would be interpreted as compressed

patch by Branimir Lambov; reviewed by Robert Stupp for CASSANDRA-14892


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9a7db292
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9a7db292
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9a7db292

Branch: refs/heads/trunk
Commit: 9a7db292cc4e470cd913f5c850982a7d7300d6c8
Parents: c5dee08
Author: Branimir Lambov 
Authored: Fri Nov 16 12:32:31 2018 +0200
Committer: Branimir Lambov 
Committed: Thu Nov 29 11:29:05 2018 +0200

--
 .../io/compress/CompressedSequentialWriter.java |  21 +++-
 .../apache/cassandra/utils/ByteBufferUtil.java  |  25 +
 .../CompressedSequentialWriterTest.java |  94 +++--
 .../cassandra/io/compress/MockCompressor.java   | 103 +++
 .../cassandra/utils/ByteBufferUtilTest.java |  64 +++-
 5 files changed, 291 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9a7db292/src/java/org/apache/cassandra/io/compress/CompressedSequentialWriter.java
--
diff --git 
a/src/java/org/apache/cassandra/io/compress/CompressedSequentialWriter.java 
b/src/java/org/apache/cassandra/io/compress/CompressedSequentialWriter.java
index c35ecc8..5be96d1 100644
--- a/src/java/org/apache/cassandra/io/compress/CompressedSequentialWriter.java
+++ b/src/java/org/apache/cassandra/io/compress/CompressedSequentialWriter.java
@@ -32,6 +32,7 @@ import 
org.apache.cassandra.io.sstable.CorruptSSTableException;
 import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
 import org.apache.cassandra.io.util.*;
 import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.utils.ByteBufferUtil;
 
 import static org.apache.cassandra.utils.Throwables.merge;
 
@@ -148,13 +149,27 @@ public class CompressedSequentialWriter extends 
SequentialWriter
 throw new RuntimeException("Compression exception", e); // 
shouldn't happen
 }
 
+int uncompressedLength = buffer.position();
 int compressedLength = compressed.position();
-uncompressedSize += buffer.position();
+uncompressedSize += uncompressedLength;
 ByteBuffer toWrite = compressed;
 if (compressedLength >= maxCompressedLength)
 {
 toWrite = buffer;
-compressedLength = buffer.position();
+if (uncompressedLength >= maxCompressedLength)
+{
+compressedLength = uncompressedLength;
+}
+else
+{
+// Pad the uncompressed data so that it reaches the max 
compressed length.
+// This could make the chunk appear longer, but this path is 
only reached at the end of the file, where
+// we use the file size to limit the buffer on reading.
+assert maxCompressedLength <= buffer.capacity();   // verified 
by CompressionParams.validate
+buffer.limit(maxCompressedLength);
+ByteBufferUtil.writeZeroes(buffer, maxCompressedLength - 
uncompressedLength);
+compressedLength = maxCompressedLength;
+}
 }
 compressedSize += compressedLength;
 
@@ -178,7 +193,7 @@ public class CompressedSequentialWriter extends 
SequentialWriter
 throw new FSWriteError(e, getPath());
 }
 if (toWrite == buffer)
-buffer.position(compressedLength);
+buffer.position(uncompressedLength);
 
 // next chunk should be written right after current + length of the 
checksum (int)
 chunkOffset += compressedLength + 4;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9a7db292/src/java/org/apache/cassandra/utils/ByteBufferUtil.java
--
diff --git a/src/java/org/apache/cassandra/utils/ByteBufferUtil.java 
b/src/java/org/apache/cassandra/utils/ByteBufferUtil.java
index 50d35bc..d6c9e52 100644
--- a/src/java/org/apache/cassandra/utils/ByteBufferUtil.java
+++ b/src/java/org/apache/cassandra/utils/ByteBufferUtil.java
@@ -284,6 +284,31 @@ public class ByteBufferUtil
 return length;
 }
 
+public static void writeZeroes(ByteBuffer dest, int count)
+{
+if (count >= 8)
+{
+// align
+while ((dest.position() & 0x7) != 0)
+{
+dest.put((byte) 0);
+--count;
+}
+}
+// write aligned longs
+while (count >= 8)
+{
+dest.putLong(0L);
+

[3/3] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2018-11-13 Thread blambov

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b80f6c65
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b80f6c65
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b80f6c65

Branch: refs/heads/trunk
Commit: b80f6c65fb0b97a8c79f6da027deac06a4af9801
Parents: caf50de a03424e
Author: Branimir Lambov 
Authored: Tue Nov 13 12:53:14 2018 +0200
Committer: Branimir Lambov 
Committed: Tue Nov 13 13:00:39 2018 +0200

--
 CHANGES.txt |  1 +
 .../db/SinglePartitionReadCommand.java  |  4 +-
 .../db/compaction/CompactionManager.java|  2 +-
 .../db/compaction/LeveledManifest.java  |  5 +-
 .../io/sstable/format/SSTableReader.java|  4 +-
 .../tools/nodetool/GarbageCollect.java  |  8 ++-
 .../apache/cassandra/cql3/GcCompactionTest.java | 71 
 .../apache/cassandra/db/ReadCommandTest.java|  2 +-
 .../LeveledCompactionStrategyTest.java  | 33 +
 9 files changed, 119 insertions(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b80f6c65/CHANGES.txt
--
diff --cc CHANGES.txt
index 5fd28bc,83e8b08..c77e7ed
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,332 -1,5 +1,333 @@@
 +4.0
 + * Lower default chunk_length_in_kb from 64kb to 16kb (CASSANDRA-13241)
 + * Startup checker should wait for count rather than percentage 
(CASSANDRA-14297)
 + * Fix incorrect sorting of replicas in 
SimpleStrategy.calculateNaturalReplicas (CASSANDRA-14862)
 + * Partitioned outbound internode TCP connections can occur when nodes 
restart (CASSANDRA-14358)
 + * Don't write to system_distributed.repair_history, system_traces.sessions, 
system_traces.events in mixed version 3.X/4.0 clusters (CASSANDRA-14841)
 + * Avoid running query to self through messaging service (CASSANDRA-14807)
 + * Allow using custom script for chronicle queue BinLog archival 
(CASSANDRA-14373)
 + * Transient->Full range movements mishandle consistency level upgrade 
(CASSANDRA-14759)
 + * ReplicaCollection follow-up (CASSANDRA-14726)
 + * Transient node receives full data requests (CASSANDRA-14762)
 + * Enable snapshot artifacts publish (CASSANDRA-12704)
 + * Introduce RangesAtEndpoint.unwrap to simplify 
StreamSession.addTransferRanges (CASSANDRA-14770)
 + * LOCAL_QUORUM may speculate to non-local nodes, resulting in Timeout 
instead of Unavailable (CASSANDRA-14735)
 + * Avoid creating empty compaction tasks after truncate (CASSANDRA-14780)
 + * Fail incremental repair prepare phase if it encounters sstables from 
un-finalized sessions (CASSANDRA-14763)
 + * Add a check for receiving digest response from transient node 
(CASSANDRA-14750)
 + * Fail query on transient replica if coordinator only expects full data 
(CASSANDRA-14704)
 + * Remove mentions of transient replication from repair path (CASSANDRA-14698)
 + * Fix handleRepairStatusChangedNotification to remove first then add 
(CASSANDRA-14720)
 + * Allow transient node to serve as a repair coordinator (CASSANDRA-14693)
 + * DecayingEstimatedHistogramReservoir.EstimatedHistogramReservoirSnapshot 
returns wrong value for size() and incorrectly calculates count 
(CASSANDRA-14696)
 + * AbstractReplicaCollection equals and hash code should throw due to 
conflict between order sensitive/insensitive uses (CASSANDRA-14700)
 + * Detect inconsistencies in repaired data on the read path (CASSANDRA-14145)
 + * Add checksumming to the native protocol (CASSANDRA-13304)
 + * Make AuthCache more easily extendable (CASSANDRA-14662)
 + * Extend RolesCache to include detailed role info (CASSANDRA-14497)
 + * Add fqltool compare (CASSANDRA-14619)
 + * Add fqltool replay (CASSANDRA-14618)
 + * Log keyspace in full query log (CASSANDRA-14656)
 + * Transient Replication and Cheap Quorums (CASSANDRA-14404)
 + * Log server-generated timestamp and nowInSeconds used by queries in FQL 
(CASSANDRA-14675)
 + * Add diagnostic events for read repairs (CASSANDRA-14668)
 + * Use consistent nowInSeconds and timestamps values within a request 
(CASSANDRA-14671)
 + * Add sampler for query time and expose with nodetool (CASSANDRA-14436)
 + * Clean up Message.Request implementations (CASSANDRA-14677)
 + * Disable old native protocol versions on demand (CASANDRA-14659)
 + * Allow specifying now-in-seconds in native protocol (CASSANDRA-14664)
 + * Improve BTree build performance by avoiding data copy (CASSANDRA-9989)
 + * Make monotonic read / read repair configurable (CASSANDRA-14635)
 + * Refactor CompactionStrategyManager (CASSANDRA-14621)
 + * Flush netty client messages immediately by default (CASSANDRA-13651)
 + * Improve read repair blocking behavior (CASSANDRA-10726)
 + * Add a

[1/3] cassandra git commit: Correct sstable sorting for garbagecollect and levelled compaction

2018-11-13 Thread blambov

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.11 d17836dec -> a03424ef9
  refs/heads/trunk caf50de31 -> b80f6c65f


Correct sstable sorting for garbagecollect and levelled compaction

patch by Branimir Lambov and Vincent White; reviewed by Zhao Yang for 
CASSANDRA-14879


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a03424ef
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a03424ef
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a03424ef

Branch: refs/heads/cassandra-3.11
Commit: a03424ef95559c9df2bb7f86e1ac1edca1436058
Parents: d17836d
Author: Branimir Lambov 
Authored: Wed Nov 7 13:10:39 2018 +0200
Committer: Branimir Lambov 
Committed: Tue Nov 13 12:50:08 2018 +0200

--
 CHANGES.txt |  1 +
 .../db/SinglePartitionReadCommand.java  |  4 +-
 .../db/compaction/CompactionManager.java|  2 +-
 .../db/compaction/LeveledManifest.java  |  5 +-
 .../io/sstable/format/SSTableReader.java|  4 +-
 .../tools/nodetool/GarbageCollect.java  |  8 ++-
 .../apache/cassandra/cql3/GcCompactionTest.java | 73 +++-
 .../LeveledCompactionStrategyTest.java  | 33 +
 8 files changed, 119 insertions(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a03424ef/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e07099a..83e8b08 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.11.4
+ * Correct sstable sorting for garbagecollect and levelled compaction 
(CASSANDRA-14870)
 Merged from 3.0:
  * Move TWCS message 'No compaction necessary for bucket size' to Trace level 
(CASSANDRA-14884)
  * Sstable min/max metadata can cause data loss (CASSANDRA-14861)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a03424ef/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
--
diff --git a/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java 
b/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
index ed98e28..bee4961 100644
--- a/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
+++ b/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
@@ -728,7 +728,7 @@ public class SinglePartitionReadCommand extends ReadCommand
  * In other words, iterating in maxTimestamp order allow to do our 
mostRecentPartitionTombstone elimination
  * in one pass, and minimize the number of sstables for which we 
read a partition tombstone.
  */
-Collections.sort(view.sstables, 
SSTableReader.maxTimestampComparator);
+Collections.sort(view.sstables, 
SSTableReader.maxTimestampDescending);
 long mostRecentPartitionTombstone = Long.MIN_VALUE;
 int nonIntersectingSSTables = 0;
 List skippedSSTablesWithTombstones = null;
@@ -916,7 +916,7 @@ public class SinglePartitionReadCommand extends ReadCommand
 }
 
 /* add the SSTables on disk */
-Collections.sort(view.sstables, SSTableReader.maxTimestampComparator);
+Collections.sort(view.sstables, SSTableReader.maxTimestampDescending);
 boolean onlyUnrepaired = true;
 // read sorted sstables
 SSTableReadMetricsCollector metricsCollector = new 
SSTableReadMetricsCollector();

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a03424ef/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index 235fe2b..61da975 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -498,7 +498,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 if 
(cfStore.getCompactionStrategyManager().onlyPurgeRepairedTombstones())
 originals = Iterables.filter(originals, 
SSTableReader::isRepaired);
 List sortedSSTables = 
Lists.newArrayList(originals);
-Collections.sort(sortedSSTables, 
SSTableReader.maxTimestampComparator);
+Collections.sort(sortedSSTables, 
SSTableReader.maxTimestampAscending);
 return sortedSSTables;
 }
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a03424ef/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java
--
diff --git

[2/3] cassandra git commit: Correct sstable sorting for garbagecollect and levelled compaction

2018-11-13 Thread blambov

Correct sstable sorting for garbagecollect and levelled compaction

patch by Branimir Lambov and Vincent White; reviewed by Zhao Yang for 
CASSANDRA-14879


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a03424ef
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a03424ef
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a03424ef

Branch: refs/heads/trunk
Commit: a03424ef95559c9df2bb7f86e1ac1edca1436058
Parents: d17836d
Author: Branimir Lambov 
Authored: Wed Nov 7 13:10:39 2018 +0200
Committer: Branimir Lambov 
Committed: Tue Nov 13 12:50:08 2018 +0200

--
 CHANGES.txt |  1 +
 .../db/SinglePartitionReadCommand.java  |  4 +-
 .../db/compaction/CompactionManager.java|  2 +-
 .../db/compaction/LeveledManifest.java  |  5 +-
 .../io/sstable/format/SSTableReader.java|  4 +-
 .../tools/nodetool/GarbageCollect.java  |  8 ++-
 .../apache/cassandra/cql3/GcCompactionTest.java | 73 +++-
 .../LeveledCompactionStrategyTest.java  | 33 +
 8 files changed, 119 insertions(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a03424ef/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e07099a..83e8b08 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.11.4
+ * Correct sstable sorting for garbagecollect and levelled compaction 
(CASSANDRA-14870)
 Merged from 3.0:
  * Move TWCS message 'No compaction necessary for bucket size' to Trace level 
(CASSANDRA-14884)
  * Sstable min/max metadata can cause data loss (CASSANDRA-14861)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a03424ef/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
--
diff --git a/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java 
b/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
index ed98e28..bee4961 100644
--- a/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
+++ b/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
@@ -728,7 +728,7 @@ public class SinglePartitionReadCommand extends ReadCommand
  * In other words, iterating in maxTimestamp order allow to do our 
mostRecentPartitionTombstone elimination
  * in one pass, and minimize the number of sstables for which we 
read a partition tombstone.
  */
-Collections.sort(view.sstables, 
SSTableReader.maxTimestampComparator);
+Collections.sort(view.sstables, 
SSTableReader.maxTimestampDescending);
 long mostRecentPartitionTombstone = Long.MIN_VALUE;
 int nonIntersectingSSTables = 0;
 List skippedSSTablesWithTombstones = null;
@@ -916,7 +916,7 @@ public class SinglePartitionReadCommand extends ReadCommand
 }
 
 /* add the SSTables on disk */
-Collections.sort(view.sstables, SSTableReader.maxTimestampComparator);
+Collections.sort(view.sstables, SSTableReader.maxTimestampDescending);
 boolean onlyUnrepaired = true;
 // read sorted sstables
 SSTableReadMetricsCollector metricsCollector = new 
SSTableReadMetricsCollector();

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a03424ef/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index 235fe2b..61da975 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -498,7 +498,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 if 
(cfStore.getCompactionStrategyManager().onlyPurgeRepairedTombstones())
 originals = Iterables.filter(originals, 
SSTableReader::isRepaired);
 List sortedSSTables = 
Lists.newArrayList(originals);
-Collections.sort(sortedSSTables, 
SSTableReader.maxTimestampComparator);
+Collections.sort(sortedSSTables, 
SSTableReader.maxTimestampAscending);
 return sortedSSTables;
 }
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a03424ef/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java 
b/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java
index ceb3811..520b08d

[07/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

2018-08-21 Thread blambov

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/299782cf
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/299782cf
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/299782cf

Branch: refs/heads/trunk
Commit: 299782cff5a56db8b8fe9ad70a43bea7b729cc99
Parents: 236c47e 49adbe7
Author: Branimir Lambov 
Authored: Tue Aug 21 11:55:38 2018 +0300
Committer: Branimir Lambov 
Committed: Tue Aug 21 11:55:38 2018 +0300

--
 .../io/sstable/IndexSummaryBuilder.java |  4 +-
 .../cassandra/io/util/DataOutputBuffer.java |  8 +-
 .../io/util/DataOutputBufferFixed.java  |  2 +-
 .../cassandra/io/util/SafeMemoryWriter.java | 16 ++--
 .../cassandra/io/util/DataOutputTest.java   |  4 +-
 .../cassandra/io/util/SafeMemoryWriterTest.java | 90 
 6 files changed, 110 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/299782cf/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/299782cf/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[09/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2018-08-21 Thread blambov

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/991e1971
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/991e1971
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/991e1971

Branch: refs/heads/trunk
Commit: 991e19711f8762bbf93d6af588cef0a14668cc59
Parents: 65a4682 299782c
Author: Branimir Lambov 
Authored: Tue Aug 21 11:56:05 2018 +0300
Committer: Branimir Lambov 
Committed: Tue Aug 21 11:56:05 2018 +0300

--
 .../io/sstable/IndexSummaryBuilder.java |  4 +-
 .../cassandra/io/util/DataOutputBuffer.java |  8 +-
 .../io/util/DataOutputBufferFixed.java  |  2 +-
 .../cassandra/io/util/SafeMemoryWriter.java | 16 ++--
 .../cassandra/io/util/DataOutputTest.java   |  4 +-
 .../cassandra/io/util/SafeMemoryWriterTest.java | 90 
 6 files changed, 110 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/991e1971/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
--
diff --cc src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
index 144edad,7586543..28ca468
--- a/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
+++ b/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
@@@ -38,43 -37,11 +38,43 @@@ public class DataOutputBuffer extends B
  /*
   * Threshold at which resizing transitions from doubling to increasing by 
50%
   */
- private static final long DOUBLING_THRESHOLD = 
Long.getLong(Config.PROPERTY_PREFIX + "DOB_DOUBLING_THRESHOLD_MB", 64);
+ static final long DOUBLING_THRESHOLD = 
Long.getLong(Config.PROPERTY_PREFIX + "DOB_DOUBLING_THRESHOLD_MB", 64);
  
 +/*
 + * Only recycle OutputBuffers up to 1Mb. Larger buffers will be trimmed 
back to this size.
 + */
 +private static final int MAX_RECYCLE_BUFFER_SIZE = 
Integer.getInteger(Config.PROPERTY_PREFIX + "dob_max_recycle_bytes", 1024 * 
1024);
 +
 +private static final int DEFAULT_INITIAL_BUFFER_SIZE = 128;
 +
 +/**
 + * Scratch buffers used mostly for serializing in memory. It's important 
to call #recycle() when finished
 + * to keep the memory overhead from being too large in the system.
 + */
 +public static final FastThreadLocal scratchBuffer = new 
FastThreadLocal()
 +{
 +protected DataOutputBuffer initialValue() throws Exception
 +{
 +return new DataOutputBuffer()
 +{
 +public void close()
 +{
 +if (buffer.capacity() <= MAX_RECYCLE_BUFFER_SIZE)
 +{
 +buffer.clear();
 +}
 +else
 +{
 +buffer = 
ByteBuffer.allocate(DEFAULT_INITIAL_BUFFER_SIZE);
 +}
 +}
 +};
 +}
 +};
 +
  public DataOutputBuffer()
  {
 -this(128);
 +this(DEFAULT_INITIAL_BUFFER_SIZE);
  }
  
  public DataOutputBuffer(int size)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/991e1971/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/991e1971/test/unit/org/apache/cassandra/io/util/DataOutputTest.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[05/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

2018-08-21 Thread blambov

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/299782cf
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/299782cf
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/299782cf

Branch: refs/heads/cassandra-3.11
Commit: 299782cff5a56db8b8fe9ad70a43bea7b729cc99
Parents: 236c47e 49adbe7
Author: Branimir Lambov 
Authored: Tue Aug 21 11:55:38 2018 +0300
Committer: Branimir Lambov 
Committed: Tue Aug 21 11:55:38 2018 +0300

--
 .../io/sstable/IndexSummaryBuilder.java |  4 +-
 .../cassandra/io/util/DataOutputBuffer.java |  8 +-
 .../io/util/DataOutputBufferFixed.java  |  2 +-
 .../cassandra/io/util/SafeMemoryWriter.java | 16 ++--
 .../cassandra/io/util/DataOutputTest.java   |  4 +-
 .../cassandra/io/util/SafeMemoryWriterTest.java | 90 
 6 files changed, 110 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/299782cf/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/299782cf/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[06/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

2018-08-21 Thread blambov

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/299782cf
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/299782cf
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/299782cf

Branch: refs/heads/cassandra-3.0
Commit: 299782cff5a56db8b8fe9ad70a43bea7b729cc99
Parents: 236c47e 49adbe7
Author: Branimir Lambov 
Authored: Tue Aug 21 11:55:38 2018 +0300
Committer: Branimir Lambov 
Committed: Tue Aug 21 11:55:38 2018 +0300

--
 .../io/sstable/IndexSummaryBuilder.java |  4 +-
 .../cassandra/io/util/DataOutputBuffer.java |  8 +-
 .../io/util/DataOutputBufferFixed.java  |  2 +-
 .../cassandra/io/util/SafeMemoryWriter.java | 16 ++--
 .../cassandra/io/util/DataOutputTest.java   |  4 +-
 .../cassandra/io/util/SafeMemoryWriterTest.java | 90 
 6 files changed, 110 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/299782cf/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/299782cf/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[02/10] cassandra git commit: Fix SafeMemoryWriter trimming and behaviour over 2G

2018-08-21 Thread blambov

Fix SafeMemoryWriter trimming and behaviour over 2G

patch by Branimir Lambov; reviewed by Benedict Elliott Smith for CASSANDRA-14649


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/49adbe7e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/49adbe7e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/49adbe7e

Branch: refs/heads/cassandra-3.0
Commit: 49adbe7e0f0c8a83f3b843b65612528498b5c9a5
Parents: 0e81892
Author: Branimir Lambov 
Authored: Thu Aug 16 16:15:07 2018 +0300
Committer: Branimir Lambov 
Committed: Tue Aug 21 11:53:30 2018 +0300

--
 .../io/sstable/IndexSummaryBuilder.java |  4 +-
 .../cassandra/io/util/DataOutputBuffer.java |  8 +-
 .../io/util/DataOutputBufferFixed.java  |  2 +-
 .../cassandra/io/util/SafeMemoryWriter.java | 16 ++--
 .../cassandra/io/util/DataOutputTest.java   |  4 +-
 .../cassandra/io/util/SafeMemoryWriterTest.java | 90 
 6 files changed, 110 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/49adbe7e/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java 
b/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
index 6110afe..0f604e0 100644
--- a/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
+++ b/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
@@ -207,8 +207,8 @@ public class IndexSummaryBuilder implements AutoCloseable
 {
 // this method should only be called when we've finished appending 
records, so we truncate the
 // memory we're using to the exact amount required to represent it 
before building our summary
-entries.setCapacity(entries.length());
-offsets.setCapacity(offsets.length());
+entries.trim();
+offsets.trim();
 }
 
 public IndexSummary build(IPartitioner partitioner)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/49adbe7e/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
--
diff --git a/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java 
b/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
index 6ea6d97..3f1e081 100644
--- a/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
+++ b/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
@@ -37,7 +37,7 @@ public class DataOutputBuffer extends 
BufferedDataOutputStreamPlus
 /*
  * Threshold at which resizing transitions from doubling to increasing by 
50%
  */
-private static final long DOUBLING_THRESHOLD = 
Long.getLong(Config.PROPERTY_PREFIX + "DOB_DOUBLING_THRESHOLD_MB", 64);
+static final long DOUBLING_THRESHOLD = Long.getLong(Config.PROPERTY_PREFIX 
+ "DOB_DOUBLING_THRESHOLD_MB", 64);
 
 public DataOutputBuffer()
 {
@@ -83,7 +83,7 @@ public class DataOutputBuffer extends 
BufferedDataOutputStreamPlus
 @Override
 protected void doFlush(int count) throws IOException
 {
-reallocate(count);
+expandToFit(count);
 }
 
 //Hack for test, make it possible to override checking the buffer capacity
@@ -119,7 +119,7 @@ public class DataOutputBuffer extends 
BufferedDataOutputStreamPlus
 return validateReallocation(newSize);
 }
 
-protected void reallocate(long count)
+protected void expandToFit(long count)
 {
 if (count <= 0)
 return;
@@ -141,7 +141,7 @@ public class DataOutputBuffer extends 
BufferedDataOutputStreamPlus
 public int write(ByteBuffer src) throws IOException
 {
 int count = src.remaining();
-reallocate(count);
+expandToFit(count);
 buffer.put(src);
 return count;
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/49adbe7e/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java
--
diff --git a/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java 
b/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java
index c815c9e..c9767fc 100644
--- a/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java
+++ b/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java
@@ -58,7 +58,7 @@ public class DataOutputBufferFixed extends DataOutputBuffer
  * @see org.apache.cassandra.io.util.DataOutputBuffer#reallocate(long)
  */
 @Override
-protected void reallocate(long newSize)
+protected void expandToFit(long newSize)
 {
 throw new BufferOverflowException();
 }

[10/10] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2018-08-21 Thread blambov

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8b9515bd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8b9515bd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8b9515bd

Branch: refs/heads/trunk
Commit: 8b9515bd2e410c634e4a31fe3e93890f1a1f8f71
Parents: ac1bb75 991e197
Author: Branimir Lambov 
Authored: Tue Aug 21 11:56:30 2018 +0300
Committer: Branimir Lambov 
Committed: Tue Aug 21 11:56:30 2018 +0300

--
 .../io/sstable/IndexSummaryBuilder.java |  4 +-
 .../cassandra/io/util/DataOutputBuffer.java |  8 +-
 .../io/util/DataOutputBufferFixed.java  |  2 +-
 .../cassandra/io/util/SafeMemoryWriter.java | 16 ++--
 .../cassandra/io/util/DataOutputTest.java   |  4 +-
 .../cassandra/io/util/SafeMemoryWriterTest.java | 90 
 6 files changed, 110 insertions(+), 14 deletions(-)
--



-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[01/10] cassandra git commit: Fix SafeMemoryWriter trimming and behaviour over 2G

2018-08-21 Thread blambov

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 0e81892d7 -> 49adbe7e0
  refs/heads/cassandra-3.0 236c47e65 -> 299782cff
  refs/heads/cassandra-3.11 65a46820b -> 991e19711
  refs/heads/trunk ac1bb7586 -> 8b9515bd2


Fix SafeMemoryWriter trimming and behaviour over 2G

patch by Branimir Lambov; reviewed by Benedict Elliott Smith for CASSANDRA-14649


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/49adbe7e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/49adbe7e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/49adbe7e

Branch: refs/heads/cassandra-2.2
Commit: 49adbe7e0f0c8a83f3b843b65612528498b5c9a5
Parents: 0e81892
Author: Branimir Lambov 
Authored: Thu Aug 16 16:15:07 2018 +0300
Committer: Branimir Lambov 
Committed: Tue Aug 21 11:53:30 2018 +0300

--
 .../io/sstable/IndexSummaryBuilder.java |  4 +-
 .../cassandra/io/util/DataOutputBuffer.java |  8 +-
 .../io/util/DataOutputBufferFixed.java  |  2 +-
 .../cassandra/io/util/SafeMemoryWriter.java | 16 ++--
 .../cassandra/io/util/DataOutputTest.java   |  4 +-
 .../cassandra/io/util/SafeMemoryWriterTest.java | 90 
 6 files changed, 110 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/49adbe7e/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java 
b/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
index 6110afe..0f604e0 100644
--- a/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
+++ b/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
@@ -207,8 +207,8 @@ public class IndexSummaryBuilder implements AutoCloseable
 {
 // this method should only be called when we've finished appending 
records, so we truncate the
 // memory we're using to the exact amount required to represent it 
before building our summary
-entries.setCapacity(entries.length());
-offsets.setCapacity(offsets.length());
+entries.trim();
+offsets.trim();
 }
 
 public IndexSummary build(IPartitioner partitioner)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/49adbe7e/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
--
diff --git a/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java 
b/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
index 6ea6d97..3f1e081 100644
--- a/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
+++ b/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
@@ -37,7 +37,7 @@ public class DataOutputBuffer extends 
BufferedDataOutputStreamPlus
 /*
  * Threshold at which resizing transitions from doubling to increasing by 
50%
  */
-private static final long DOUBLING_THRESHOLD = 
Long.getLong(Config.PROPERTY_PREFIX + "DOB_DOUBLING_THRESHOLD_MB", 64);
+static final long DOUBLING_THRESHOLD = Long.getLong(Config.PROPERTY_PREFIX 
+ "DOB_DOUBLING_THRESHOLD_MB", 64);
 
 public DataOutputBuffer()
 {
@@ -83,7 +83,7 @@ public class DataOutputBuffer extends 
BufferedDataOutputStreamPlus
 @Override
 protected void doFlush(int count) throws IOException
 {
-reallocate(count);
+expandToFit(count);
 }
 
 //Hack for test, make it possible to override checking the buffer capacity
@@ -119,7 +119,7 @@ public class DataOutputBuffer extends 
BufferedDataOutputStreamPlus
 return validateReallocation(newSize);
 }
 
-protected void reallocate(long count)
+protected void expandToFit(long count)
 {
 if (count <= 0)
 return;
@@ -141,7 +141,7 @@ public class DataOutputBuffer extends 
BufferedDataOutputStreamPlus
 public int write(ByteBuffer src) throws IOException
 {
 int count = src.remaining();
-reallocate(count);
+expandToFit(count);
 buffer.put(src);
 return count;
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/49adbe7e/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java
--
diff --git a/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java 
b/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java
index c815c9e..c9767fc 100644
--- a/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java
+++ b/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java
@@ -58,7 +58,7 @@ public class DataOutputBufferFixed extends DataOutputBuffer
  * @see

[04/10] cassandra git commit: Fix SafeMemoryWriter trimming and behaviour over 2G

2018-08-21 Thread blambov

Fix SafeMemoryWriter trimming and behaviour over 2G

patch by Branimir Lambov; reviewed by Benedict Elliott Smith for CASSANDRA-14649


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/49adbe7e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/49adbe7e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/49adbe7e

Branch: refs/heads/trunk
Commit: 49adbe7e0f0c8a83f3b843b65612528498b5c9a5
Parents: 0e81892
Author: Branimir Lambov 
Authored: Thu Aug 16 16:15:07 2018 +0300
Committer: Branimir Lambov 
Committed: Tue Aug 21 11:53:30 2018 +0300

--
 .../io/sstable/IndexSummaryBuilder.java |  4 +-
 .../cassandra/io/util/DataOutputBuffer.java |  8 +-
 .../io/util/DataOutputBufferFixed.java  |  2 +-
 .../cassandra/io/util/SafeMemoryWriter.java | 16 ++--
 .../cassandra/io/util/DataOutputTest.java   |  4 +-
 .../cassandra/io/util/SafeMemoryWriterTest.java | 90 
 6 files changed, 110 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/49adbe7e/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java 
b/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
index 6110afe..0f604e0 100644
--- a/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
+++ b/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
@@ -207,8 +207,8 @@ public class IndexSummaryBuilder implements AutoCloseable
 {
 // this method should only be called when we've finished appending 
records, so we truncate the
 // memory we're using to the exact amount required to represent it 
before building our summary
-entries.setCapacity(entries.length());
-offsets.setCapacity(offsets.length());
+entries.trim();
+offsets.trim();
 }
 
 public IndexSummary build(IPartitioner partitioner)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/49adbe7e/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
--
diff --git a/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java 
b/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
index 6ea6d97..3f1e081 100644
--- a/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
+++ b/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
@@ -37,7 +37,7 @@ public class DataOutputBuffer extends 
BufferedDataOutputStreamPlus
 /*
  * Threshold at which resizing transitions from doubling to increasing by 
50%
  */
-private static final long DOUBLING_THRESHOLD = 
Long.getLong(Config.PROPERTY_PREFIX + "DOB_DOUBLING_THRESHOLD_MB", 64);
+static final long DOUBLING_THRESHOLD = Long.getLong(Config.PROPERTY_PREFIX 
+ "DOB_DOUBLING_THRESHOLD_MB", 64);
 
 public DataOutputBuffer()
 {
@@ -83,7 +83,7 @@ public class DataOutputBuffer extends 
BufferedDataOutputStreamPlus
 @Override
 protected void doFlush(int count) throws IOException
 {
-reallocate(count);
+expandToFit(count);
 }
 
 //Hack for test, make it possible to override checking the buffer capacity
@@ -119,7 +119,7 @@ public class DataOutputBuffer extends 
BufferedDataOutputStreamPlus
 return validateReallocation(newSize);
 }
 
-protected void reallocate(long count)
+protected void expandToFit(long count)
 {
 if (count <= 0)
 return;
@@ -141,7 +141,7 @@ public class DataOutputBuffer extends 
BufferedDataOutputStreamPlus
 public int write(ByteBuffer src) throws IOException
 {
 int count = src.remaining();
-reallocate(count);
+expandToFit(count);
 buffer.put(src);
 return count;
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/49adbe7e/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java
--
diff --git a/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java 
b/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java
index c815c9e..c9767fc 100644
--- a/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java
+++ b/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java
@@ -58,7 +58,7 @@ public class DataOutputBufferFixed extends DataOutputBuffer
  * @see org.apache.cassandra.io.util.DataOutputBuffer#reallocate(long)
  */
 @Override
-protected void reallocate(long newSize)
+protected void expandToFit(long newSize)
 {
 throw new BufferOverflowException();
 }

[08/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2018-08-21 Thread blambov

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/991e1971
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/991e1971
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/991e1971

Branch: refs/heads/cassandra-3.11
Commit: 991e19711f8762bbf93d6af588cef0a14668cc59
Parents: 65a4682 299782c
Author: Branimir Lambov 
Authored: Tue Aug 21 11:56:05 2018 +0300
Committer: Branimir Lambov 
Committed: Tue Aug 21 11:56:05 2018 +0300

--
 .../io/sstable/IndexSummaryBuilder.java |  4 +-
 .../cassandra/io/util/DataOutputBuffer.java |  8 +-
 .../io/util/DataOutputBufferFixed.java  |  2 +-
 .../cassandra/io/util/SafeMemoryWriter.java | 16 ++--
 .../cassandra/io/util/DataOutputTest.java   |  4 +-
 .../cassandra/io/util/SafeMemoryWriterTest.java | 90 
 6 files changed, 110 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/991e1971/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
--
diff --cc src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
index 144edad,7586543..28ca468
--- a/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
+++ b/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
@@@ -38,43 -37,11 +38,43 @@@ public class DataOutputBuffer extends B
  /*
   * Threshold at which resizing transitions from doubling to increasing by 
50%
   */
- private static final long DOUBLING_THRESHOLD = 
Long.getLong(Config.PROPERTY_PREFIX + "DOB_DOUBLING_THRESHOLD_MB", 64);
+ static final long DOUBLING_THRESHOLD = 
Long.getLong(Config.PROPERTY_PREFIX + "DOB_DOUBLING_THRESHOLD_MB", 64);
  
 +/*
 + * Only recycle OutputBuffers up to 1Mb. Larger buffers will be trimmed 
back to this size.
 + */
 +private static final int MAX_RECYCLE_BUFFER_SIZE = 
Integer.getInteger(Config.PROPERTY_PREFIX + "dob_max_recycle_bytes", 1024 * 
1024);
 +
 +private static final int DEFAULT_INITIAL_BUFFER_SIZE = 128;
 +
 +/**
 + * Scratch buffers used mostly for serializing in memory. It's important 
to call #recycle() when finished
 + * to keep the memory overhead from being too large in the system.
 + */
 +public static final FastThreadLocal scratchBuffer = new 
FastThreadLocal()
 +{
 +protected DataOutputBuffer initialValue() throws Exception
 +{
 +return new DataOutputBuffer()
 +{
 +public void close()
 +{
 +if (buffer.capacity() <= MAX_RECYCLE_BUFFER_SIZE)
 +{
 +buffer.clear();
 +}
 +else
 +{
 +buffer = 
ByteBuffer.allocate(DEFAULT_INITIAL_BUFFER_SIZE);
 +}
 +}
 +};
 +}
 +};
 +
  public DataOutputBuffer()
  {
 -this(128);
 +this(DEFAULT_INITIAL_BUFFER_SIZE);
  }
  
  public DataOutputBuffer(int size)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/991e1971/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/991e1971/test/unit/org/apache/cassandra/io/util/DataOutputTest.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[03/10] cassandra git commit: Fix SafeMemoryWriter trimming and behaviour over 2G

2018-08-21 Thread blambov

Fix SafeMemoryWriter trimming and behaviour over 2G

patch by Branimir Lambov; reviewed by Benedict Elliott Smith for CASSANDRA-14649


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/49adbe7e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/49adbe7e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/49adbe7e

Branch: refs/heads/cassandra-3.11
Commit: 49adbe7e0f0c8a83f3b843b65612528498b5c9a5
Parents: 0e81892
Author: Branimir Lambov 
Authored: Thu Aug 16 16:15:07 2018 +0300
Committer: Branimir Lambov 
Committed: Tue Aug 21 11:53:30 2018 +0300

--
 .../io/sstable/IndexSummaryBuilder.java |  4 +-
 .../cassandra/io/util/DataOutputBuffer.java |  8 +-
 .../io/util/DataOutputBufferFixed.java  |  2 +-
 .../cassandra/io/util/SafeMemoryWriter.java | 16 ++--
 .../cassandra/io/util/DataOutputTest.java   |  4 +-
 .../cassandra/io/util/SafeMemoryWriterTest.java | 90 
 6 files changed, 110 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/49adbe7e/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java 
b/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
index 6110afe..0f604e0 100644
--- a/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
+++ b/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
@@ -207,8 +207,8 @@ public class IndexSummaryBuilder implements AutoCloseable
 {
 // this method should only be called when we've finished appending 
records, so we truncate the
 // memory we're using to the exact amount required to represent it 
before building our summary
-entries.setCapacity(entries.length());
-offsets.setCapacity(offsets.length());
+entries.trim();
+offsets.trim();
 }
 
 public IndexSummary build(IPartitioner partitioner)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/49adbe7e/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
--
diff --git a/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java 
b/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
index 6ea6d97..3f1e081 100644
--- a/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
+++ b/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
@@ -37,7 +37,7 @@ public class DataOutputBuffer extends 
BufferedDataOutputStreamPlus
 /*
  * Threshold at which resizing transitions from doubling to increasing by 
50%
  */
-private static final long DOUBLING_THRESHOLD = 
Long.getLong(Config.PROPERTY_PREFIX + "DOB_DOUBLING_THRESHOLD_MB", 64);
+static final long DOUBLING_THRESHOLD = Long.getLong(Config.PROPERTY_PREFIX 
+ "DOB_DOUBLING_THRESHOLD_MB", 64);
 
 public DataOutputBuffer()
 {
@@ -83,7 +83,7 @@ public class DataOutputBuffer extends 
BufferedDataOutputStreamPlus
 @Override
 protected void doFlush(int count) throws IOException
 {
-reallocate(count);
+expandToFit(count);
 }
 
 //Hack for test, make it possible to override checking the buffer capacity
@@ -119,7 +119,7 @@ public class DataOutputBuffer extends 
BufferedDataOutputStreamPlus
 return validateReallocation(newSize);
 }
 
-protected void reallocate(long count)
+protected void expandToFit(long count)
 {
 if (count <= 0)
 return;
@@ -141,7 +141,7 @@ public class DataOutputBuffer extends 
BufferedDataOutputStreamPlus
 public int write(ByteBuffer src) throws IOException
 {
 int count = src.remaining();
-reallocate(count);
+expandToFit(count);
 buffer.put(src);
 return count;
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/49adbe7e/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java
--
diff --git a/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java 
b/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java
index c815c9e..c9767fc 100644
--- a/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java
+++ b/src/java/org/apache/cassandra/io/util/DataOutputBufferFixed.java
@@ -58,7 +58,7 @@ public class DataOutputBufferFixed extends DataOutputBuffer
  * @see org.apache.cassandra.io.util.DataOutputBuffer#reallocate(long)
  */
 @Override
-protected void reallocate(long newSize)
+protected void expandToFit(long newSize)
 {
 throw new BufferOverflowException();
 }

cassandra git commit: Add more testing of uncompressed chunks (CASSANDRA-10520), fix problem with min_compress_ratio: 1 and disallow ratio < 1

2017-09-15 Thread blambov

Repository: cassandra
Updated Branches:
  refs/heads/trunk 587c67e3b -> 17a358c2c


Add more testing of uncompressed chunks (CASSANDRA-10520), fix problem with 
min_compress_ratio: 1 and disallow ratio < 1

patch by Branimir Lambov; reviewed by Dimitar Dimitrov for CASSANDRA-13703


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/17a358c2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/17a358c2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/17a358c2

Branch: refs/heads/trunk
Commit: 17a358c2cc2c583c3e0fa046ca8dee6d743ad1c5
Parents: 587c67e
Author: Branimir Lambov 
Authored: Thu Jul 20 18:07:49 2017 +0300
Committer: Branimir Lambov 
Committed: Fri Sep 15 11:14:18 2017 +0300

--
 CHANGES.txt |  1 +
 .../io/compress/CompressedSequentialWriter.java |  4 ++--
 .../io/util/CompressedChunkReader.java  | 13 ++--
 .../cassandra/schema/CompressionParams.java | 21 +---
 .../cql3/validation/operations/AlterTest.java   | 12 +++
 .../cql3/validation/operations/CreateTest.java  | 12 ++-
 6 files changed, 50 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/17a358c2/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index fbd08e5..afd1c7f 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 4.0
+ * Fix problem with min_compress_ratio: 1 and disallow ratio < 1 
(CASSANDRA-13703)
  * Add extra information to SASI timeout exception (CASSANDRA-13677)
  * Add incremental repair support for --hosts, --force, and subrange repair 
(CASSANDRA-13818)
  * Rework CompactionStrategyManager.getScanners synchronization 
(CASSANDRA-13786)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/17a358c2/src/java/org/apache/cassandra/io/compress/CompressedSequentialWriter.java
--
diff --git 
a/src/java/org/apache/cassandra/io/compress/CompressedSequentialWriter.java 
b/src/java/org/apache/cassandra/io/compress/CompressedSequentialWriter.java
index b4ea61f..8955d4f 100644
--- a/src/java/org/apache/cassandra/io/compress/CompressedSequentialWriter.java
+++ b/src/java/org/apache/cassandra/io/compress/CompressedSequentialWriter.java
@@ -151,7 +151,7 @@ public class CompressedSequentialWriter extends 
SequentialWriter
 int compressedLength = compressed.position();
 uncompressedSize += buffer.position();
 ByteBuffer toWrite = compressed;
-if (compressedLength > maxCompressedLength)
+if (compressedLength >= maxCompressedLength)
 {
 toWrite = buffer;
 compressedLength = buffer.position();
@@ -240,7 +240,7 @@ public class CompressedSequentialWriter extends 
SequentialWriter
 // Repopulate buffer from compressed data
 buffer.clear();
 compressed.flip();
-if (chunkSize <= maxCompressedLength)
+if (chunkSize < maxCompressedLength)
 compressor.uncompress(compressed, buffer);
 else
 buffer.put(compressed);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/17a358c2/src/java/org/apache/cassandra/io/util/CompressedChunkReader.java
--
diff --git a/src/java/org/apache/cassandra/io/util/CompressedChunkReader.java 
b/src/java/org/apache/cassandra/io/util/CompressedChunkReader.java
index 15f9fa0..5ae083b 100644
--- a/src/java/org/apache/cassandra/io/util/CompressedChunkReader.java
+++ b/src/java/org/apache/cassandra/io/util/CompressedChunkReader.java
@@ -20,7 +20,6 @@ package org.apache.cassandra.io.util;
 
 import java.io.IOException;
 import java.nio.ByteBuffer;
-import java.util.concurrent.ThreadLocalRandom;
 
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.primitives.Ints;
@@ -50,9 +49,9 @@ public abstract class CompressedChunkReader extends 
AbstractReaderFileProxy impl
 return metadata.parameters.getCrcCheckChance();
 }
 
-public boolean maybeCheckCrc()
+boolean shouldCheckCrc()
 {
-return metadata.parameters.maybeCheckCrc();
+return metadata.parameters.shouldCheckCrc();
 }
 
 @Override
@@ -116,7 +115,7 @@ public abstract class CompressedChunkReader extends 
AbstractReaderFileProxy impl
 assert position <= fileLength;
 
 CompressionMetadata.Chunk chunk = metadata.chunkFor(position);
-if (chunk.length <= maxCompressedLength)
+if (chunk.length < maxCompressedLength)

[3/3] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2017-03-30 Thread blambov

Merge branch 'cassandra-3.11' into trunk

# Conflicts:
#   test/unit/org/apache/cassandra/db/KeyspaceTest.java


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bb4c5c3c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bb4c5c3c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bb4c5c3c

Branch: refs/heads/trunk
Commit: bb4c5c3c4adf4c76e53fed297036489cee1a2768
Parents: b3465f9 81b8895
Author: Branimir Lambov 
Authored: Thu Mar 30 14:38:23 2017 +0300
Committer: Branimir Lambov 
Committed: Thu Mar 30 14:38:23 2017 +0300

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/db/KeyspaceTest.java   | 72 
 2 files changed, 45 insertions(+), 28 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/bb4c5c3c/CHANGES.txt
--
diff --cc CHANGES.txt
index 2040524,3ead1d1..6a164ee
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,52 -1,5 +1,53 @@@
 +4.0
 + * Outbound TCP connections ignore internode authenticator (CASSANDRA-13324)
 + * Upgrade junit from 4.6 to 4.12 (CASSANDRA-13360)
 + * Cleanup ParentRepairSession after repairs (CASSANDRA-13359)
 + * Incremental repair not streaming correct sstables (CASSANDRA-13328)
 + * Upgrade the jna version to 4.3.0 (CASSANDRA-13300)
 + * Add the currentTimestamp, currentDate, currentTime and currentTimeUUID 
functions (CASSANDRA-13132)
 + * Remove config option index_interval (CASSANDRA-10671)
 + * Reduce lock contention for collection types and serializers 
(CASSANDRA-13271)
 + * Make it possible to override MessagingService.Verb ids (CASSANDRA-13283)
 + * Avoid synchronized on prepareForRepair in ActiveRepairService 
(CASSANDRA-9292)
 + * Adds the ability to use uncompressed chunks in compressed files 
(CASSANDRA-10520)
 + * Don't flush sstables when streaming for incremental repair 
(CASSANDRA-13226)
 + * Remove unused method (CASSANDRA-13227)
 + * Fix minor bugs related to #9143 (CASSANDRA-13217)
 + * Output warning if user increases RF (CASSANDRA-13079)
 + * Remove pre-3.0 streaming compatibility code for 4.0 (CASSANDRA-13081)
 + * Add support for + and - operations on dates (CASSANDRA-11936)
 + * Fix consistency of incrementally repaired data (CASSANDRA-9143)
 + * Increase commitlog version (CASSANDRA-13161)
 + * Make TableMetadata immutable, optimize Schema (CASSANDRA-9425)
 + * Refactor ColumnCondition (CASSANDRA-12981)
 + * Parallelize streaming of different keyspaces (CASSANDRA-4663)
 + * Improved compactions metrics (CASSANDRA-13015)
 + * Speed-up start-up sequence by avoiding un-needed flushes (CASSANDRA-13031)
 + * Use Caffeine (W-TinyLFU) for on-heap caches (CASSANDRA-10855)
 + * Thrift removal (CASSANDRA-5)
 + * Remove pre-3.0 compatibility code for 4.0 (CASSANDRA-12716)
 + * Add column definition kind to dropped columns in schema (CASSANDRA-12705)
 + * Add (automate) Nodetool Documentation (CASSANDRA-12672)
 + * Update bundled cqlsh python driver to 3.7.0 (CASSANDRA-12736)
 + * Reject invalid replication settings when creating or altering a keyspace 
(CASSANDRA-12681)
 + * Clean up the SSTableReader#getScanner API wrt removal of RateLimiter 
(CASSANDRA-12422)
 + * Use new token allocation for non bootstrap case as well (CASSANDRA-13080)
 + * Avoid byte-array copy when key cache is disabled (CASSANDRA-13084)
 + * Require forceful decommission if number of nodes is less than replication 
factor (CASSANDRA-12510)
 + * Allow IN restrictions on column families with collections (CASSANDRA-12654)
 + * Log message size in trace message in OutboundTcpConnection 
(CASSANDRA-13028)
 + * Add timeUnit Days for cassandra-stress (CASSANDRA-13029)
 + * Add mutation size and batch metrics (CASSANDRA-12649)
 + * Add method to get size of endpoints to TokenMetadata (CASSANDRA-12999)
 + * Expose time spent waiting in thread pool queue (CASSANDRA-8398)
 + * Conditionally update index built status to avoid unnecessary flushes 
(CASSANDRA-12969)
 + * cqlsh auto completion: refactor definition of compaction strategy options 
(CASSANDRA-12946)
 + * Add support for arithmetic operators (CASSANDRA-11935)
 + * Add histogram for delay to deliver hints (CASSANDRA-13234)
 +
 +
  3.11.0
+  * Fix testLimitSSTables flake caused by concurrent flush (CASSANDRA-12820)
   * cdc column addition strikes again (CASSANDRA-13382)
   * Fix static column indexes (CASSANDRA-13277) 
   * DataOutputBuffer.asNewBuffer broken (CASSANDRA-13298)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bb4c5c3c/test/unit/org/apache/cassandra/db/KeyspaceTest.java
--

[1/3] cassandra git commit: Fix testLimitSSTables flake caused by concurrent flush

2017-03-30 Thread blambov

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.11 0b1675d43 -> 81b889515
  refs/heads/trunk b3465f937 -> bb4c5c3c4


Fix testLimitSSTables flake caused by concurrent flush

Patch by Branimir Lambov; reviewed by Stefania Alborghetti for
CASSANDRA-12820


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/81b88951
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/81b88951
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/81b88951

Branch: refs/heads/cassandra-3.11
Commit: 81b8895151742668ff5035960612d8c4325a1761
Parents: 0b1675d
Author: Branimir Lambov 
Authored: Mon Jan 23 17:20:43 2017 +0200
Committer: Branimir Lambov 
Committed: Thu Mar 30 14:36:07 2017 +0300

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/db/KeyspaceTest.java   | 74 
 2 files changed, 46 insertions(+), 29 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/81b88951/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 9cde2d8..3ead1d1 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.11.0
+ * Fix testLimitSSTables flake caused by concurrent flush (CASSANDRA-12820)
  * cdc column addition strikes again (CASSANDRA-13382)
  * Fix static column indexes (CASSANDRA-13277) 
  * DataOutputBuffer.asNewBuffer broken (CASSANDRA-13298)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/81b88951/test/unit/org/apache/cassandra/db/KeyspaceTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/KeyspaceTest.java 
b/test/unit/org/apache/cassandra/db/KeyspaceTest.java
index 5036749..f2a9984 100644
--- a/test/unit/org/apache/cassandra/db/KeyspaceTest.java
+++ b/test/unit/org/apache/cassandra/db/KeyspaceTest.java
@@ -24,6 +24,7 @@ import java.util.*;
 import org.apache.cassandra.Util;
 import org.apache.cassandra.cql3.CQLTester;
 import org.apache.cassandra.cql3.ColumnIdentifier;
+import org.apache.cassandra.cql3.UntypedResultSet;
 import org.apache.cassandra.db.rows.Cell;
 import org.apache.cassandra.db.rows.Row;
 import org.apache.cassandra.db.rows.RowIterator;
@@ -41,14 +42,34 @@ import static org.junit.Assert.*;
 
 public class KeyspaceTest extends CQLTester
 {
+// Test needs synchronous table drop to avoid flushes causing flaky 
failures of testLimitSSTables
+
+@Override
+protected String createTable(String query)
+{
+return super.createTable(KEYSPACE_PER_TEST, query);
+}
+
+@Override
+protected UntypedResultSet execute(String query, Object... values) throws 
Throwable
+{
+return executeFormattedQuery(formatQuery(KEYSPACE_PER_TEST, query), 
values);
+}
+
+@Override
+public ColumnFamilyStore getCurrentColumnFamilyStore()
+{
+return super.getCurrentColumnFamilyStore(KEYSPACE_PER_TEST);
+}
+
 @Test
 public void testGetRowNoColumns() throws Throwable
 {
-String tableName = createTable("CREATE TABLE %s (a text, b int, c int, 
PRIMARY KEY (a, b))");
+createTable("CREATE TABLE %s (a text, b int, c int, PRIMARY KEY (a, 
b))");
 
 execute("INSERT INTO %s (a, b, c) VALUES (?, ?, ?)", "0", 0, 0);
 
-final ColumnFamilyStore cfs = 
Keyspace.open(KEYSPACE).getColumnFamilyStore(tableName);
+final ColumnFamilyStore cfs = getCurrentColumnFamilyStore();
 
 for (int round = 0; round < 2; round++)
 {
@@ -69,12 +90,12 @@ public class KeyspaceTest extends CQLTester
 @Test
 public void testGetRowSingleColumn() throws Throwable
 {
-String tableName = createTable("CREATE TABLE %s (a text, b int, c int, 
PRIMARY KEY (a, b))");
+createTable("CREATE TABLE %s (a text, b int, c int, PRIMARY KEY (a, 
b))");
 
 for (int i = 0; i < 2; i++)
 execute("INSERT INTO %s (a, b, c) VALUES (?, ?, ?)", "0", i, i);
 
-final ColumnFamilyStore cfs = 
Keyspace.open(KEYSPACE).getColumnFamilyStore(tableName);
+final ColumnFamilyStore cfs = getCurrentColumnFamilyStore();
 
 for (int round = 0; round < 2; round++)
 {
@@ -104,11 +125,11 @@ public class KeyspaceTest extends CQLTester
 @Test
 public void testGetSliceBloomFilterFalsePositive() throws Throwable
 {
-String tableName = createTable("CREATE TABLE %s (a text, b int, c int, 
PRIMARY KEY (a, b))");
+createTable("CREATE TABLE %s (a text, b int, c int, PRIMARY KEY (a, 
b))");
 
 execute("INSERT INTO %s (a, b, c) VALUES (?, ?, ?)", "1", 1, 1);
 
-final ColumnFamilyStore cfs = 
Keyspace.open(KEYSPACE).getColumnFamilyStore(tableName);
+final

[2/3] cassandra git commit: Fix testLimitSSTables flake caused by concurrent flush

2017-03-30 Thread blambov

Fix testLimitSSTables flake caused by concurrent flush

Patch by Branimir Lambov; reviewed by Stefania Alborghetti for
CASSANDRA-12820


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/81b88951
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/81b88951
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/81b88951

Branch: refs/heads/trunk
Commit: 81b8895151742668ff5035960612d8c4325a1761
Parents: 0b1675d
Author: Branimir Lambov 
Authored: Mon Jan 23 17:20:43 2017 +0200
Committer: Branimir Lambov 
Committed: Thu Mar 30 14:36:07 2017 +0300

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/db/KeyspaceTest.java   | 74 
 2 files changed, 46 insertions(+), 29 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/81b88951/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 9cde2d8..3ead1d1 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.11.0
+ * Fix testLimitSSTables flake caused by concurrent flush (CASSANDRA-12820)
  * cdc column addition strikes again (CASSANDRA-13382)
  * Fix static column indexes (CASSANDRA-13277) 
  * DataOutputBuffer.asNewBuffer broken (CASSANDRA-13298)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/81b88951/test/unit/org/apache/cassandra/db/KeyspaceTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/KeyspaceTest.java 
b/test/unit/org/apache/cassandra/db/KeyspaceTest.java
index 5036749..f2a9984 100644
--- a/test/unit/org/apache/cassandra/db/KeyspaceTest.java
+++ b/test/unit/org/apache/cassandra/db/KeyspaceTest.java
@@ -24,6 +24,7 @@ import java.util.*;
 import org.apache.cassandra.Util;
 import org.apache.cassandra.cql3.CQLTester;
 import org.apache.cassandra.cql3.ColumnIdentifier;
+import org.apache.cassandra.cql3.UntypedResultSet;
 import org.apache.cassandra.db.rows.Cell;
 import org.apache.cassandra.db.rows.Row;
 import org.apache.cassandra.db.rows.RowIterator;
@@ -41,14 +42,34 @@ import static org.junit.Assert.*;
 
 public class KeyspaceTest extends CQLTester
 {
+// Test needs synchronous table drop to avoid flushes causing flaky 
failures of testLimitSSTables
+
+@Override
+protected String createTable(String query)
+{
+return super.createTable(KEYSPACE_PER_TEST, query);
+}
+
+@Override
+protected UntypedResultSet execute(String query, Object... values) throws 
Throwable
+{
+return executeFormattedQuery(formatQuery(KEYSPACE_PER_TEST, query), 
values);
+}
+
+@Override
+public ColumnFamilyStore getCurrentColumnFamilyStore()
+{
+return super.getCurrentColumnFamilyStore(KEYSPACE_PER_TEST);
+}
+
 @Test
 public void testGetRowNoColumns() throws Throwable
 {
-String tableName = createTable("CREATE TABLE %s (a text, b int, c int, 
PRIMARY KEY (a, b))");
+createTable("CREATE TABLE %s (a text, b int, c int, PRIMARY KEY (a, 
b))");
 
 execute("INSERT INTO %s (a, b, c) VALUES (?, ?, ?)", "0", 0, 0);
 
-final ColumnFamilyStore cfs = 
Keyspace.open(KEYSPACE).getColumnFamilyStore(tableName);
+final ColumnFamilyStore cfs = getCurrentColumnFamilyStore();
 
 for (int round = 0; round < 2; round++)
 {
@@ -69,12 +90,12 @@ public class KeyspaceTest extends CQLTester
 @Test
 public void testGetRowSingleColumn() throws Throwable
 {
-String tableName = createTable("CREATE TABLE %s (a text, b int, c int, 
PRIMARY KEY (a, b))");
+createTable("CREATE TABLE %s (a text, b int, c int, PRIMARY KEY (a, 
b))");
 
 for (int i = 0; i < 2; i++)
 execute("INSERT INTO %s (a, b, c) VALUES (?, ?, ?)", "0", i, i);
 
-final ColumnFamilyStore cfs = 
Keyspace.open(KEYSPACE).getColumnFamilyStore(tableName);
+final ColumnFamilyStore cfs = getCurrentColumnFamilyStore();
 
 for (int round = 0; round < 2; round++)
 {
@@ -104,11 +125,11 @@ public class KeyspaceTest extends CQLTester
 @Test
 public void testGetSliceBloomFilterFalsePositive() throws Throwable
 {
-String tableName = createTable("CREATE TABLE %s (a text, b int, c int, 
PRIMARY KEY (a, b))");
+createTable("CREATE TABLE %s (a text, b int, c int, PRIMARY KEY (a, 
b))");
 
 execute("INSERT INTO %s (a, b, c) VALUES (?, ?, ?)", "1", 1, 1);
 
-final ColumnFamilyStore cfs = 
Keyspace.open(KEYSPACE).getColumnFamilyStore(tableName);
+final ColumnFamilyStore cfs = getCurrentColumnFamilyStore();
 
 // check empty reads on the partitions before and after the existing 
one

cassandra git commit: Change default min_compress_ratio to 0 to avoid sending parameter to old nodes

2017-03-29 Thread blambov

Repository: cassandra
Updated Branches:
  refs/heads/trunk 07795f101 -> b953f9eb2


Change default min_compress_ratio to 0 to avoid sending parameter to old
nodes

Follow-up commit to CASSANDRA-10520

patch by Branimir Lambov; reviewed by Robert Stupp for CASSANDRA-10520


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b953f9eb
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b953f9eb
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b953f9eb

Branch: refs/heads/trunk
Commit: b953f9eb2a3a1ba992fbb561b1e35531607739f3
Parents: 07795f1
Author: Branimir Lambov 
Authored: Fri Feb 24 11:02:08 2017 +0200
Committer: Branimir Lambov 
Committed: Wed Mar 29 14:42:29 2017 +0300

--
 pylib/cqlshlib/test/test_cqlsh_output.py  |  2 +-
 .../org/apache/cassandra/schema/CompressionParams.java|  7 +--
 .../cassandra/cql3/validation/operations/AlterTest.java   |  8 
 .../cassandra/cql3/validation/operations/CreateTest.java  | 10 +-
 4 files changed, 15 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b953f9eb/pylib/cqlshlib/test/test_cqlsh_output.py
--
diff --git a/pylib/cqlshlib/test/test_cqlsh_output.py 
b/pylib/cqlshlib/test/test_cqlsh_output.py
index f4aaa57..b7240f1 100644
--- a/pylib/cqlshlib/test/test_cqlsh_output.py
+++ b/pylib/cqlshlib/test/test_cqlsh_output.py
@@ -615,7 +615,7 @@ class TestCqlshOutput(BaseTestCase):
 AND cdc = false
 AND comment = ''
 AND compaction = {'class': 
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
'max_threshold': '32', 'min_threshold': '4'}
-AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor', 'min_compress_ratio': '1.1'}
+AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}
 AND crc_check_chance = 1.0
 AND dclocal_read_repair_chance = 0.1
 AND default_time_to_live = 0

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b953f9eb/src/java/org/apache/cassandra/schema/CompressionParams.java
--
diff --git a/src/java/org/apache/cassandra/schema/CompressionParams.java 
b/src/java/org/apache/cassandra/schema/CompressionParams.java
index a319311..c4b52df 100644
--- a/src/java/org/apache/cassandra/schema/CompressionParams.java
+++ b/src/java/org/apache/cassandra/schema/CompressionParams.java
@@ -54,7 +54,9 @@ public final class CompressionParams
 private static volatile boolean hasLoggedCrcCheckChanceWarning;
 
 public static final int DEFAULT_CHUNK_LENGTH = 65536;
-public static final double DEFAULT_MIN_COMPRESS_RATIO = 1.1;
+public static final double DEFAULT_MIN_COMPRESS_RATIO = 0.0;// 
Since pre-4.0 versions do not understand the
+// new 
compression parameter we can't use a
+// 
different default value.
 public static final IVersionedSerializer serializer = 
new Serializer();
 
 public static final String CLASS = "class";
@@ -499,7 +501,8 @@ public final class CompressionParams
 Map options = new HashMap<>(otherOptions);
 options.put(CLASS, sstableCompressor.getClass().getName());
 options.put(CHUNK_LENGTH_IN_KB, chunkLengthInKB());
-options.put(MIN_COMPRESS_RATIO, String.valueOf(minCompressRatio));
+if (minCompressRatio != DEFAULT_MIN_COMPRESS_RATIO)
+options.put(MIN_COMPRESS_RATIO, String.valueOf(minCompressRatio));
 
 return options;
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b953f9eb/test/unit/org/apache/cassandra/cql3/validation/operations/AlterTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/cql3/validation/operations/AlterTest.java 
b/test/unit/org/apache/cassandra/cql3/validation/operations/AlterTest.java
index 3e6f0db..dce72f9 100644
--- a/test/unit/org/apache/cassandra/cql3/validation/operations/AlterTest.java
+++ b/test/unit/org/apache/cassandra/cql3/validation/operations/AlterTest.java
@@ -328,7 +328,7 @@ public class AlterTest extends CQLTester
   SchemaKeyspace.TABLES),
KEYSPACE,
currentTable()),
-   row(map("chunk_length_in_kb", "64", "class",

[2/2] cassandra git commit: Adds the ability to use uncompressed chunks in compressed files

2017-02-21 Thread blambov

Adds the ability to use uncompressed chunks in compressed files

Triggered when size of compressed data surpasses a configurable
threshold.

Patch by Branimir Lambov; reviewed by Ropert Stupp for CASSANDRA-10520


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f97db26f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f97db26f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f97db26f

Branch: refs/heads/trunk
Commit: f97db26f8e9989d2294cccbea8a06589253313f2
Parents: 0ae0495
Author: Branimir Lambov 
Authored: Wed Oct 14 16:50:52 2015 +0300
Committer: Branimir Lambov 
Committed: Tue Feb 21 18:05:18 2017 +0200

--
 CHANGES.txt |   1 +
 .../io/compress/CompressedSequentialWriter.java |  25 +++-
 .../io/compress/CompressionMetadata.java|  27 +++-
 .../cassandra/io/sstable/format/Version.java|   2 +
 .../io/sstable/format/big/BigFormat.java|  12 +-
 .../io/util/CompressedChunkReader.java  | 100 +++--
 .../cassandra/schema/CompressionParams.java | 146 ++-
 .../cassandra/service/AbstractReadExecutor.java |  14 +-
 .../compress/CompressedInputStream.java |  11 +-
 .../streaming/messages/FileMessageHeader.java   |   5 +-
 .../streaming/messages/StreamMessage.java   |   4 +-
 .../legacy_na_clust/na-1-big-CompressionInfo.db | Bin 0 -> 87 bytes
 .../legacy_na_clust/na-1-big-Data.db| Bin 0 -> 5359 bytes
 .../legacy_na_clust/na-1-big-Digest.crc32   |   1 +
 .../legacy_na_clust/na-1-big-Filter.db  | Bin 0 -> 24 bytes
 .../legacy_na_clust/na-1-big-Index.db   | Bin 0 -> 157553 bytes
 .../legacy_na_clust/na-1-big-Statistics.db  | Bin 0 -> 7070 bytes
 .../legacy_na_clust/na-1-big-Summary.db | Bin 0 -> 47 bytes
 .../legacy_na_clust/na-1-big-TOC.txt|   8 +
 .../legacy_na_clust/na-1-bti-CompressionInfo.db | Bin 0 -> 83 bytes
 .../legacy_na_clust/na-1-bti-Data.db| Bin 0 -> 5315 bytes
 .../legacy_na_clust/na-1-bti-Digest.crc32   |   1 +
 .../legacy_na_clust/na-1-bti-Filter.db  | Bin 0 -> 24 bytes
 .../legacy_na_clust/na-1-bti-Partitions.db  | Bin 0 -> 63 bytes
 .../legacy_na_clust/na-1-bti-Rows.db| Bin 0 -> 738 bytes
 .../legacy_na_clust/na-1-bti-Statistics.db  | Bin 0 -> 7086 bytes
 .../legacy_na_clust/na-1-bti-TOC.txt|   8 +
 .../na-1-big-CompressionInfo.db | Bin 0 -> 87 bytes
 .../legacy_na_clust_compact/na-1-big-Data.db| Bin 0 -> 5426 bytes
 .../na-1-big-Digest.crc32   |   1 +
 .../legacy_na_clust_compact/na-1-big-Filter.db  | Bin 0 -> 24 bytes
 .../legacy_na_clust_compact/na-1-big-Index.db   | Bin 0 -> 157553 bytes
 .../na-1-big-Statistics.db  | Bin 0 -> 7070 bytes
 .../legacy_na_clust_compact/na-1-big-Summary.db | Bin 0 -> 47 bytes
 .../legacy_na_clust_compact/na-1-big-TOC.txt|   8 +
 .../na-1-bti-CompressionInfo.db | Bin 0 -> 83 bytes
 .../legacy_na_clust_compact/na-1-bti-Data.db| Bin 0 -> 5398 bytes
 .../na-1-bti-Digest.crc32   |   1 +
 .../legacy_na_clust_compact/na-1-bti-Filter.db  | Bin 0 -> 24 bytes
 .../na-1-bti-Partitions.db  | Bin 0 -> 63 bytes
 .../legacy_na_clust_compact/na-1-bti-Rows.db| Bin 0 -> 738 bytes
 .../na-1-bti-Statistics.db  | Bin 0 -> 7086 bytes
 .../legacy_na_clust_compact/na-1-bti-TOC.txt|   8 +
 .../na-1-big-CompressionInfo.db | Bin 0 -> 79 bytes
 .../legacy_na_clust_counter/na-1-big-Data.db| Bin 0 -> 6015 bytes
 .../na-1-big-Digest.crc32   |   1 +
 .../legacy_na_clust_counter/na-1-big-Filter.db  | Bin 0 -> 24 bytes
 .../legacy_na_clust_counter/na-1-big-Index.db   | Bin 0 -> 157553 bytes
 .../na-1-big-Statistics.db  | Bin 0 -> 7079 bytes
 .../legacy_na_clust_counter/na-1-big-Summary.db | Bin 0 -> 47 bytes
 .../legacy_na_clust_counter/na-1-big-TOC.txt|   8 +
 .../na-1-bti-CompressionInfo.db | Bin 0 -> 75 bytes
 .../legacy_na_clust_counter/na-1-bti-Data.db| Bin 0 -> 5950 bytes
 .../na-1-bti-Digest.crc32   |   1 +
 .../legacy_na_clust_counter/na-1-bti-Filter.db  | Bin 0 -> 24 bytes
 .../na-1-bti-Partitions.db  | Bin 0 -> 63 bytes
 .../legacy_na_clust_counter/na-1-bti-Rows.db| Bin 0 -> 738 bytes
 .../na-1-bti-Statistics.db  | Bin 0 -> 7095 bytes
 .../legacy_na_clust_counter/na-1-bti-TOC.txt|   8 +
 .../na-1-big-CompressionInfo.db | Bin 0 -> 79 bytes
 .../na-1-big-Data.db| Bin 0 -> 6014 bytes
 .../na-1-big-Digest.crc32   |   1 +
 .../na-1-big-Filter.db  | Bin 0 -> 24 bytes

[1/2] cassandra git commit: Adds the ability to use uncompressed chunks in compressed files

2017-02-21 Thread blambov

Repository: cassandra
Updated Branches:
  refs/heads/trunk 0ae0495ea -> f97db26f8


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f97db26f/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-big-TOC.txt
--
diff --git 
a/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-big-TOC.txt 
b/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-big-TOC.txt
new file mode 100644
index 000..ae620de
--- /dev/null
+++ 
b/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-big-TOC.txt
@@ -0,0 +1,8 @@
+Summary.db
+Filter.db
+Index.db
+CompressionInfo.db
+Data.db
+TOC.txt
+Statistics.db
+Digest.crc32

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f97db26f/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-CompressionInfo.db
--
diff --git 
a/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-CompressionInfo.db
 
b/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-CompressionInfo.db
new file mode 100644
index 000..0b7faea
Binary files /dev/null and 
b/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-CompressionInfo.db
 differ

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f97db26f/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Data.db
--
diff --git 
a/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Data.db 
b/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Data.db
new file mode 100644
index 000..277996b
Binary files /dev/null and 
b/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Data.db 
differ

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f97db26f/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Digest.crc32
--
diff --git 
a/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Digest.crc32
 
b/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Digest.crc32
new file mode 100644
index 000..654f52b
--- /dev/null
+++ 
b/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Digest.crc32
@@ -0,0 +1 @@
+4102718625
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f97db26f/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Filter.db
--
diff --git 
a/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Filter.db
 
b/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Filter.db
new file mode 100644
index 000..2e1d5d2
Binary files /dev/null and 
b/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Filter.db
 differ

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f97db26f/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Partitions.db
--
diff --git 
a/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Partitions.db
 
b/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Partitions.db
new file mode 100644
index 000..f297888
Binary files /dev/null and 
b/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Partitions.db
 differ

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f97db26f/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Rows.db
--
diff --git 
a/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Rows.db 
b/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Rows.db
new file mode 100644
index 000..e69de29

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f97db26f/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Statistics.db
--
diff --git 
a/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Statistics.db
 
b/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Statistics.db
new file mode 100644
index 000..62b5d3f
Binary files /dev/null and 
b/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-Statistics.db
 differ

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f97db26f/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-TOC.txt
--
diff --git 
a/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-TOC.txt 
b/test/data/legacy-sstables/na/legacy_tables/legacy_na_simple/na-1-bti-TOC.txt
new file mode 100644
index 000..c20f4a8
---

[1/3] cassandra git commit: Fix flaky GcCompactionTest

2017-01-25 Thread blambov

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.11 601606719 -> 237e14dd9
  refs/heads/trunk 3465799a3 -> 2987a7093


Fix flaky GcCompactionTest

Patch by Branimir Lambov; reviewed by Stefania Alborghetti for
CASSANDRA-12664

Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/237e14dd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/237e14dd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/237e14dd

Branch: refs/heads/cassandra-3.11
Commit: 237e14dd99cf3975d54cb1e41126c40bc183f3b4
Parents: 6016067
Author: Branimir Lambov 
Authored: Mon Jan 23 17:42:19 2017 +0200
Committer: Branimir Lambov 
Committed: Wed Jan 25 10:10:43 2017 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/cql3/GcCompactionTest.java | 25 
 2 files changed, 26 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/237e14dd/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 2f0e8f1..55762e2 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.10
+ * Fix flaky GcCompactionTest (CASSANDRA-12664)
  * Fix TestHintedHandoff.hintedhandoff_decom_test (CASSANDRA-13058)
  * Fixed query monitoring for range queries (CASSANDRA-13050)
  * Remove outboundBindAny configuration property (CASSANDRA-12673)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/237e14dd/test/unit/org/apache/cassandra/cql3/GcCompactionTest.java
--
diff --git a/test/unit/org/apache/cassandra/cql3/GcCompactionTest.java 
b/test/unit/org/apache/cassandra/cql3/GcCompactionTest.java
index 6fed033..a31b9a1 100644
--- a/test/unit/org/apache/cassandra/cql3/GcCompactionTest.java
+++ b/test/unit/org/apache/cassandra/cql3/GcCompactionTest.java
@@ -41,6 +41,31 @@ public class GcCompactionTest extends CQLTester
 static final int KEY_COUNT = 10;
 static final int CLUSTERING_COUNT = 20;
 
+// Test needs synchronous table drop to avoid flushes causing flaky 
failures
+
+@Override
+protected String createTable(String query)
+{
+return super.createTable(KEYSPACE_PER_TEST, query);
+}
+
+@Override
+protected UntypedResultSet execute(String query, Object... values) throws 
Throwable
+{
+return executeFormattedQuery(formatQuery(KEYSPACE_PER_TEST, query), 
values);
+}
+
+@Override
+public ColumnFamilyStore getCurrentColumnFamilyStore()
+{
+return super.getCurrentColumnFamilyStore(KEYSPACE_PER_TEST);
+}
+
+public void flush()
+{
+flush(KEYSPACE_PER_TEST);
+}
+
 @Test
 public void testGcCompactionPartitions() throws Throwable
 {

[3/3] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2017-01-25 Thread blambov

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2987a709
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2987a709
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2987a709

Branch: refs/heads/trunk
Commit: 2987a7093c6e167649274f4aa8bf0242eb4dea91
Parents: 3465799 237e14d
Author: Branimir Lambov 
Authored: Wed Jan 25 10:17:15 2017 +0200
Committer: Branimir Lambov 
Committed: Wed Jan 25 10:17:15 2017 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/cql3/GcCompactionTest.java | 25 
 2 files changed, 26 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/2987a709/CHANGES.txt
--
diff --cc CHANGES.txt
index dd25cac,55762e2..e5bb09d
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,35 -1,5 +1,36 @@@
 +4.0
 + * Parallelize streaming of different keyspaces (4663)
 + * Improved compactions metrics (CASSANDRA-13015)
 + * Speed-up start-up sequence by avoiding un-needed flushes (CASSANDRA-13031)
 + * Use Caffeine (W-TinyLFU) for on-heap caches (CASSANDRA-10855)
 + * Thrift removal (CASSANDRA-5)
 + * Remove pre-3.0 compatibility code for 4.0 (CASSANDRA-12716)
 + * Add column definition kind to dropped columns in schema (CASSANDRA-12705)
 + * Add (automate) Nodetool Documentation (CASSANDRA-12672)
 + * Update bundled cqlsh python driver to 3.7.0 (CASSANDRA-12736)
 + * Reject invalid replication settings when creating or altering a keyspace 
(CASSANDRA-12681)
 + * Clean up the SSTableReader#getScanner API wrt removal of RateLimiter 
(CASSANDRA-12422)
 + * Use new token allocation for non bootstrap case as well (CASSANDRA-13080)
 + * Avoid byte-array copy when key cache is disabled (CASSANDRA-13084)
 + * More fixes to the TokenAllocator (CASSANDRA-12990)
 + * Require forceful decommission if number of nodes is less than replication 
factor (CASSANDRA-12510)
 + * Allow IN restrictions on column families with collections (CASSANDRA-12654)
 + * Move to FastThreadLocalThread and FastThreadLocal (CASSANDRA-13034)
 + * nodetool stopdaemon errors out (CASSANDRA-13030)
 + * Log message size in trace message in OutboundTcpConnection 
(CASSANDRA-13028)
 + * Add timeUnit Days for cassandra-stress (CASSANDRA-13029)
 + * Add mutation size and batch metrics (CASSANDRA-12649)
 + * Add method to get size of endpoints to TokenMetadata (CASSANDRA-12999)
 + * Fix primary index calculation for SASI (CASSANDRA-12910)
 + * Expose time spent waiting in thread pool queue (CASSANDRA-8398)
 + * Conditionally update index built status to avoid unnecessary flushes 
(CASSANDRA-12969)
 + * NoReplicationTokenAllocator should work with zero replication factor 
(CASSANDRA-12983)
 + * cqlsh auto completion: refactor definition of compaction strategy options 
(CASSANDRA-12946)
 + * Add support for arithmetic operators (CASSANDRA-11935)
 + * Tables in system_distributed should not use gcgs of 0 (CASSANDRA-12954)
 +
  3.10
+  * Fix flaky GcCompactionTest (CASSANDRA-12664)
   * Fix TestHintedHandoff.hintedhandoff_decom_test (CASSANDRA-13058)
   * Fixed query monitoring for range queries (CASSANDRA-13050)
   * Remove outboundBindAny configuration property (CASSANDRA-12673)

[2/3] cassandra git commit: Fix flaky GcCompactionTest

2017-01-25 Thread blambov

Fix flaky GcCompactionTest

Patch by Branimir Lambov; reviewed by Stefania Alborghetti for
CASSANDRA-12664

Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/237e14dd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/237e14dd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/237e14dd

Branch: refs/heads/trunk
Commit: 237e14dd99cf3975d54cb1e41126c40bc183f3b4
Parents: 6016067
Author: Branimir Lambov 
Authored: Mon Jan 23 17:42:19 2017 +0200
Committer: Branimir Lambov 
Committed: Wed Jan 25 10:10:43 2017 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/cql3/GcCompactionTest.java | 25 
 2 files changed, 26 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/237e14dd/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 2f0e8f1..55762e2 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.10
+ * Fix flaky GcCompactionTest (CASSANDRA-12664)
  * Fix TestHintedHandoff.hintedhandoff_decom_test (CASSANDRA-13058)
  * Fixed query monitoring for range queries (CASSANDRA-13050)
  * Remove outboundBindAny configuration property (CASSANDRA-12673)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/237e14dd/test/unit/org/apache/cassandra/cql3/GcCompactionTest.java
--
diff --git a/test/unit/org/apache/cassandra/cql3/GcCompactionTest.java 
b/test/unit/org/apache/cassandra/cql3/GcCompactionTest.java
index 6fed033..a31b9a1 100644
--- a/test/unit/org/apache/cassandra/cql3/GcCompactionTest.java
+++ b/test/unit/org/apache/cassandra/cql3/GcCompactionTest.java
@@ -41,6 +41,31 @@ public class GcCompactionTest extends CQLTester
 static final int KEY_COUNT = 10;
 static final int CLUSTERING_COUNT = 20;
 
+// Test needs synchronous table drop to avoid flushes causing flaky 
failures
+
+@Override
+protected String createTable(String query)
+{
+return super.createTable(KEYSPACE_PER_TEST, query);
+}
+
+@Override
+protected UntypedResultSet execute(String query, Object... values) throws 
Throwable
+{
+return executeFormattedQuery(formatQuery(KEYSPACE_PER_TEST, query), 
values);
+}
+
+@Override
+public ColumnFamilyStore getCurrentColumnFamilyStore()
+{
+return super.getCurrentColumnFamilyStore(KEYSPACE_PER_TEST);
+}
+
+public void flush()
+{
+flush(KEYSPACE_PER_TEST);
+}
+
 @Test
 public void testGcCompactionPartitions() throws Throwable
 {

[01/12] cassandra git commit: Revert "Make sure sstables only get committed when it's safe to discard commit log records"

2016-12-06 Thread blambov

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.11 2f268eda3 -> bed3def9a
  refs/heads/cassandra-3.X 5439d94c5 -> f1423806e
  refs/heads/trunk 9a7baa145 -> 48591489d


Revert "Make sure sstables only get committed when it's safe to discard commit 
log records"

This reverts commit 6f90e55e7e23cbe814a3232c8d1ec67f2ff2a537 as it was using a 
wrong version of the patch.


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d2ba715f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d2ba715f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d2ba715f

Branch: refs/heads/cassandra-3.11
Commit: d2ba715f2456e1aa821c01941f90b6a58f54e6c4
Parents: 6f90e55
Author: Branimir Lambov 
Authored: Tue Dec 6 14:06:48 2016 +0200
Committer: Branimir Lambov 
Committed: Tue Dec 6 14:06:48 2016 +0200

--
 CHANGES.txt |  1 -
 .../apache/cassandra/db/ColumnFamilyStore.java  | 77 -
 src/java/org/apache/cassandra/db/Memtable.java  | 81 ++
 .../miscellaneous/ColumnFamilyStoreTest.java| 90 
 4 files changed, 63 insertions(+), 186 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d2ba715f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 5242adf..5cacdd0 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,4 @@
 3.0.11
- * Make sure sstables only get committed when it's safe to discard commit log 
records (CASSANDRA-12956)
  * Reject default_time_to_live option when creating or altering MVs 
(CASSANDRA-12868)
  * Nodetool should use a more sane max heap size (CASSANDRA-12739)
  * LocalToken ensures token values are cloned on heap (CASSANDRA-12651)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d2ba715f/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 113e10d..d2a51a9 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -63,7 +63,6 @@ import org.apache.cassandra.io.FSWriteError;
 import org.apache.cassandra.io.sstable.Component;
 import org.apache.cassandra.io.sstable.Descriptor;
 import org.apache.cassandra.io.sstable.SSTableMultiWriter;
-import org.apache.cassandra.io.sstable.SSTableTxnWriter;
 import org.apache.cassandra.io.sstable.format.*;
 import org.apache.cassandra.io.sstable.format.big.BigFormat;
 import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
@@ -82,7 +81,6 @@ import org.json.simple.JSONArray;
 import org.json.simple.JSONObject;
 
 import static org.apache.cassandra.utils.Throwables.maybeFail;
-import static org.apache.cassandra.utils.Throwables.merge;
 
 public class ColumnFamilyStore implements ColumnFamilyStoreMBean
 {
@@ -126,8 +124,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 private static final Logger logger = 
LoggerFactory.getLogger(ColumnFamilyStore.class);
 
-@VisibleForTesting
-public static final ExecutorService flushExecutor = new 
JMXEnabledThreadPoolExecutor(DatabaseDescriptor.getFlushWriters(),
+private static final ExecutorService flushExecutor = new 
JMXEnabledThreadPoolExecutor(DatabaseDescriptor.getFlushWriters(),

   StageManager.KEEPALIVE,

   TimeUnit.SECONDS,

   new LinkedBlockingQueue(),
@@ -924,9 +921,8 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 {
 final boolean flushSecondaryIndexes;
 final OpOrder.Barrier writeBarrier;
-final CountDownLatch memtablesFlushLatch = new CountDownLatch(1);
-final CountDownLatch secondaryIndexFlushLatch = new CountDownLatch(1);
-volatile Throwable flushFailure = null;
+final CountDownLatch latch = new CountDownLatch(1);
+volatile FSWriteError flushFailure = null;
 final List memtables;
 
 private PostFlush(boolean flushSecondaryIndexes, OpOrder.Barrier 
writeBarrier,
@@ -947,27 +943,15 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
  * TODO: SecondaryIndex should support setBarrier(), so custom 
implementations can co-ordinate exactly
  * with CL as we do with memtables/CFS-backed SecondaryIndexes.
  */
-

[06/12] cassandra git commit: Make sure sstables only get committed when it's safe to discard commit log records

2016-12-06 Thread blambov

Make sure sstables only get committed when it's safe to discard commit log 
records

Patch by Alex Petrov; reviewed by Branimir Lambov for CASSANDRA-12956


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0ecef315
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0ecef315
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0ecef315

Branch: refs/heads/trunk
Commit: 0ecef31548c287ac2d9f818413457bc947362733
Parents: d2ba715
Author: Alex Petrov 
Authored: Tue Nov 29 22:58:36 2016 +0100
Committer: Branimir Lambov 
Committed: Tue Dec 6 14:10:00 2016 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 45 +---
 .../apache/cassandra/index/CustomIndexTest.java | 37 
 3 files changed, 58 insertions(+), 25 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/0ecef315/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 5cacdd0..5242adf 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Make sure sstables only get committed when it's safe to discard commit log 
records (CASSANDRA-12956)
  * Reject default_time_to_live option when creating or altering MVs 
(CASSANDRA-12868)
  * Nodetool should use a more sane max heap size (CASSANDRA-12739)
  * LocalToken ensures token values are cloned on heap (CASSANDRA-12651)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0ecef315/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index d2a51a9..71e1653 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -919,34 +919,17 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
  */
 private final class PostFlush implements Callable
 {
-final boolean flushSecondaryIndexes;
-final OpOrder.Barrier writeBarrier;
 final CountDownLatch latch = new CountDownLatch(1);
-volatile FSWriteError flushFailure = null;
+volatile Throwable flushFailure = null;
 final List memtables;
 
-private PostFlush(boolean flushSecondaryIndexes, OpOrder.Barrier 
writeBarrier,
-  List memtables)
+private PostFlush(List memtables)
 {
-this.writeBarrier = writeBarrier;
-this.flushSecondaryIndexes = flushSecondaryIndexes;
 this.memtables = memtables;
 }
 
 public ReplayPosition call()
 {
-writeBarrier.await();
-
-/**
- * we can flush 2is as soon as the barrier completes, as they will 
be consistent with (or ahead of) the
- * flushed memtables and CL position, which is as good as we can 
guarantee.
- * TODO: SecondaryIndex should support setBarrier(), so custom 
implementations can co-ordinate exactly
- * with CL as we do with memtables/CFS-backed SecondaryIndexes.
- */
-
-if (flushSecondaryIndexes)
-indexManager.flushAllNonCFSBackedIndexesBlocking();
-
 try
 {
 // we wait on the latch for the commitLogUpperBound to be set, 
and so that waiters
@@ -970,7 +953,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 metric.pendingFlushes.dec();
 
 if (flushFailure != null)
-throw flushFailure;
+Throwables.propagate(flushFailure);
 
 return commitLogUpperBound;
 }
@@ -1029,7 +1012,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 // since this happens after wiring up the commitLogUpperBound, we 
also know all operations with earlier
 // replay positions have also completed, i.e. the memtables are 
done and ready to flush
 writeBarrier.issue();
-postFlush = new PostFlush(!truncate, writeBarrier, memtables);
+postFlush = new PostFlush(memtables);
 }
 
 public void run()
@@ -1047,24 +1030,36 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 try
 {
+boolean flushNonCf2i = true;
 for (Memtable memtable : memtables)
 {
 Collection readers = 
Collections.emptyList();
 if (!memtable.isClean() && !truncate)
+{
+

[05/12] cassandra git commit: Make sure sstables only get committed when it's safe to discard commit log records

2016-12-06 Thread blambov

Make sure sstables only get committed when it's safe to discard commit log 
records

Patch by Alex Petrov; reviewed by Branimir Lambov for CASSANDRA-12956


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0ecef315
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0ecef315
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0ecef315

Branch: refs/heads/cassandra-3.X
Commit: 0ecef31548c287ac2d9f818413457bc947362733
Parents: d2ba715
Author: Alex Petrov 
Authored: Tue Nov 29 22:58:36 2016 +0100
Committer: Branimir Lambov 
Committed: Tue Dec 6 14:10:00 2016 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 45 +---
 .../apache/cassandra/index/CustomIndexTest.java | 37 
 3 files changed, 58 insertions(+), 25 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/0ecef315/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 5cacdd0..5242adf 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Make sure sstables only get committed when it's safe to discard commit log 
records (CASSANDRA-12956)
  * Reject default_time_to_live option when creating or altering MVs 
(CASSANDRA-12868)
  * Nodetool should use a more sane max heap size (CASSANDRA-12739)
  * LocalToken ensures token values are cloned on heap (CASSANDRA-12651)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0ecef315/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index d2a51a9..71e1653 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -919,34 +919,17 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
  */
 private final class PostFlush implements Callable
 {
-final boolean flushSecondaryIndexes;
-final OpOrder.Barrier writeBarrier;
 final CountDownLatch latch = new CountDownLatch(1);
-volatile FSWriteError flushFailure = null;
+volatile Throwable flushFailure = null;
 final List memtables;
 
-private PostFlush(boolean flushSecondaryIndexes, OpOrder.Barrier 
writeBarrier,
-  List memtables)
+private PostFlush(List memtables)
 {
-this.writeBarrier = writeBarrier;
-this.flushSecondaryIndexes = flushSecondaryIndexes;
 this.memtables = memtables;
 }
 
 public ReplayPosition call()
 {
-writeBarrier.await();
-
-/**
- * we can flush 2is as soon as the barrier completes, as they will 
be consistent with (or ahead of) the
- * flushed memtables and CL position, which is as good as we can 
guarantee.
- * TODO: SecondaryIndex should support setBarrier(), so custom 
implementations can co-ordinate exactly
- * with CL as we do with memtables/CFS-backed SecondaryIndexes.
- */
-
-if (flushSecondaryIndexes)
-indexManager.flushAllNonCFSBackedIndexesBlocking();
-
 try
 {
 // we wait on the latch for the commitLogUpperBound to be set, 
and so that waiters
@@ -970,7 +953,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 metric.pendingFlushes.dec();
 
 if (flushFailure != null)
-throw flushFailure;
+Throwables.propagate(flushFailure);
 
 return commitLogUpperBound;
 }
@@ -1029,7 +1012,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 // since this happens after wiring up the commitLogUpperBound, we 
also know all operations with earlier
 // replay positions have also completed, i.e. the memtables are 
done and ready to flush
 writeBarrier.issue();
-postFlush = new PostFlush(!truncate, writeBarrier, memtables);
+postFlush = new PostFlush(memtables);
 }
 
 public void run()
@@ -1047,24 +1030,36 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 try
 {
+boolean flushNonCf2i = true;
 for (Memtable memtable : memtables)
 {
 Collection readers = 
Collections.emptyList();
 if (!memtable.isClean() && !truncate)
+{
+

[02/12] cassandra git commit: Revert "Make sure sstables only get committed when it's safe to discard commit log records"

2016-12-06 Thread blambov

Revert "Make sure sstables only get committed when it's safe to discard commit 
log records"

This reverts commit 6f90e55e7e23cbe814a3232c8d1ec67f2ff2a537 as it was using a 
wrong version of the patch.


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d2ba715f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d2ba715f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d2ba715f

Branch: refs/heads/cassandra-3.X
Commit: d2ba715f2456e1aa821c01941f90b6a58f54e6c4
Parents: 6f90e55
Author: Branimir Lambov 
Authored: Tue Dec 6 14:06:48 2016 +0200
Committer: Branimir Lambov 
Committed: Tue Dec 6 14:06:48 2016 +0200

--
 CHANGES.txt |  1 -
 .../apache/cassandra/db/ColumnFamilyStore.java  | 77 -
 src/java/org/apache/cassandra/db/Memtable.java  | 81 ++
 .../miscellaneous/ColumnFamilyStoreTest.java| 90 
 4 files changed, 63 insertions(+), 186 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d2ba715f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 5242adf..5cacdd0 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,4 @@
 3.0.11
- * Make sure sstables only get committed when it's safe to discard commit log 
records (CASSANDRA-12956)
  * Reject default_time_to_live option when creating or altering MVs 
(CASSANDRA-12868)
  * Nodetool should use a more sane max heap size (CASSANDRA-12739)
  * LocalToken ensures token values are cloned on heap (CASSANDRA-12651)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d2ba715f/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 113e10d..d2a51a9 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -63,7 +63,6 @@ import org.apache.cassandra.io.FSWriteError;
 import org.apache.cassandra.io.sstable.Component;
 import org.apache.cassandra.io.sstable.Descriptor;
 import org.apache.cassandra.io.sstable.SSTableMultiWriter;
-import org.apache.cassandra.io.sstable.SSTableTxnWriter;
 import org.apache.cassandra.io.sstable.format.*;
 import org.apache.cassandra.io.sstable.format.big.BigFormat;
 import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
@@ -82,7 +81,6 @@ import org.json.simple.JSONArray;
 import org.json.simple.JSONObject;
 
 import static org.apache.cassandra.utils.Throwables.maybeFail;
-import static org.apache.cassandra.utils.Throwables.merge;
 
 public class ColumnFamilyStore implements ColumnFamilyStoreMBean
 {
@@ -126,8 +124,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 private static final Logger logger = 
LoggerFactory.getLogger(ColumnFamilyStore.class);
 
-@VisibleForTesting
-public static final ExecutorService flushExecutor = new 
JMXEnabledThreadPoolExecutor(DatabaseDescriptor.getFlushWriters(),
+private static final ExecutorService flushExecutor = new 
JMXEnabledThreadPoolExecutor(DatabaseDescriptor.getFlushWriters(),

   StageManager.KEEPALIVE,

   TimeUnit.SECONDS,

   new LinkedBlockingQueue(),
@@ -924,9 +921,8 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 {
 final boolean flushSecondaryIndexes;
 final OpOrder.Barrier writeBarrier;
-final CountDownLatch memtablesFlushLatch = new CountDownLatch(1);
-final CountDownLatch secondaryIndexFlushLatch = new CountDownLatch(1);
-volatile Throwable flushFailure = null;
+final CountDownLatch latch = new CountDownLatch(1);
+volatile FSWriteError flushFailure = null;
 final List memtables;
 
 private PostFlush(boolean flushSecondaryIndexes, OpOrder.Barrier 
writeBarrier,
@@ -947,27 +943,15 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
  * TODO: SecondaryIndex should support setBarrier(), so custom 
implementations can co-ordinate exactly
  * with CL as we do with memtables/CFS-backed SecondaryIndexes.
  */
-try
-{
-if (flushSecondaryIndexes)
-{
-indexManager.flushAllNonCFSBackedIndexesBlocking();
-}

[11/12] cassandra git commit: Merge branch 'cassandra-3.11' into cassandra-3.X

2016-12-06 Thread blambov

Merge branch 'cassandra-3.11' into cassandra-3.X


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f1423806
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f1423806
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f1423806

Branch: refs/heads/cassandra-3.X
Commit: f1423806e7263cbb7cb357f728b5b5181362d892
Parents: 5439d94 bed3def
Author: Branimir Lambov 
Authored: Tue Dec 6 14:28:27 2016 +0200
Committer: Branimir Lambov 
Committed: Tue Dec 6 14:28:27 2016 +0200

--

--

[10/12] cassandra git commit: Merge branch 'cassandra-3.11' into cassandra-3.X

2016-12-06 Thread blambov

Merge branch 'cassandra-3.11' into cassandra-3.X


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f1423806
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f1423806
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f1423806

Branch: refs/heads/trunk
Commit: f1423806e7263cbb7cb357f728b5b5181362d892
Parents: 5439d94 bed3def
Author: Branimir Lambov 
Authored: Tue Dec 6 14:28:27 2016 +0200
Committer: Branimir Lambov 
Committed: Tue Dec 6 14:28:27 2016 +0200

--

--

[03/12] cassandra git commit: Revert "Make sure sstables only get committed when it's safe to discard commit log records"

2016-12-06 Thread blambov

Revert "Make sure sstables only get committed when it's safe to discard commit 
log records"

This reverts commit 6f90e55e7e23cbe814a3232c8d1ec67f2ff2a537 as it was using a 
wrong version of the patch.


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d2ba715f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d2ba715f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d2ba715f

Branch: refs/heads/trunk
Commit: d2ba715f2456e1aa821c01941f90b6a58f54e6c4
Parents: 6f90e55
Author: Branimir Lambov 
Authored: Tue Dec 6 14:06:48 2016 +0200
Committer: Branimir Lambov 
Committed: Tue Dec 6 14:06:48 2016 +0200

--
 CHANGES.txt |  1 -
 .../apache/cassandra/db/ColumnFamilyStore.java  | 77 -
 src/java/org/apache/cassandra/db/Memtable.java  | 81 ++
 .../miscellaneous/ColumnFamilyStoreTest.java| 90 
 4 files changed, 63 insertions(+), 186 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d2ba715f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 5242adf..5cacdd0 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,4 @@
 3.0.11
- * Make sure sstables only get committed when it's safe to discard commit log 
records (CASSANDRA-12956)
  * Reject default_time_to_live option when creating or altering MVs 
(CASSANDRA-12868)
  * Nodetool should use a more sane max heap size (CASSANDRA-12739)
  * LocalToken ensures token values are cloned on heap (CASSANDRA-12651)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d2ba715f/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 113e10d..d2a51a9 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -63,7 +63,6 @@ import org.apache.cassandra.io.FSWriteError;
 import org.apache.cassandra.io.sstable.Component;
 import org.apache.cassandra.io.sstable.Descriptor;
 import org.apache.cassandra.io.sstable.SSTableMultiWriter;
-import org.apache.cassandra.io.sstable.SSTableTxnWriter;
 import org.apache.cassandra.io.sstable.format.*;
 import org.apache.cassandra.io.sstable.format.big.BigFormat;
 import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
@@ -82,7 +81,6 @@ import org.json.simple.JSONArray;
 import org.json.simple.JSONObject;
 
 import static org.apache.cassandra.utils.Throwables.maybeFail;
-import static org.apache.cassandra.utils.Throwables.merge;
 
 public class ColumnFamilyStore implements ColumnFamilyStoreMBean
 {
@@ -126,8 +124,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 private static final Logger logger = 
LoggerFactory.getLogger(ColumnFamilyStore.class);
 
-@VisibleForTesting
-public static final ExecutorService flushExecutor = new 
JMXEnabledThreadPoolExecutor(DatabaseDescriptor.getFlushWriters(),
+private static final ExecutorService flushExecutor = new 
JMXEnabledThreadPoolExecutor(DatabaseDescriptor.getFlushWriters(),

   StageManager.KEEPALIVE,

   TimeUnit.SECONDS,

   new LinkedBlockingQueue(),
@@ -924,9 +921,8 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 {
 final boolean flushSecondaryIndexes;
 final OpOrder.Barrier writeBarrier;
-final CountDownLatch memtablesFlushLatch = new CountDownLatch(1);
-final CountDownLatch secondaryIndexFlushLatch = new CountDownLatch(1);
-volatile Throwable flushFailure = null;
+final CountDownLatch latch = new CountDownLatch(1);
+volatile FSWriteError flushFailure = null;
 final List memtables;
 
 private PostFlush(boolean flushSecondaryIndexes, OpOrder.Barrier 
writeBarrier,
@@ -947,27 +943,15 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
  * TODO: SecondaryIndex should support setBarrier(), so custom 
implementations can co-ordinate exactly
  * with CL as we do with memtables/CFS-backed SecondaryIndexes.
  */
-try
-{
-if (flushSecondaryIndexes)
-{
-indexManager.flushAllNonCFSBackedIndexesBlocking();
-}
-

[12/12] cassandra git commit: Merge branch 'cassandra-3.X' into trunk

2016-12-06 Thread blambov

Merge branch 'cassandra-3.X' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/48591489
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/48591489
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/48591489

Branch: refs/heads/trunk
Commit: 48591489dd214d5b4df8d1c9e8c5ce1ff1abff93
Parents: 9a7baa1 f142380
Author: Branimir Lambov 
Authored: Tue Dec 6 14:29:32 2016 +0200
Committer: Branimir Lambov 
Committed: Tue Dec 6 14:29:32 2016 +0200

--

--

[09/12] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2016-12-06 Thread blambov

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bed3def9
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bed3def9
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bed3def9

Branch: refs/heads/cassandra-3.11
Commit: bed3def9a0188daad4b3306d5aea28b416be85c2
Parents: 2f268ed 0ecef31
Author: Branimir Lambov 
Authored: Tue Dec 6 14:27:46 2016 +0200
Committer: Branimir Lambov 
Committed: Tue Dec 6 14:27:46 2016 +0200

--

--

[08/12] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2016-12-06 Thread blambov

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bed3def9
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bed3def9
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bed3def9

Branch: refs/heads/trunk
Commit: bed3def9a0188daad4b3306d5aea28b416be85c2
Parents: 2f268ed 0ecef31
Author: Branimir Lambov 
Authored: Tue Dec 6 14:27:46 2016 +0200
Committer: Branimir Lambov 
Committed: Tue Dec 6 14:27:46 2016 +0200

--

--

[07/12] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2016-12-06 Thread blambov

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bed3def9
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bed3def9
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bed3def9

Branch: refs/heads/cassandra-3.X
Commit: bed3def9a0188daad4b3306d5aea28b416be85c2
Parents: 2f268ed 0ecef31
Author: Branimir Lambov 
Authored: Tue Dec 6 14:27:46 2016 +0200
Committer: Branimir Lambov 
Committed: Tue Dec 6 14:27:46 2016 +0200

--

--

[04/12] cassandra git commit: Make sure sstables only get committed when it's safe to discard commit log records

2016-12-06 Thread blambov

Make sure sstables only get committed when it's safe to discard commit log 
records

Patch by Alex Petrov; reviewed by Branimir Lambov for CASSANDRA-12956


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0ecef315
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0ecef315
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0ecef315

Branch: refs/heads/cassandra-3.11
Commit: 0ecef31548c287ac2d9f818413457bc947362733
Parents: d2ba715
Author: Alex Petrov 
Authored: Tue Nov 29 22:58:36 2016 +0100
Committer: Branimir Lambov 
Committed: Tue Dec 6 14:10:00 2016 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 45 +---
 .../apache/cassandra/index/CustomIndexTest.java | 37 
 3 files changed, 58 insertions(+), 25 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/0ecef315/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 5cacdd0..5242adf 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Make sure sstables only get committed when it's safe to discard commit log 
records (CASSANDRA-12956)
  * Reject default_time_to_live option when creating or altering MVs 
(CASSANDRA-12868)
  * Nodetool should use a more sane max heap size (CASSANDRA-12739)
  * LocalToken ensures token values are cloned on heap (CASSANDRA-12651)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0ecef315/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index d2a51a9..71e1653 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -919,34 +919,17 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
  */
 private final class PostFlush implements Callable
 {
-final boolean flushSecondaryIndexes;
-final OpOrder.Barrier writeBarrier;
 final CountDownLatch latch = new CountDownLatch(1);
-volatile FSWriteError flushFailure = null;
+volatile Throwable flushFailure = null;
 final List memtables;
 
-private PostFlush(boolean flushSecondaryIndexes, OpOrder.Barrier 
writeBarrier,
-  List memtables)
+private PostFlush(List memtables)
 {
-this.writeBarrier = writeBarrier;
-this.flushSecondaryIndexes = flushSecondaryIndexes;
 this.memtables = memtables;
 }
 
 public ReplayPosition call()
 {
-writeBarrier.await();
-
-/**
- * we can flush 2is as soon as the barrier completes, as they will 
be consistent with (or ahead of) the
- * flushed memtables and CL position, which is as good as we can 
guarantee.
- * TODO: SecondaryIndex should support setBarrier(), so custom 
implementations can co-ordinate exactly
- * with CL as we do with memtables/CFS-backed SecondaryIndexes.
- */
-
-if (flushSecondaryIndexes)
-indexManager.flushAllNonCFSBackedIndexesBlocking();
-
 try
 {
 // we wait on the latch for the commitLogUpperBound to be set, 
and so that waiters
@@ -970,7 +953,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 metric.pendingFlushes.dec();
 
 if (flushFailure != null)
-throw flushFailure;
+Throwables.propagate(flushFailure);
 
 return commitLogUpperBound;
 }
@@ -1029,7 +1012,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 // since this happens after wiring up the commitLogUpperBound, we 
also know all operations with earlier
 // replay positions have also completed, i.e. the memtables are 
done and ready to flush
 writeBarrier.issue();
-postFlush = new PostFlush(!truncate, writeBarrier, memtables);
+postFlush = new PostFlush(memtables);
 }
 
 public void run()
@@ -1047,24 +1030,36 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 try
 {
+boolean flushNonCf2i = true;
 for (Memtable memtable : memtables)
 {
 Collection readers = 
Collections.emptyList();
 if (!memtable.isClean() && !truncate)
+{
+

[2/2] cassandra git commit: Make sure sstables only get committed when it's safe to discard commit log records

2016-12-06 Thread blambov

Make sure sstables only get committed when it's safe to discard commit log 
records

Patch by Alex Petrov; reviewed by Branimir Lambov for CASSANDRA-12956


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0ecef315
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0ecef315
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0ecef315

Branch: refs/heads/cassandra-3.0
Commit: 0ecef31548c287ac2d9f818413457bc947362733
Parents: d2ba715
Author: Alex Petrov 
Authored: Tue Nov 29 22:58:36 2016 +0100
Committer: Branimir Lambov 
Committed: Tue Dec 6 14:10:00 2016 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 45 +---
 .../apache/cassandra/index/CustomIndexTest.java | 37 
 3 files changed, 58 insertions(+), 25 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/0ecef315/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 5cacdd0..5242adf 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Make sure sstables only get committed when it's safe to discard commit log 
records (CASSANDRA-12956)
  * Reject default_time_to_live option when creating or altering MVs 
(CASSANDRA-12868)
  * Nodetool should use a more sane max heap size (CASSANDRA-12739)
  * LocalToken ensures token values are cloned on heap (CASSANDRA-12651)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0ecef315/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index d2a51a9..71e1653 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -919,34 +919,17 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
  */
 private final class PostFlush implements Callable
 {
-final boolean flushSecondaryIndexes;
-final OpOrder.Barrier writeBarrier;
 final CountDownLatch latch = new CountDownLatch(1);
-volatile FSWriteError flushFailure = null;
+volatile Throwable flushFailure = null;
 final List memtables;
 
-private PostFlush(boolean flushSecondaryIndexes, OpOrder.Barrier 
writeBarrier,
-  List memtables)
+private PostFlush(List memtables)
 {
-this.writeBarrier = writeBarrier;
-this.flushSecondaryIndexes = flushSecondaryIndexes;
 this.memtables = memtables;
 }
 
 public ReplayPosition call()
 {
-writeBarrier.await();
-
-/**
- * we can flush 2is as soon as the barrier completes, as they will 
be consistent with (or ahead of) the
- * flushed memtables and CL position, which is as good as we can 
guarantee.
- * TODO: SecondaryIndex should support setBarrier(), so custom 
implementations can co-ordinate exactly
- * with CL as we do with memtables/CFS-backed SecondaryIndexes.
- */
-
-if (flushSecondaryIndexes)
-indexManager.flushAllNonCFSBackedIndexesBlocking();
-
 try
 {
 // we wait on the latch for the commitLogUpperBound to be set, 
and so that waiters
@@ -970,7 +953,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 metric.pendingFlushes.dec();
 
 if (flushFailure != null)
-throw flushFailure;
+Throwables.propagate(flushFailure);
 
 return commitLogUpperBound;
 }
@@ -1029,7 +1012,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 // since this happens after wiring up the commitLogUpperBound, we 
also know all operations with earlier
 // replay positions have also completed, i.e. the memtables are 
done and ready to flush
 writeBarrier.issue();
-postFlush = new PostFlush(!truncate, writeBarrier, memtables);
+postFlush = new PostFlush(memtables);
 }
 
 public void run()
@@ -1047,24 +1030,36 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 try
 {
+boolean flushNonCf2i = true;
 for (Memtable memtable : memtables)
 {
 Collection readers = 
Collections.emptyList();
 if (!memtable.isClean() && !truncate)
+{
+

[1/2] cassandra git commit: Revert "Make sure sstables only get committed when it's safe to discard commit log records"

2016-12-06 Thread blambov

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 6f90e55e7 -> 0ecef3154


Revert "Make sure sstables only get committed when it's safe to discard commit 
log records"

This reverts commit 6f90e55e7e23cbe814a3232c8d1ec67f2ff2a537 as it was using a 
wrong version of the patch.


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d2ba715f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d2ba715f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d2ba715f

Branch: refs/heads/cassandra-3.0
Commit: d2ba715f2456e1aa821c01941f90b6a58f54e6c4
Parents: 6f90e55
Author: Branimir Lambov 
Authored: Tue Dec 6 14:06:48 2016 +0200
Committer: Branimir Lambov 
Committed: Tue Dec 6 14:06:48 2016 +0200

--
 CHANGES.txt |  1 -
 .../apache/cassandra/db/ColumnFamilyStore.java  | 77 -
 src/java/org/apache/cassandra/db/Memtable.java  | 81 ++
 .../miscellaneous/ColumnFamilyStoreTest.java| 90 
 4 files changed, 63 insertions(+), 186 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d2ba715f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 5242adf..5cacdd0 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,4 @@
 3.0.11
- * Make sure sstables only get committed when it's safe to discard commit log 
records (CASSANDRA-12956)
  * Reject default_time_to_live option when creating or altering MVs 
(CASSANDRA-12868)
  * Nodetool should use a more sane max heap size (CASSANDRA-12739)
  * LocalToken ensures token values are cloned on heap (CASSANDRA-12651)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d2ba715f/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 113e10d..d2a51a9 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -63,7 +63,6 @@ import org.apache.cassandra.io.FSWriteError;
 import org.apache.cassandra.io.sstable.Component;
 import org.apache.cassandra.io.sstable.Descriptor;
 import org.apache.cassandra.io.sstable.SSTableMultiWriter;
-import org.apache.cassandra.io.sstable.SSTableTxnWriter;
 import org.apache.cassandra.io.sstable.format.*;
 import org.apache.cassandra.io.sstable.format.big.BigFormat;
 import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
@@ -82,7 +81,6 @@ import org.json.simple.JSONArray;
 import org.json.simple.JSONObject;
 
 import static org.apache.cassandra.utils.Throwables.maybeFail;
-import static org.apache.cassandra.utils.Throwables.merge;
 
 public class ColumnFamilyStore implements ColumnFamilyStoreMBean
 {
@@ -126,8 +124,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 private static final Logger logger = 
LoggerFactory.getLogger(ColumnFamilyStore.class);
 
-@VisibleForTesting
-public static final ExecutorService flushExecutor = new 
JMXEnabledThreadPoolExecutor(DatabaseDescriptor.getFlushWriters(),
+private static final ExecutorService flushExecutor = new 
JMXEnabledThreadPoolExecutor(DatabaseDescriptor.getFlushWriters(),

   StageManager.KEEPALIVE,

   TimeUnit.SECONDS,

   new LinkedBlockingQueue(),
@@ -924,9 +921,8 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 {
 final boolean flushSecondaryIndexes;
 final OpOrder.Barrier writeBarrier;
-final CountDownLatch memtablesFlushLatch = new CountDownLatch(1);
-final CountDownLatch secondaryIndexFlushLatch = new CountDownLatch(1);
-volatile Throwable flushFailure = null;
+final CountDownLatch latch = new CountDownLatch(1);
+volatile FSWriteError flushFailure = null;
 final List memtables;
 
 private PostFlush(boolean flushSecondaryIndexes, OpOrder.Barrier 
writeBarrier,
@@ -947,27 +943,15 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
  * TODO: SecondaryIndex should support setBarrier(), so custom 
implementations can co-ordinate exactly
  * with CL as we do with memtables/CFS-backed SecondaryIndexes.
  */
-try
-{
-if (flushSecondaryIndexes)
-{

[04/10] cassandra git commit: Make sure sstables only get committed when it's safe to discard commit log records

2016-12-06 Thread blambov

Make sure sstables only get committed when it's safe to discard commit log 
records

Patch by Alex Petrov; reviewed by Branimir Lambov for CASSANDRA-12956


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6f90e55e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6f90e55e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6f90e55e

Branch: refs/heads/trunk
Commit: 6f90e55e7e23cbe814a3232c8d1ec67f2ff2a537
Parents: 5f64ed7
Author: Alex Petrov 
Authored: Tue Nov 29 22:58:36 2016 +0100
Committer: Branimir Lambov 
Committed: Tue Dec 6 12:10:31 2016 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 77 +
 src/java/org/apache/cassandra/db/Memtable.java  | 81 --
 .../miscellaneous/ColumnFamilyStoreTest.java| 90 
 4 files changed, 186 insertions(+), 63 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/6f90e55e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 5cacdd0..5242adf 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Make sure sstables only get committed when it's safe to discard commit log 
records (CASSANDRA-12956)
  * Reject default_time_to_live option when creating or altering MVs 
(CASSANDRA-12868)
  * Nodetool should use a more sane max heap size (CASSANDRA-12739)
  * LocalToken ensures token values are cloned on heap (CASSANDRA-12651)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/6f90e55e/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index d2a51a9..113e10d 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -63,6 +63,7 @@ import org.apache.cassandra.io.FSWriteError;
 import org.apache.cassandra.io.sstable.Component;
 import org.apache.cassandra.io.sstable.Descriptor;
 import org.apache.cassandra.io.sstable.SSTableMultiWriter;
+import org.apache.cassandra.io.sstable.SSTableTxnWriter;
 import org.apache.cassandra.io.sstable.format.*;
 import org.apache.cassandra.io.sstable.format.big.BigFormat;
 import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
@@ -81,6 +82,7 @@ import org.json.simple.JSONArray;
 import org.json.simple.JSONObject;
 
 import static org.apache.cassandra.utils.Throwables.maybeFail;
+import static org.apache.cassandra.utils.Throwables.merge;
 
 public class ColumnFamilyStore implements ColumnFamilyStoreMBean
 {
@@ -124,7 +126,8 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 private static final Logger logger = 
LoggerFactory.getLogger(ColumnFamilyStore.class);
 
-private static final ExecutorService flushExecutor = new 
JMXEnabledThreadPoolExecutor(DatabaseDescriptor.getFlushWriters(),
+@VisibleForTesting
+public static final ExecutorService flushExecutor = new 
JMXEnabledThreadPoolExecutor(DatabaseDescriptor.getFlushWriters(),

   StageManager.KEEPALIVE,

   TimeUnit.SECONDS,

   new LinkedBlockingQueue(),
@@ -921,8 +924,9 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 {
 final boolean flushSecondaryIndexes;
 final OpOrder.Barrier writeBarrier;
-final CountDownLatch latch = new CountDownLatch(1);
-volatile FSWriteError flushFailure = null;
+final CountDownLatch memtablesFlushLatch = new CountDownLatch(1);
+final CountDownLatch secondaryIndexFlushLatch = new CountDownLatch(1);
+volatile Throwable flushFailure = null;
 final List memtables;
 
 private PostFlush(boolean flushSecondaryIndexes, OpOrder.Barrier 
writeBarrier,
@@ -943,15 +947,27 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
  * TODO: SecondaryIndex should support setBarrier(), so custom 
implementations can co-ordinate exactly
  * with CL as we do with memtables/CFS-backed SecondaryIndexes.
  */
-
-if (flushSecondaryIndexes)
-indexManager.flushAllNonCFSBackedIndexesBlocking();
+try
+{
+if (flushSecondaryIndexes)
+{
+

[03/10] cassandra git commit: Make sure sstables only get committed when it's safe to discard commit log records

2016-12-06 Thread blambov

Make sure sstables only get committed when it's safe to discard commit log 
records

Patch by Alex Petrov; reviewed by Branimir Lambov for CASSANDRA-12956


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6f90e55e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6f90e55e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6f90e55e

Branch: refs/heads/cassandra-3.X
Commit: 6f90e55e7e23cbe814a3232c8d1ec67f2ff2a537
Parents: 5f64ed7
Author: Alex Petrov 
Authored: Tue Nov 29 22:58:36 2016 +0100
Committer: Branimir Lambov 
Committed: Tue Dec 6 12:10:31 2016 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 77 +
 src/java/org/apache/cassandra/db/Memtable.java  | 81 --
 .../miscellaneous/ColumnFamilyStoreTest.java| 90 
 4 files changed, 186 insertions(+), 63 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/6f90e55e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 5cacdd0..5242adf 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Make sure sstables only get committed when it's safe to discard commit log 
records (CASSANDRA-12956)
  * Reject default_time_to_live option when creating or altering MVs 
(CASSANDRA-12868)
  * Nodetool should use a more sane max heap size (CASSANDRA-12739)
  * LocalToken ensures token values are cloned on heap (CASSANDRA-12651)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/6f90e55e/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index d2a51a9..113e10d 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -63,6 +63,7 @@ import org.apache.cassandra.io.FSWriteError;
 import org.apache.cassandra.io.sstable.Component;
 import org.apache.cassandra.io.sstable.Descriptor;
 import org.apache.cassandra.io.sstable.SSTableMultiWriter;
+import org.apache.cassandra.io.sstable.SSTableTxnWriter;
 import org.apache.cassandra.io.sstable.format.*;
 import org.apache.cassandra.io.sstable.format.big.BigFormat;
 import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
@@ -81,6 +82,7 @@ import org.json.simple.JSONArray;
 import org.json.simple.JSONObject;
 
 import static org.apache.cassandra.utils.Throwables.maybeFail;
+import static org.apache.cassandra.utils.Throwables.merge;
 
 public class ColumnFamilyStore implements ColumnFamilyStoreMBean
 {
@@ -124,7 +126,8 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 private static final Logger logger = 
LoggerFactory.getLogger(ColumnFamilyStore.class);
 
-private static final ExecutorService flushExecutor = new 
JMXEnabledThreadPoolExecutor(DatabaseDescriptor.getFlushWriters(),
+@VisibleForTesting
+public static final ExecutorService flushExecutor = new 
JMXEnabledThreadPoolExecutor(DatabaseDescriptor.getFlushWriters(),

   StageManager.KEEPALIVE,

   TimeUnit.SECONDS,

   new LinkedBlockingQueue(),
@@ -921,8 +924,9 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 {
 final boolean flushSecondaryIndexes;
 final OpOrder.Barrier writeBarrier;
-final CountDownLatch latch = new CountDownLatch(1);
-volatile FSWriteError flushFailure = null;
+final CountDownLatch memtablesFlushLatch = new CountDownLatch(1);
+final CountDownLatch secondaryIndexFlushLatch = new CountDownLatch(1);
+volatile Throwable flushFailure = null;
 final List memtables;
 
 private PostFlush(boolean flushSecondaryIndexes, OpOrder.Barrier 
writeBarrier,
@@ -943,15 +947,27 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
  * TODO: SecondaryIndex should support setBarrier(), so custom 
implementations can co-ordinate exactly
  * with CL as we do with memtables/CFS-backed SecondaryIndexes.
  */
-
-if (flushSecondaryIndexes)
-indexManager.flushAllNonCFSBackedIndexesBlocking();
+try
+{
+if (flushSecondaryIndexes)
+{
+

[06/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2016-12-06 Thread blambov

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2f268eda
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2f268eda
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2f268eda

Branch: refs/heads/cassandra-3.11
Commit: 2f268eda3d44f8b14b71b7f4b3f4c25e2dfb2c11
Parents: b207f2e 6f90e55
Author: Branimir Lambov 
Authored: Tue Dec 6 12:11:15 2016 +0200
Committer: Branimir Lambov 
Committed: Tue Dec 6 12:12:19 2016 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 41 
 .../apache/cassandra/index/CustomIndexTest.java | 37 ++
 3 files changed, 55 insertions(+), 24 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/2f268eda/CHANGES.txt
--
diff --cc CHANGES.txt
index c5d2da2,5242adf..6da6b4f
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,113 -1,5 +1,114 @@@
 -3.0.11
 +3.10
 + * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
 + * Remove timing window in test case (CASSANDRA-12875)
 + * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
 + * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)
 + * Fix validation of non-frozen UDT cells (CASSANDRA-12916)
 + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903)
 + * Fix Murmur3PartitionerTest (CASSANDRA-12858)
 + * Move cqlsh syntax rules into separate module and allow easier 
customization (CASSANDRA-12897)
 + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283)
 + * Fix cassandra-stress truncate option (CASSANDRA-12695)
 + * Fix crossNode value when receiving messages (CASSANDRA-12791)
 + * Don't load MX4J beans twice (CASSANDRA-12869)
 + * Extend native protocol request flags, add versions to SUPPORTED, and 
introduce ProtocolVersion enum (CASSANDRA-12838)
 + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836)
 + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845)
 + * Properly format IPv6 addresses when logging JMX service URL 
(CASSANDRA-12454)
 + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777)
 + * Use non-token restrictions for bounds when token restrictions are 
overridden (CASSANDRA-12419)
 + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803)
 + * Use different build directories for Eclipse and Ant (CASSANDRA-12466)
 + * Avoid potential AttributeError in cqlsh due to no table metadata 
(CASSANDRA-12815)
 + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster 
(CASSANDRA-12812)
 + * Upgrade commons-codec to 1.9 (CASSANDRA-12790)
 + * Make the fanout size for LeveledCompactionStrategy to be configurable 
(CASSANDRA-11550)
 + * Add duration data type (CASSANDRA-11873)
 + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784)
 + * Improve sum aggregate functions (CASSANDRA-12417)
 + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes 
in CASSANDRA-10876 (CASSANDRA-12761)
 + * cqlsh fails to format collections when using aliases (CASSANDRA-11534)
 + * Check for hash conflicts in prepared statements (CASSANDRA-12733)
 + * Exit query parsing upon first error (CASSANDRA-12598)
 + * Fix cassandra-stress to use single seed in UUID generation 
(CASSANDRA-12729)
 + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450)
 + * Config class uses boxed types but DD exposes primitive types 
(CASSANDRA-12199)
 + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461)
 + * Add hint delivery metrics (CASSANDRA-12693)
 + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731)
 + * ColumnIndex does not reuse buffer (CASSANDRA-12502)
 + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697)
 + * Upgrade metrics-reporter dependencies (CASSANDRA-12089)
 + * Tune compaction thread count via nodetool (CASSANDRA-12248)
 + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232)
 + * Include repair session IDs in repair start message (CASSANDRA-12532)
 + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039)
 + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667)
 + * Support optional backpressure strategies at the coordinator 
(CASSANDRA-9318)
 + * Make randompartitioner work with new vnode allocation (CASSANDRA-12647)
 + * Fix cassandra-stress graphing (CASSANDRA-12237)
 + * Allow filtering on partition key columns for queries without secondary 
indexes (CASSANDRA-11031)
 + * Fix Cassandra Stress reporting thread model and

[08/10] cassandra git commit: Merge branch 'cassandra-3.11' into cassandra-3.X

2016-12-06 Thread blambov

Merge branch 'cassandra-3.11' into cassandra-3.X


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5439d94c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5439d94c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5439d94c

Branch: refs/heads/cassandra-3.X
Commit: 5439d94c546331b30acf0d43a503e9426364e81a
Parents: 838a21d 2f268ed
Author: Branimir Lambov 
Authored: Tue Dec 6 12:13:23 2016 +0200
Committer: Branimir Lambov 
Committed: Tue Dec 6 12:14:24 2016 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 41 
 .../apache/cassandra/index/CustomIndexTest.java | 37 ++
 3 files changed, 55 insertions(+), 24 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5439d94c/CHANGES.txt
--

[10/10] cassandra git commit: Merge branch 'cassandra-3.X' into trunk

2016-12-06 Thread blambov

Merge branch 'cassandra-3.X' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9a7baa14
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9a7baa14
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9a7baa14

Branch: refs/heads/trunk
Commit: 9a7baa145398aa0b1970d70ca508f4a0a6e8e01c
Parents: 8ddbb74 5439d94
Author: Branimir Lambov 
Authored: Tue Dec 6 12:17:35 2016 +0200
Committer: Branimir Lambov 
Committed: Tue Dec 6 12:18:09 2016 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 41 
 .../apache/cassandra/index/CustomIndexTest.java | 37 ++
 3 files changed, 55 insertions(+), 24 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9a7baa14/CHANGES.txt
--
diff --cc CHANGES.txt
index 90003ec,bddd823..40407bc
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -122,8 -113,9 +122,9 @@@
   * Remove pre-startup check for open JMX port (CASSANDRA-12074)
   * Remove compaction Severity from DynamicEndpointSnitch (CASSANDRA-11738)
   * Restore resumable hints delivery (CASSANDRA-11960)
 - * Properly report LWT contention (CASSANDRA-12626)
 + * Properly record CAS contention (CASSANDRA-12626)
  Merged from 3.0:
+  * Make sure sstables only get committed when it's safe to discard commit log 
records (CASSANDRA-12956)
   * Reject default_time_to_live option when creating or altering MVs 
(CASSANDRA-12868)
   * Nodetool should use a more sane max heap size (CASSANDRA-12739)
   * LocalToken ensures token values are cloned on heap (CASSANDRA-12651)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9a7baa14/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

[01/10] cassandra git commit: Make sure sstables only get committed when it's safe to discard commit log records

2016-12-06 Thread blambov

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 5f64ed7cc -> 6f90e55e7
  refs/heads/cassandra-3.11 b207f2e3b -> 2f268eda3
  refs/heads/cassandra-3.X 838a21d40 -> 5439d94c5
  refs/heads/trunk 8ddbb7493 -> 9a7baa145


Make sure sstables only get committed when it's safe to discard commit log 
records

Patch by Alex Petrov; reviewed by Branimir Lambov for CASSANDRA-12956


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6f90e55e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6f90e55e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6f90e55e

Branch: refs/heads/cassandra-3.0
Commit: 6f90e55e7e23cbe814a3232c8d1ec67f2ff2a537
Parents: 5f64ed7
Author: Alex Petrov 
Authored: Tue Nov 29 22:58:36 2016 +0100
Committer: Branimir Lambov 
Committed: Tue Dec 6 12:10:31 2016 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 77 +
 src/java/org/apache/cassandra/db/Memtable.java  | 81 --
 .../miscellaneous/ColumnFamilyStoreTest.java| 90 
 4 files changed, 186 insertions(+), 63 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/6f90e55e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 5cacdd0..5242adf 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Make sure sstables only get committed when it's safe to discard commit log 
records (CASSANDRA-12956)
  * Reject default_time_to_live option when creating or altering MVs 
(CASSANDRA-12868)
  * Nodetool should use a more sane max heap size (CASSANDRA-12739)
  * LocalToken ensures token values are cloned on heap (CASSANDRA-12651)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/6f90e55e/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index d2a51a9..113e10d 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -63,6 +63,7 @@ import org.apache.cassandra.io.FSWriteError;
 import org.apache.cassandra.io.sstable.Component;
 import org.apache.cassandra.io.sstable.Descriptor;
 import org.apache.cassandra.io.sstable.SSTableMultiWriter;
+import org.apache.cassandra.io.sstable.SSTableTxnWriter;
 import org.apache.cassandra.io.sstable.format.*;
 import org.apache.cassandra.io.sstable.format.big.BigFormat;
 import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
@@ -81,6 +82,7 @@ import org.json.simple.JSONArray;
 import org.json.simple.JSONObject;
 
 import static org.apache.cassandra.utils.Throwables.maybeFail;
+import static org.apache.cassandra.utils.Throwables.merge;
 
 public class ColumnFamilyStore implements ColumnFamilyStoreMBean
 {
@@ -124,7 +126,8 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 private static final Logger logger = 
LoggerFactory.getLogger(ColumnFamilyStore.class);
 
-private static final ExecutorService flushExecutor = new 
JMXEnabledThreadPoolExecutor(DatabaseDescriptor.getFlushWriters(),
+@VisibleForTesting
+public static final ExecutorService flushExecutor = new 
JMXEnabledThreadPoolExecutor(DatabaseDescriptor.getFlushWriters(),

   StageManager.KEEPALIVE,

   TimeUnit.SECONDS,

   new LinkedBlockingQueue(),
@@ -921,8 +924,9 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 {
 final boolean flushSecondaryIndexes;
 final OpOrder.Barrier writeBarrier;
-final CountDownLatch latch = new CountDownLatch(1);
-volatile FSWriteError flushFailure = null;
+final CountDownLatch memtablesFlushLatch = new CountDownLatch(1);
+final CountDownLatch secondaryIndexFlushLatch = new CountDownLatch(1);
+volatile Throwable flushFailure = null;
 final List memtables;
 
 private PostFlush(boolean flushSecondaryIndexes, OpOrder.Barrier 
writeBarrier,
@@ -943,15 +947,27 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
  * TODO: SecondaryIndex should support setBarrier(), so custom 
implementations can co-ordinate exactly
  * with CL as we do with memtables/CFS-backed SecondaryIndexes.
  */
-
-

[02/10] cassandra git commit: Make sure sstables only get committed when it's safe to discard commit log records

2016-12-06 Thread blambov

Make sure sstables only get committed when it's safe to discard commit log 
records

Patch by Alex Petrov; reviewed by Branimir Lambov for CASSANDRA-12956


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6f90e55e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6f90e55e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6f90e55e

Branch: refs/heads/cassandra-3.11
Commit: 6f90e55e7e23cbe814a3232c8d1ec67f2ff2a537
Parents: 5f64ed7
Author: Alex Petrov 
Authored: Tue Nov 29 22:58:36 2016 +0100
Committer: Branimir Lambov 
Committed: Tue Dec 6 12:10:31 2016 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 77 +
 src/java/org/apache/cassandra/db/Memtable.java  | 81 --
 .../miscellaneous/ColumnFamilyStoreTest.java| 90 
 4 files changed, 186 insertions(+), 63 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/6f90e55e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 5cacdd0..5242adf 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Make sure sstables only get committed when it's safe to discard commit log 
records (CASSANDRA-12956)
  * Reject default_time_to_live option when creating or altering MVs 
(CASSANDRA-12868)
  * Nodetool should use a more sane max heap size (CASSANDRA-12739)
  * LocalToken ensures token values are cloned on heap (CASSANDRA-12651)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/6f90e55e/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index d2a51a9..113e10d 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -63,6 +63,7 @@ import org.apache.cassandra.io.FSWriteError;
 import org.apache.cassandra.io.sstable.Component;
 import org.apache.cassandra.io.sstable.Descriptor;
 import org.apache.cassandra.io.sstable.SSTableMultiWriter;
+import org.apache.cassandra.io.sstable.SSTableTxnWriter;
 import org.apache.cassandra.io.sstable.format.*;
 import org.apache.cassandra.io.sstable.format.big.BigFormat;
 import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
@@ -81,6 +82,7 @@ import org.json.simple.JSONArray;
 import org.json.simple.JSONObject;
 
 import static org.apache.cassandra.utils.Throwables.maybeFail;
+import static org.apache.cassandra.utils.Throwables.merge;
 
 public class ColumnFamilyStore implements ColumnFamilyStoreMBean
 {
@@ -124,7 +126,8 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 private static final Logger logger = 
LoggerFactory.getLogger(ColumnFamilyStore.class);
 
-private static final ExecutorService flushExecutor = new 
JMXEnabledThreadPoolExecutor(DatabaseDescriptor.getFlushWriters(),
+@VisibleForTesting
+public static final ExecutorService flushExecutor = new 
JMXEnabledThreadPoolExecutor(DatabaseDescriptor.getFlushWriters(),

   StageManager.KEEPALIVE,

   TimeUnit.SECONDS,

   new LinkedBlockingQueue(),
@@ -921,8 +924,9 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 {
 final boolean flushSecondaryIndexes;
 final OpOrder.Barrier writeBarrier;
-final CountDownLatch latch = new CountDownLatch(1);
-volatile FSWriteError flushFailure = null;
+final CountDownLatch memtablesFlushLatch = new CountDownLatch(1);
+final CountDownLatch secondaryIndexFlushLatch = new CountDownLatch(1);
+volatile Throwable flushFailure = null;
 final List memtables;
 
 private PostFlush(boolean flushSecondaryIndexes, OpOrder.Barrier 
writeBarrier,
@@ -943,15 +947,27 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
  * TODO: SecondaryIndex should support setBarrier(), so custom 
implementations can co-ordinate exactly
  * with CL as we do with memtables/CFS-backed SecondaryIndexes.
  */
-
-if (flushSecondaryIndexes)
-indexManager.flushAllNonCFSBackedIndexesBlocking();
+try
+{
+if (flushSecondaryIndexes)
+{
+

[09/10] cassandra git commit: Merge branch 'cassandra-3.11' into cassandra-3.X

2016-12-06 Thread blambov

Merge branch 'cassandra-3.11' into cassandra-3.X


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5439d94c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5439d94c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5439d94c

Branch: refs/heads/trunk
Commit: 5439d94c546331b30acf0d43a503e9426364e81a
Parents: 838a21d 2f268ed
Author: Branimir Lambov 
Authored: Tue Dec 6 12:13:23 2016 +0200
Committer: Branimir Lambov 
Committed: Tue Dec 6 12:14:24 2016 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 41 
 .../apache/cassandra/index/CustomIndexTest.java | 37 ++
 3 files changed, 55 insertions(+), 24 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5439d94c/CHANGES.txt
--

[05/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2016-12-06 Thread blambov

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2f268eda
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2f268eda
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2f268eda

Branch: refs/heads/cassandra-3.X
Commit: 2f268eda3d44f8b14b71b7f4b3f4c25e2dfb2c11
Parents: b207f2e 6f90e55
Author: Branimir Lambov 
Authored: Tue Dec 6 12:11:15 2016 +0200
Committer: Branimir Lambov 
Committed: Tue Dec 6 12:12:19 2016 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 41 
 .../apache/cassandra/index/CustomIndexTest.java | 37 ++
 3 files changed, 55 insertions(+), 24 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/2f268eda/CHANGES.txt
--
diff --cc CHANGES.txt
index c5d2da2,5242adf..6da6b4f
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,113 -1,5 +1,114 @@@
 -3.0.11
 +3.10
 + * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
 + * Remove timing window in test case (CASSANDRA-12875)
 + * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
 + * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)
 + * Fix validation of non-frozen UDT cells (CASSANDRA-12916)
 + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903)
 + * Fix Murmur3PartitionerTest (CASSANDRA-12858)
 + * Move cqlsh syntax rules into separate module and allow easier 
customization (CASSANDRA-12897)
 + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283)
 + * Fix cassandra-stress truncate option (CASSANDRA-12695)
 + * Fix crossNode value when receiving messages (CASSANDRA-12791)
 + * Don't load MX4J beans twice (CASSANDRA-12869)
 + * Extend native protocol request flags, add versions to SUPPORTED, and 
introduce ProtocolVersion enum (CASSANDRA-12838)
 + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836)
 + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845)
 + * Properly format IPv6 addresses when logging JMX service URL 
(CASSANDRA-12454)
 + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777)
 + * Use non-token restrictions for bounds when token restrictions are 
overridden (CASSANDRA-12419)
 + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803)
 + * Use different build directories for Eclipse and Ant (CASSANDRA-12466)
 + * Avoid potential AttributeError in cqlsh due to no table metadata 
(CASSANDRA-12815)
 + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster 
(CASSANDRA-12812)
 + * Upgrade commons-codec to 1.9 (CASSANDRA-12790)
 + * Make the fanout size for LeveledCompactionStrategy to be configurable 
(CASSANDRA-11550)
 + * Add duration data type (CASSANDRA-11873)
 + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784)
 + * Improve sum aggregate functions (CASSANDRA-12417)
 + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes 
in CASSANDRA-10876 (CASSANDRA-12761)
 + * cqlsh fails to format collections when using aliases (CASSANDRA-11534)
 + * Check for hash conflicts in prepared statements (CASSANDRA-12733)
 + * Exit query parsing upon first error (CASSANDRA-12598)
 + * Fix cassandra-stress to use single seed in UUID generation 
(CASSANDRA-12729)
 + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450)
 + * Config class uses boxed types but DD exposes primitive types 
(CASSANDRA-12199)
 + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461)
 + * Add hint delivery metrics (CASSANDRA-12693)
 + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731)
 + * ColumnIndex does not reuse buffer (CASSANDRA-12502)
 + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697)
 + * Upgrade metrics-reporter dependencies (CASSANDRA-12089)
 + * Tune compaction thread count via nodetool (CASSANDRA-12248)
 + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232)
 + * Include repair session IDs in repair start message (CASSANDRA-12532)
 + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039)
 + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667)
 + * Support optional backpressure strategies at the coordinator 
(CASSANDRA-9318)
 + * Make randompartitioner work with new vnode allocation (CASSANDRA-12647)
 + * Fix cassandra-stress graphing (CASSANDRA-12237)
 + * Allow filtering on partition key columns for queries without secondary 
indexes (CASSANDRA-11031)
 + * Fix Cassandra Stress reporting thread model and precision

[07/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2016-12-06 Thread blambov

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2f268eda
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2f268eda
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2f268eda

Branch: refs/heads/trunk
Commit: 2f268eda3d44f8b14b71b7f4b3f4c25e2dfb2c11
Parents: b207f2e 6f90e55
Author: Branimir Lambov 
Authored: Tue Dec 6 12:11:15 2016 +0200
Committer: Branimir Lambov 
Committed: Tue Dec 6 12:12:19 2016 +0200

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  | 41 
 .../apache/cassandra/index/CustomIndexTest.java | 37 ++
 3 files changed, 55 insertions(+), 24 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/2f268eda/CHANGES.txt
--
diff --cc CHANGES.txt
index c5d2da2,5242adf..6da6b4f
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,113 -1,5 +1,114 @@@
 -3.0.11
 +3.10
 + * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
 + * Remove timing window in test case (CASSANDRA-12875)
 + * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
 + * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)
 + * Fix validation of non-frozen UDT cells (CASSANDRA-12916)
 + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903)
 + * Fix Murmur3PartitionerTest (CASSANDRA-12858)
 + * Move cqlsh syntax rules into separate module and allow easier 
customization (CASSANDRA-12897)
 + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283)
 + * Fix cassandra-stress truncate option (CASSANDRA-12695)
 + * Fix crossNode value when receiving messages (CASSANDRA-12791)
 + * Don't load MX4J beans twice (CASSANDRA-12869)
 + * Extend native protocol request flags, add versions to SUPPORTED, and 
introduce ProtocolVersion enum (CASSANDRA-12838)
 + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836)
 + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845)
 + * Properly format IPv6 addresses when logging JMX service URL 
(CASSANDRA-12454)
 + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777)
 + * Use non-token restrictions for bounds when token restrictions are 
overridden (CASSANDRA-12419)
 + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803)
 + * Use different build directories for Eclipse and Ant (CASSANDRA-12466)
 + * Avoid potential AttributeError in cqlsh due to no table metadata 
(CASSANDRA-12815)
 + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster 
(CASSANDRA-12812)
 + * Upgrade commons-codec to 1.9 (CASSANDRA-12790)
 + * Make the fanout size for LeveledCompactionStrategy to be configurable 
(CASSANDRA-11550)
 + * Add duration data type (CASSANDRA-11873)
 + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784)
 + * Improve sum aggregate functions (CASSANDRA-12417)
 + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes 
in CASSANDRA-10876 (CASSANDRA-12761)
 + * cqlsh fails to format collections when using aliases (CASSANDRA-11534)
 + * Check for hash conflicts in prepared statements (CASSANDRA-12733)
 + * Exit query parsing upon first error (CASSANDRA-12598)
 + * Fix cassandra-stress to use single seed in UUID generation 
(CASSANDRA-12729)
 + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450)
 + * Config class uses boxed types but DD exposes primitive types 
(CASSANDRA-12199)
 + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461)
 + * Add hint delivery metrics (CASSANDRA-12693)
 + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731)
 + * ColumnIndex does not reuse buffer (CASSANDRA-12502)
 + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697)
 + * Upgrade metrics-reporter dependencies (CASSANDRA-12089)
 + * Tune compaction thread count via nodetool (CASSANDRA-12248)
 + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232)
 + * Include repair session IDs in repair start message (CASSANDRA-12532)
 + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039)
 + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667)
 + * Support optional backpressure strategies at the coordinator 
(CASSANDRA-9318)
 + * Make randompartitioner work with new vnode allocation (CASSANDRA-12647)
 + * Fix cassandra-stress graphing (CASSANDRA-12237)
 + * Allow filtering on partition key columns for queries without secondary 
indexes (CASSANDRA-11031)
 + * Fix Cassandra Stress reporting thread model and precision

[3/6] cassandra git commit: Use correct bounds for all-data range when filtering

2016-11-30 Thread blambov

Use correct bounds for all-data range when filtering

Patch by Alex Petrov; reviewed by Branimir Lambov for CASSANDRA-12666.


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f00e4316
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f00e4316
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f00e4316

Branch: refs/heads/trunk
Commit: f00e43167ab11f58af20439a300bdf82664abdb0
Parents: 8de24ca
Author: Alex Petrov 
Authored: Sun Sep 18 11:09:47 2016 +0200
Committer: Branimir Lambov 
Committed: Wed Nov 30 14:39:33 2016 +0200

--
 CHANGES.txt   |  1 +
 .../cql3/restrictions/StatementRestrictions.java  | 10 +-
 2 files changed, 6 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f00e4316/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 72d6a1f..b238018 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.10
+ * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
  * Remove timing window in test case (CASSANDRA-12875)
  * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
  * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f00e4316/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
--
diff --git 
a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java 
b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
index 53ac68c..2d04633 100644
--- a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
+++ b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
@@ -624,11 +624,6 @@ public final class StatementRestrictions
  */
 private ByteBuffer getPartitionKeyBound(Bound b, QueryOptions options)
 {
-// Deal with unrestricted partition key components (special-casing is 
required to deal with 2i queries on the
-// first component of a composite partition key) queries that filter 
on the partition key.
-if (partitionKeyRestrictions.needFiltering(cfm))
-return ByteBufferUtil.EMPTY_BYTE_BUFFER;
-
 // We deal with IN queries for keys in other places, so we know 
buildBound will return only one result
 return partitionKeyRestrictions.bounds(b, options).get(0);
 }
@@ -654,6 +649,11 @@ public final class StatementRestrictions
 private AbstractBounds 
getPartitionKeyBounds(IPartitioner p,
 
QueryOptions options)
 {
+// Deal with unrestricted partition key components (special-casing is 
required to deal with 2i queries on the
+// first component of a composite partition key) queries that filter 
on the partition key.
+if (partitionKeyRestrictions.needFiltering(cfm))
+return new Range<>(p.getMinimumToken().minKeyBound(), 
p.getMinimumToken().maxKeyBound());
+
 ByteBuffer startKeyBytes = getPartitionKeyBound(Bound.START, options);
 ByteBuffer finishKeyBytes = getPartitionKeyBound(Bound.END, options);

[6/6] cassandra git commit: Merge branch 'cassandra-3.X' into trunk

2016-11-30 Thread blambov

Merge branch 'cassandra-3.X' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cba3c014
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cba3c014
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cba3c014

Branch: refs/heads/trunk
Commit: cba3c0141acbe5f327622c87253c0e8316918867
Parents: bcb6762 2ff97fe
Author: Branimir Lambov 
Authored: Wed Nov 30 14:48:24 2016 +0200
Committer: Branimir Lambov 
Committed: Wed Nov 30 14:49:11 2016 +0200

--
 CHANGES.txt   |  1 +
 .../cql3/restrictions/StatementRestrictions.java  | 10 +-
 2 files changed, 6 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/cba3c014/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cba3c014/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
--

[4/6] cassandra git commit: Merge branch 'cassandra-3.11' into cassandra-3.X

2016-11-30 Thread blambov

Merge branch 'cassandra-3.11' into cassandra-3.X


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2ff97fec
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2ff97fec
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2ff97fec

Branch: refs/heads/trunk
Commit: 2ff97fec30023b8eb45d9ade82fc6a659486f1c6
Parents: 0475922 f00e431
Author: Branimir Lambov 
Authored: Wed Nov 30 14:46:39 2016 +0200
Committer: Branimir Lambov 
Committed: Wed Nov 30 14:47:22 2016 +0200

--
 CHANGES.txt   |  1 +
 .../cql3/restrictions/StatementRestrictions.java  | 10 +-
 2 files changed, 6 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/2ff97fec/CHANGES.txt
--
diff --cc CHANGES.txt
index 47b7c2a,b238018..64ed71c
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,9 -1,5 +1,10 @@@
 +3.12
 + * cqlsh auto completion: refactor definition of compaction strategy options 
(CASSANDRA-12946)
 + * Add support for arithmetic operators (CASSANDRA-11935)
 +
 +
  3.10
+  * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
   * Remove timing window in test case (CASSANDRA-12875)
   * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
   * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)

[1/6] cassandra git commit: Use correct bounds for all-data range when filtering

2016-11-30 Thread blambov

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.11 8de24ca68 -> f00e43167
  refs/heads/cassandra-3.X 047592238 -> 2ff97fec3
  refs/heads/trunk bcb676223 -> cba3c0141


Use correct bounds for all-data range when filtering

Patch by Alex Petrov; reviewed by Branimir Lambov for CASSANDRA-12666.


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f00e4316
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f00e4316
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f00e4316

Branch: refs/heads/cassandra-3.11
Commit: f00e43167ab11f58af20439a300bdf82664abdb0
Parents: 8de24ca
Author: Alex Petrov 
Authored: Sun Sep 18 11:09:47 2016 +0200
Committer: Branimir Lambov 
Committed: Wed Nov 30 14:39:33 2016 +0200

--
 CHANGES.txt   |  1 +
 .../cql3/restrictions/StatementRestrictions.java  | 10 +-
 2 files changed, 6 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f00e4316/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 72d6a1f..b238018 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.10
+ * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
  * Remove timing window in test case (CASSANDRA-12875)
  * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
  * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f00e4316/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
--
diff --git 
a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java 
b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
index 53ac68c..2d04633 100644
--- a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
+++ b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
@@ -624,11 +624,6 @@ public final class StatementRestrictions
  */
 private ByteBuffer getPartitionKeyBound(Bound b, QueryOptions options)
 {
-// Deal with unrestricted partition key components (special-casing is 
required to deal with 2i queries on the
-// first component of a composite partition key) queries that filter 
on the partition key.
-if (partitionKeyRestrictions.needFiltering(cfm))
-return ByteBufferUtil.EMPTY_BYTE_BUFFER;
-
 // We deal with IN queries for keys in other places, so we know 
buildBound will return only one result
 return partitionKeyRestrictions.bounds(b, options).get(0);
 }
@@ -654,6 +649,11 @@ public final class StatementRestrictions
 private AbstractBounds 
getPartitionKeyBounds(IPartitioner p,
 
QueryOptions options)
 {
+// Deal with unrestricted partition key components (special-casing is 
required to deal with 2i queries on the
+// first component of a composite partition key) queries that filter 
on the partition key.
+if (partitionKeyRestrictions.needFiltering(cfm))
+return new Range<>(p.getMinimumToken().minKeyBound(), 
p.getMinimumToken().maxKeyBound());
+
 ByteBuffer startKeyBytes = getPartitionKeyBound(Bound.START, options);
 ByteBuffer finishKeyBytes = getPartitionKeyBound(Bound.END, options);

[5/6] cassandra git commit: Merge branch 'cassandra-3.11' into cassandra-3.X

2016-11-30 Thread blambov

Merge branch 'cassandra-3.11' into cassandra-3.X


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2ff97fec
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2ff97fec
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2ff97fec

Branch: refs/heads/cassandra-3.X
Commit: 2ff97fec30023b8eb45d9ade82fc6a659486f1c6
Parents: 0475922 f00e431
Author: Branimir Lambov 
Authored: Wed Nov 30 14:46:39 2016 +0200
Committer: Branimir Lambov 
Committed: Wed Nov 30 14:47:22 2016 +0200

--
 CHANGES.txt   |  1 +
 .../cql3/restrictions/StatementRestrictions.java  | 10 +-
 2 files changed, 6 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/2ff97fec/CHANGES.txt
--
diff --cc CHANGES.txt
index 47b7c2a,b238018..64ed71c
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,9 -1,5 +1,10 @@@
 +3.12
 + * cqlsh auto completion: refactor definition of compaction strategy options 
(CASSANDRA-12946)
 + * Add support for arithmetic operators (CASSANDRA-11935)
 +
 +
  3.10
+  * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
   * Remove timing window in test case (CASSANDRA-12875)
   * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
   * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)

[2/6] cassandra git commit: Use correct bounds for all-data range when filtering

2016-11-30 Thread blambov

Use correct bounds for all-data range when filtering

Patch by Alex Petrov; reviewed by Branimir Lambov for CASSANDRA-12666.


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f00e4316
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f00e4316
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f00e4316

Branch: refs/heads/cassandra-3.X
Commit: f00e43167ab11f58af20439a300bdf82664abdb0
Parents: 8de24ca
Author: Alex Petrov 
Authored: Sun Sep 18 11:09:47 2016 +0200
Committer: Branimir Lambov 
Committed: Wed Nov 30 14:39:33 2016 +0200

--
 CHANGES.txt   |  1 +
 .../cql3/restrictions/StatementRestrictions.java  | 10 +-
 2 files changed, 6 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f00e4316/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 72d6a1f..b238018 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.10
+ * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
  * Remove timing window in test case (CASSANDRA-12875)
  * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
  * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f00e4316/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
--
diff --git 
a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java 
b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
index 53ac68c..2d04633 100644
--- a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
+++ b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
@@ -624,11 +624,6 @@ public final class StatementRestrictions
  */
 private ByteBuffer getPartitionKeyBound(Bound b, QueryOptions options)
 {
-// Deal with unrestricted partition key components (special-casing is 
required to deal with 2i queries on the
-// first component of a composite partition key) queries that filter 
on the partition key.
-if (partitionKeyRestrictions.needFiltering(cfm))
-return ByteBufferUtil.EMPTY_BYTE_BUFFER;
-
 // We deal with IN queries for keys in other places, so we know 
buildBound will return only one result
 return partitionKeyRestrictions.bounds(b, options).get(0);
 }
@@ -654,6 +649,11 @@ public final class StatementRestrictions
 private AbstractBounds 
getPartitionKeyBounds(IPartitioner p,
 
QueryOptions options)
 {
+// Deal with unrestricted partition key components (special-casing is 
required to deal with 2i queries on the
+// first component of a composite partition key) queries that filter 
on the partition key.
+if (partitionKeyRestrictions.needFiltering(cfm))
+return new Range<>(p.getMinimumToken().minKeyBound(), 
p.getMinimumToken().maxKeyBound());
+
 ByteBuffer startKeyBytes = getPartitionKeyBound(Bound.START, options);
 ByteBuffer finishKeyBytes = getPartitionKeyBound(Bound.END, options);

[05/13] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

2016-11-18 Thread blambov

http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb41380c/test/unit/org/apache/cassandra/db/compaction/CompactionsPurgeTest.java
--
diff --cc test/unit/org/apache/cassandra/db/compaction/CompactionsPurgeTest.java
index 26d53ed,4a1f2ca..436b916
--- a/test/unit/org/apache/cassandra/db/compaction/CompactionsPurgeTest.java
+++ b/test/unit/org/apache/cassandra/db/compaction/CompactionsPurgeTest.java
@@@ -86,17 -98,16 +86,17 @@@ public class CompactionsPurgeTes
  String cfName = "Standard1";
  ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(cfName);
  
 -DecoratedKey key = Util.dk("key1");
 -Mutation rm;
 +String key = "key1";
  
  // inserts
 -rm = new Mutation(KEYSPACE1, key.getKey());
  for (int i = 0; i < 10; i++)
  {
 -rm.add(cfName, cellname(String.valueOf(i)), 
ByteBufferUtil.EMPTY_BYTE_BUFFER, 0);
 +RowUpdateBuilder builder = new RowUpdateBuilder(cfs.metadata, 0, 
key);
 +builder.clustering(String.valueOf(i))
- .add("val", ByteBufferUtil.EMPTY_BYTE_BUFFER)
- .build().applyUnsafe();
++   .add("val", ByteBufferUtil.EMPTY_BYTE_BUFFER)
++   .build().applyUnsafe();
  }
 -rm.applyUnsafe();
 +
  cfs.forceBlockingFlush();
  
  // deletes
@@@ -107,19 -120,148 +107,149 @@@
  cfs.forceBlockingFlush();
  
  // resurrect one column
 -rm = new Mutation(KEYSPACE1, key.getKey());
 -rm.add(cfName, cellname(String.valueOf(5)), 
ByteBufferUtil.EMPTY_BYTE_BUFFER, 2);
 -rm.applyUnsafe();
 +RowUpdateBuilder builder = new RowUpdateBuilder(cfs.metadata, 2, key);
 +builder.clustering(String.valueOf(5))
- .add("val", ByteBufferUtil.EMPTY_BYTE_BUFFER)
- .build().applyUnsafe();
++   .add("val", ByteBufferUtil.EMPTY_BYTE_BUFFER)
++   .build().applyUnsafe();
 +
  cfs.forceBlockingFlush();
  
  // major compact and test that all columns but the resurrected one is 
completely gone
  
FBUtilities.waitOnFutures(CompactionManager.instance.submitMaximal(cfs, 
Integer.MAX_VALUE, false));
 -cfs.invalidateCachedRow(key);
 -ColumnFamily cf = 
cfs.getColumnFamily(QueryFilter.getIdentityFilter(key, cfName, 
System.currentTimeMillis()));
 -assertColumns(cf, "5");
 -assertNotNull(cf.getColumn(cellname(String.valueOf(5;
 +cfs.invalidateCachedPartition(dk(key));
++
++ImmutableBTreePartition partition = 
Util.getOnlyPartitionUnfiltered(Util.cmd(cfs, key).build());
++assertEquals(1, partition.rowCount());
+ }
+ 
+ @Test
+ public void testMajorCompactionPurgeTombstonesWithMaxTimestamp()
+ {
+ CompactionManager.instance.disableAutoCompaction();
+ 
+ Keyspace keyspace = Keyspace.open(KEYSPACE1);
+ String cfName = "Standard1";
+ ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(cfName);
+ 
 -DecoratedKey key = Util.dk("key1");
 -Mutation rm;
++String key = "key1";
+ 
+ // inserts
 -rm = new Mutation(KEYSPACE1, key.getKey());
+ for (int i = 0; i < 10; i++)
+ {
 -rm.add(cfName, cellname(String.valueOf(i)), 
ByteBufferUtil.EMPTY_BYTE_BUFFER, 0);
++RowUpdateBuilder builder = new RowUpdateBuilder(cfs.metadata, 0, 
key);
++builder.clustering(String.valueOf(i))
++   .add("val", ByteBufferUtil.EMPTY_BYTE_BUFFER)
++   .build().applyUnsafe();
+ }
 -rm.apply();
+ cfs.forceBlockingFlush();
+ 
+ // deletes
+ for (int i = 0; i < 10; i++)
+ {
 -rm = new Mutation(KEYSPACE1, key.getKey());
 -rm.delete(cfName, cellname(String.valueOf(i)), Long.MAX_VALUE);
 -rm.apply();
++RowUpdateBuilder.deleteRow(cfs.metadata, Long.MAX_VALUE, key, 
String.valueOf(i)).applyUnsafe();
+ }
+ cfs.forceBlockingFlush();
+ 
+ // major compact - tombstones should be purged
+ 
FBUtilities.waitOnFutures(CompactionManager.instance.submitMaximal(cfs, 
Integer.MAX_VALUE, false));
+ 
+ // resurrect one column
 -rm = new Mutation(KEYSPACE1, key.getKey());
 -rm.add(cfName, cellname(String.valueOf(5)), 
ByteBufferUtil.EMPTY_BYTE_BUFFER, 2);
 -rm.apply();
++RowUpdateBuilder builder = new RowUpdateBuilder(cfs.metadata, 2, key);
++builder.clustering(String.valueOf(5))
++   .add("val", ByteBufferUtil.EMPTY_BYTE_BUFFER)
++   .build().applyUnsafe();
++
+ cfs.forceBlockingFlush();
+ 
 -cfs.invalidateCachedRow(key);
 -ColumnFamily cf = 
cfs.getColumnFamily(QueryFilter.getIdentityFilter(key, cfName, 
System.currentTimeMillis()));
 -

[13/13] cassandra git commit: Merge branch 'cassandra-3.X' into trunk

2016-11-18 Thread blambov

Merge branch 'cassandra-3.X' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/29cb5910
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/29cb5910
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/29cb5910

Branch: refs/heads/trunk
Commit: 29cb5910612b603e312d1e99555113d827f6cd9b
Parents: f55d1b9 f33cd55
Author: Branimir Lambov 
Authored: Fri Nov 18 12:45:24 2016 +0200
Committer: Branimir Lambov 
Committed: Fri Nov 18 12:45:24 2016 +0200

--
 CHANGES.txt |   2 +
 .../org/apache/cassandra/db/ReadCommand.java|   5 +-
 .../db/compaction/CompactionController.java |  46 +--
 .../db/compaction/CompactionIterator.java   |  22 +--
 .../db/compaction/CompactionManager.java|   5 +-
 .../db/compaction/SSTableSplitter.java  |   5 +-
 .../cassandra/db/compaction/Upgrader.java   |   5 +-
 .../cassandra/db/compaction/Verifier.java   |   5 +-
 .../cassandra/db/partitions/PurgeFunction.java  |   6 +-
 .../db/compaction/CompactionControllerTest.java |  21 ++-
 .../db/compaction/CompactionsPurgeTest.java | 138 ++-
 11 files changed, 213 insertions(+), 47 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/29cb5910/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/29cb5910/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--

[04/13] cassandra git commit: Rework tombstone purgeability checks to not use sentinel timestamp to indicate unconditional purgeability

2016-11-18 Thread blambov

Rework tombstone purgeability checks to not use sentinel timestamp to indicate 
unconditional purgeability

patch by Joel Knighton; reviewed by Branimir Lambov for CASSANDRA-12792


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7d2fdfeb
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7d2fdfeb
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7d2fdfeb

Branch: refs/heads/trunk
Commit: 7d2fdfeb41eca9badaf10b906b6afe077d166348
Parents: 84b9e72
Author: Joel Knighton 
Authored: Thu Oct 20 22:51:51 2016 -0500
Committer: Branimir Lambov 
Committed: Fri Nov 18 12:25:29 2016 +0200

--
 CHANGES.txt |   1 +
 .../db/compaction/CompactionController.java |  58 +---
 .../db/compaction/CompactionManager.java|   6 +-
 .../db/compaction/LazilyCompactedRow.java   |  24 ++--
 .../db/compaction/SSTableSplitter.java  |   7 +-
 .../cassandra/db/compaction/Scrubber.java   |   6 +-
 .../cassandra/db/compaction/Upgrader.java   |   6 +-
 .../cassandra/db/compaction/Verifier.java   |   6 +-
 .../db/compaction/CompactionControllerTest.java |  22 ++--
 .../db/compaction/CompactionsPurgeTest.java | 131 +++
 10 files changed, 219 insertions(+), 48 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7d2fdfeb/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 3482052..54dc4b5 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.9
+ * Fix purgeability of tombstones with max timestamp (CASSANDRA-12792)
  * Fail repair if participant dies during sync or anticompaction 
(CASSANDRA-12901)
  * cqlsh COPY: unprotected pk values before converting them if not using 
prepared statements (CASSANDRA-12863)
  * Fix Util.spinAssertEquals (CASSANDRA-12283)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/7d2fdfeb/src/java/org/apache/cassandra/db/compaction/CompactionController.java
--
diff --git 
a/src/java/org/apache/cassandra/db/compaction/CompactionController.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
index 699bc55..e895573 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
@@ -19,6 +19,9 @@ package org.apache.cassandra.db.compaction;
 
 import java.util.*;
 
+import com.google.common.base.Predicate;
+import com.google.common.base.Predicates;
+
 import org.apache.cassandra.io.sstable.format.SSTableReader;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -186,36 +189,59 @@ public class CompactionController implements AutoCloseable
 }
 
 /**
- * @return the largest timestamp before which it's okay to drop tombstones 
for the given partition;
- * i.e., after the maxPurgeableTimestamp there may exist newer data that 
still needs to be suppressed
- * in other sstables.  This returns the minimum timestamp for any SSTable 
that contains this partition and is not
- * participating in this compaction, or memtable that contains this 
partition,
- * or LONG.MAX_VALUE if no SSTable or memtable exist.
+ * @param key
+ * @return a predicate for whether tombstones marked for deletion at the 
given time for the given partition are
+ * purgeable; we calculate this by checking whether the deletion time is 
less than the min timestamp of all SSTables
+ * containing his partition and not participating in the compaction. This 
means there isn't any data in those
+ * sstables that might still need to be suppressed by a tombstone at this 
timestamp.
  */
-public long maxPurgeableTimestamp(DecoratedKey key)
+public Predicate getPurgeEvaluator(DecoratedKey key)
 {
 if (NEVER_PURGE_TOMBSTONES)
-return Long.MIN_VALUE;
+return Predicates.alwaysFalse();
 
-long min = Long.MAX_VALUE;
 overlapIterator.update(key);
-for (SSTableReader sstable : overlapIterator.overlaps())
+Set filteredSSTables = overlapIterator.overlaps();
+Iterable memtables = 
cfs.getTracker().getView().getAllMemtables();
+long minTimestampSeen = Long.MAX_VALUE;
+boolean hasTimestamp = false;
+
+for (SSTableReader sstable: filteredSSTables)
 {
 // if we don't have bloom filter(bf_fp_chance=1.0 or filter file 
is missing),
 // we check index file instead.
-if (sstable.getBloomFilter() instanceof AlwaysPresentFilter && 
sstable.getPosition(key, SSTableReader.Operator.EQ, false) != null)
-

[02/13] cassandra git commit: Rework tombstone purgeability checks to not use sentinel timestamp to indicate unconditional purgeability

2016-11-18 Thread blambov

Rework tombstone purgeability checks to not use sentinel timestamp to indicate 
unconditional purgeability

patch by Joel Knighton; reviewed by Branimir Lambov for CASSANDRA-12792


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7d2fdfeb
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7d2fdfeb
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7d2fdfeb

Branch: refs/heads/cassandra-3.0
Commit: 7d2fdfeb41eca9badaf10b906b6afe077d166348
Parents: 84b9e72
Author: Joel Knighton 
Authored: Thu Oct 20 22:51:51 2016 -0500
Committer: Branimir Lambov 
Committed: Fri Nov 18 12:25:29 2016 +0200

--
 CHANGES.txt |   1 +
 .../db/compaction/CompactionController.java |  58 +---
 .../db/compaction/CompactionManager.java|   6 +-
 .../db/compaction/LazilyCompactedRow.java   |  24 ++--
 .../db/compaction/SSTableSplitter.java  |   7 +-
 .../cassandra/db/compaction/Scrubber.java   |   6 +-
 .../cassandra/db/compaction/Upgrader.java   |   6 +-
 .../cassandra/db/compaction/Verifier.java   |   6 +-
 .../db/compaction/CompactionControllerTest.java |  22 ++--
 .../db/compaction/CompactionsPurgeTest.java | 131 +++
 10 files changed, 219 insertions(+), 48 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7d2fdfeb/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 3482052..54dc4b5 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.9
+ * Fix purgeability of tombstones with max timestamp (CASSANDRA-12792)
  * Fail repair if participant dies during sync or anticompaction 
(CASSANDRA-12901)
  * cqlsh COPY: unprotected pk values before converting them if not using 
prepared statements (CASSANDRA-12863)
  * Fix Util.spinAssertEquals (CASSANDRA-12283)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/7d2fdfeb/src/java/org/apache/cassandra/db/compaction/CompactionController.java
--
diff --git 
a/src/java/org/apache/cassandra/db/compaction/CompactionController.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
index 699bc55..e895573 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
@@ -19,6 +19,9 @@ package org.apache.cassandra.db.compaction;
 
 import java.util.*;
 
+import com.google.common.base.Predicate;
+import com.google.common.base.Predicates;
+
 import org.apache.cassandra.io.sstable.format.SSTableReader;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -186,36 +189,59 @@ public class CompactionController implements AutoCloseable
 }
 
 /**
- * @return the largest timestamp before which it's okay to drop tombstones 
for the given partition;
- * i.e., after the maxPurgeableTimestamp there may exist newer data that 
still needs to be suppressed
- * in other sstables.  This returns the minimum timestamp for any SSTable 
that contains this partition and is not
- * participating in this compaction, or memtable that contains this 
partition,
- * or LONG.MAX_VALUE if no SSTable or memtable exist.
+ * @param key
+ * @return a predicate for whether tombstones marked for deletion at the 
given time for the given partition are
+ * purgeable; we calculate this by checking whether the deletion time is 
less than the min timestamp of all SSTables
+ * containing his partition and not participating in the compaction. This 
means there isn't any data in those
+ * sstables that might still need to be suppressed by a tombstone at this 
timestamp.
  */
-public long maxPurgeableTimestamp(DecoratedKey key)
+public Predicate getPurgeEvaluator(DecoratedKey key)
 {
 if (NEVER_PURGE_TOMBSTONES)
-return Long.MIN_VALUE;
+return Predicates.alwaysFalse();
 
-long min = Long.MAX_VALUE;
 overlapIterator.update(key);
-for (SSTableReader sstable : overlapIterator.overlaps())
+Set filteredSSTables = overlapIterator.overlaps();
+Iterable memtables = 
cfs.getTracker().getView().getAllMemtables();
+long minTimestampSeen = Long.MAX_VALUE;
+boolean hasTimestamp = false;
+
+for (SSTableReader sstable: filteredSSTables)
 {
 // if we don't have bloom filter(bf_fp_chance=1.0 or filter file 
is missing),
 // we check index file instead.
-if (sstable.getBloomFilter() instanceof AlwaysPresentFilter && 
sstable.getPosition(key, SSTableReader.Operator.EQ, false) != null)
-

[09/13] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

2016-11-18 Thread blambov

http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb41380c/test/unit/org/apache/cassandra/db/compaction/CompactionsPurgeTest.java
--
diff --cc test/unit/org/apache/cassandra/db/compaction/CompactionsPurgeTest.java
index 26d53ed,4a1f2ca..436b916
--- a/test/unit/org/apache/cassandra/db/compaction/CompactionsPurgeTest.java
+++ b/test/unit/org/apache/cassandra/db/compaction/CompactionsPurgeTest.java
@@@ -86,17 -98,16 +86,17 @@@ public class CompactionsPurgeTes
  String cfName = "Standard1";
  ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(cfName);
  
 -DecoratedKey key = Util.dk("key1");
 -Mutation rm;
 +String key = "key1";
  
  // inserts
 -rm = new Mutation(KEYSPACE1, key.getKey());
  for (int i = 0; i < 10; i++)
  {
 -rm.add(cfName, cellname(String.valueOf(i)), 
ByteBufferUtil.EMPTY_BYTE_BUFFER, 0);
 +RowUpdateBuilder builder = new RowUpdateBuilder(cfs.metadata, 0, 
key);
 +builder.clustering(String.valueOf(i))
- .add("val", ByteBufferUtil.EMPTY_BYTE_BUFFER)
- .build().applyUnsafe();
++   .add("val", ByteBufferUtil.EMPTY_BYTE_BUFFER)
++   .build().applyUnsafe();
  }
 -rm.applyUnsafe();
 +
  cfs.forceBlockingFlush();
  
  // deletes
@@@ -107,19 -120,148 +107,149 @@@
  cfs.forceBlockingFlush();
  
  // resurrect one column
 -rm = new Mutation(KEYSPACE1, key.getKey());
 -rm.add(cfName, cellname(String.valueOf(5)), 
ByteBufferUtil.EMPTY_BYTE_BUFFER, 2);
 -rm.applyUnsafe();
 +RowUpdateBuilder builder = new RowUpdateBuilder(cfs.metadata, 2, key);
 +builder.clustering(String.valueOf(5))
- .add("val", ByteBufferUtil.EMPTY_BYTE_BUFFER)
- .build().applyUnsafe();
++   .add("val", ByteBufferUtil.EMPTY_BYTE_BUFFER)
++   .build().applyUnsafe();
 +
  cfs.forceBlockingFlush();
  
  // major compact and test that all columns but the resurrected one is 
completely gone
  
FBUtilities.waitOnFutures(CompactionManager.instance.submitMaximal(cfs, 
Integer.MAX_VALUE, false));
 -cfs.invalidateCachedRow(key);
 -ColumnFamily cf = 
cfs.getColumnFamily(QueryFilter.getIdentityFilter(key, cfName, 
System.currentTimeMillis()));
 -assertColumns(cf, "5");
 -assertNotNull(cf.getColumn(cellname(String.valueOf(5;
 +cfs.invalidateCachedPartition(dk(key));
++
++ImmutableBTreePartition partition = 
Util.getOnlyPartitionUnfiltered(Util.cmd(cfs, key).build());
++assertEquals(1, partition.rowCount());
+ }
+ 
+ @Test
+ public void testMajorCompactionPurgeTombstonesWithMaxTimestamp()
+ {
+ CompactionManager.instance.disableAutoCompaction();
+ 
+ Keyspace keyspace = Keyspace.open(KEYSPACE1);
+ String cfName = "Standard1";
+ ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(cfName);
+ 
 -DecoratedKey key = Util.dk("key1");
 -Mutation rm;
++String key = "key1";
+ 
+ // inserts
 -rm = new Mutation(KEYSPACE1, key.getKey());
+ for (int i = 0; i < 10; i++)
+ {
 -rm.add(cfName, cellname(String.valueOf(i)), 
ByteBufferUtil.EMPTY_BYTE_BUFFER, 0);
++RowUpdateBuilder builder = new RowUpdateBuilder(cfs.metadata, 0, 
key);
++builder.clustering(String.valueOf(i))
++   .add("val", ByteBufferUtil.EMPTY_BYTE_BUFFER)
++   .build().applyUnsafe();
+ }
 -rm.apply();
+ cfs.forceBlockingFlush();
+ 
+ // deletes
+ for (int i = 0; i < 10; i++)
+ {
 -rm = new Mutation(KEYSPACE1, key.getKey());
 -rm.delete(cfName, cellname(String.valueOf(i)), Long.MAX_VALUE);
 -rm.apply();
++RowUpdateBuilder.deleteRow(cfs.metadata, Long.MAX_VALUE, key, 
String.valueOf(i)).applyUnsafe();
+ }
+ cfs.forceBlockingFlush();
+ 
+ // major compact - tombstones should be purged
+ 
FBUtilities.waitOnFutures(CompactionManager.instance.submitMaximal(cfs, 
Integer.MAX_VALUE, false));
+ 
+ // resurrect one column
 -rm = new Mutation(KEYSPACE1, key.getKey());
 -rm.add(cfName, cellname(String.valueOf(5)), 
ByteBufferUtil.EMPTY_BYTE_BUFFER, 2);
 -rm.apply();
++RowUpdateBuilder builder = new RowUpdateBuilder(cfs.metadata, 2, key);
++builder.clustering(String.valueOf(5))
++   .add("val", ByteBufferUtil.EMPTY_BYTE_BUFFER)
++   .build().applyUnsafe();
++
+ cfs.forceBlockingFlush();
+ 
 -cfs.invalidateCachedRow(key);
 -ColumnFamily cf = 
cfs.getColumnFamily(QueryFilter.getIdentityFilter(key, cfName, 
System.currentTimeMillis()));
 -

[12/13] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.X

2016-11-18 Thread blambov

Merge branch 'cassandra-3.0' into cassandra-3.X


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f33cd55a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f33cd55a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f33cd55a

Branch: refs/heads/trunk
Commit: f33cd55a5bbf9a8ba0073c606b971d3b3fc85471
Parents: 490c1c2 eb41380
Author: Branimir Lambov 
Authored: Fri Nov 18 12:43:04 2016 +0200
Committer: Branimir Lambov 
Committed: Fri Nov 18 12:44:09 2016 +0200

--
 CHANGES.txt |   2 +
 .../org/apache/cassandra/db/ReadCommand.java|   5 +-
 .../db/compaction/CompactionController.java |  46 +--
 .../db/compaction/CompactionIterator.java   |  22 +--
 .../db/compaction/CompactionManager.java|   5 +-
 .../db/compaction/SSTableSplitter.java  |   5 +-
 .../cassandra/db/compaction/Upgrader.java   |   5 +-
 .../cassandra/db/compaction/Verifier.java   |   5 +-
 .../cassandra/db/partitions/PurgeFunction.java  |   6 +-
 .../db/compaction/CompactionControllerTest.java |  21 ++-
 .../db/compaction/CompactionsPurgeTest.java | 138 ++-
 11 files changed, 213 insertions(+), 47 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f33cd55a/CHANGES.txt
--
diff --cc CHANGES.txt
index 6ca26f9,8a3ac65..ee73b81
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -150,6 -37,7 +150,8 @@@ Merged from 3.0
   * Correct log message for statistics of offheap memtable flush 
(CASSANDRA-12776)
   * Explicitly set locale for string validation 
(CASSANDRA-12541,CASSANDRA-12542,CASSANDRA-12543,CASSANDRA-12545)
  Merged from 2.2:
++===
+  * Fix purgeability of tombstones with max timestamp (CASSANDRA-12792)
   * Fail repair if participant dies during sync or anticompaction 
(CASSANDRA-12901)
   * cqlsh COPY: unprotected pk values before converting them if not using 
prepared statements (CASSANDRA-12863)
   * Fix Util.spinAssertEquals (CASSANDRA-12283)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f33cd55a/src/java/org/apache/cassandra/db/ReadCommand.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f33cd55a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
--
diff --cc src/java/org/apache/cassandra/db/compaction/CompactionController.java
index b34eee6,34d093e..64c35d9
--- a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
@@@ -18,13 -18,10 +18,14 @@@
  package org.apache.cassandra.db.compaction;
  
  import java.util.*;
+ import java.util.function.Predicate;
  
  import org.apache.cassandra.db.Memtable;
 +import org.apache.cassandra.db.rows.UnfilteredRowIterator;
 +
 +import com.google.common.base.Predicates;
  import com.google.common.collect.Iterables;
 +import com.google.common.util.concurrent.RateLimiter;
  
  import org.apache.cassandra.db.partitions.Partition;
  import org.apache.cassandra.io.sstable.format.SSTableReader;
@@@ -213,20 -194,24 +214,24 @@@ public class CompactionController imple
  }
  
  /**
-  * @return the largest timestamp before which it's okay to drop 
tombstones for the given partition;
-  * i.e., after the maxPurgeableTimestamp there may exist newer data that 
still needs to be suppressed
-  * in other sstables.  This returns the minimum timestamp for any SSTable 
that contains this partition and is not
-  * participating in this compaction, or memtable that contains this 
partition,
-  * or LONG.MAX_VALUE if no SSTable or memtable exist.
+  * @param key
+  * @return a predicate for whether tombstones marked for deletion at the 
given time for the given partition are
+  * purgeable; we calculate this by checking whether the deletion time is 
less than the min timestamp of all SSTables
+  * containing his partition and not participating in the compaction. This 
means there isn't any data in those
+  * sstables that might still need to be suppressed by a tombstone at this 
timestamp.
   */
- public long maxPurgeableTimestamp(DecoratedKey key)
+ public Predicate getPurgeEvaluator(DecoratedKey key)
  {
 -if (!compactingRepaired() || NEVER_PURGE_TOMBSTONES)
 +if (NEVER_PURGE_TOMBSTONES || !compactingRepaired())
- return Long.MIN_VALUE;
+ return time -> false;
  
- long min = Long.MAX_VALUE;
  overlapIterator.update(key);
- for (SSTableReader sstable :

[06/13] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

2016-11-18 Thread blambov

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/eb41380c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/eb41380c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/eb41380c

Branch: refs/heads/cassandra-3.X
Commit: eb41380cc27277e34edf2c74f535588fd1382a9a
Parents: 14f36fc 7d2fdfe
Author: Branimir Lambov 
Authored: Fri Nov 18 12:35:32 2016 +0200
Committer: Branimir Lambov 
Committed: Fri Nov 18 12:36:26 2016 +0200

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/db/ReadCommand.java|   5 +-
 .../db/compaction/CompactionController.java |  50 ---
 .../db/compaction/CompactionIterator.java   |  22 +--
 .../db/compaction/CompactionManager.java|   5 +-
 .../db/compaction/SSTableSplitter.java  |   5 +-
 .../cassandra/db/compaction/Upgrader.java   |   5 +-
 .../cassandra/db/compaction/Verifier.java   |   5 +-
 .../cassandra/db/partitions/PurgeFunction.java  |   6 +-
 .../db/compaction/CompactionControllerTest.java |  21 ++-
 .../db/compaction/CompactionsPurgeTest.java | 138 ++-
 11 files changed, 213 insertions(+), 50 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb41380c/CHANGES.txt
--
diff --cc CHANGES.txt
index efc681d,54dc4b5..8a3ac65
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,42 -1,5 +1,43 @@@
 -2.2.9
 +3.0.11
 + * Prevent reloading of logback.xml from UDF sandbox (CASSANDRA-12535)
 +
 +3.0.10
 + * Disallow offheap_buffers memtable allocation (CASSANDRA-11039)
 + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283)
 + * Pass root cause to CorruptBlockException when uncompression failed 
(CASSANDRA-12889)
 + * Fix partition count log during compaction (CASSANDRA-12184)
 + * Batch with multiple conditional updates for the same partition causes 
AssertionError (CASSANDRA-12867)
 + * Make AbstractReplicationStrategy extendable from outside its package 
(CASSANDRA-12788)
 + * Fix CommitLogTest.testDeleteIfNotDirty (CASSANDRA-12854)
 + * Don't tell users to turn off consistent rangemovements during rebuild. 
(CASSANDRA-12296)
 + * Avoid deadlock due to materialized view lock contention (CASSANDRA-12689)
 + * Fix for KeyCacheCqlTest flakiness (CASSANDRA-12801)
 + * Include SSTable filename in compacting large row message (CASSANDRA-12384)
 + * Fix potential socket leak (CASSANDRA-12329, CASSANDRA-12330)
 + * Fix ViewTest.testCompaction (CASSANDRA-12789)
 + * Improve avg aggregate functions (CASSANDRA-12417)
 + * Preserve quoted reserved keyword column names in MV creation 
(CASSANDRA-11803)
 + * nodetool stopdaemon errors out (CASSANDRA-12646)
 + * Split materialized view mutations on build to prevent OOM (CASSANDRA-12268)
 + * mx4j does not work in 3.0.8 (CASSANDRA-12274)
 + * Abort cqlsh copy-from in case of no answer after prolonged period of time 
(CASSANDRA-12740)
 + * Avoid sstable corrupt exception due to dropped static column 
(CASSANDRA-12582)
 + * Make stress use client mode to avoid checking commit log size on startup 
(CASSANDRA-12478)
 + * Fix exceptions with new vnode allocation (CASSANDRA-12715)
 + * Unify drain and shutdown processes (CASSANDRA-12509)
 + * Fix NPE in ComponentOfSlice.isEQ() (CASSANDRA-12706)
 + * Fix failure in LogTransactionTest (CASSANDRA-12632)
 + * Fix potentially incomplete non-frozen UDT values when querying with the
 +   full primary key specified (CASSANDRA-12605)
 + * Skip writing MV mutations to commitlog on mutation.applyUnsafe() 
(CASSANDRA-11670)
 + * Establish consistent distinction between non-existing partition and NULL 
value for LWTs on static columns (CASSANDRA-12060)
 + * Extend ColumnIdentifier.internedInstances key to include the type that 
generated the byte buffer (CASSANDRA-12516)
 + * Backport CASSANDRA-10756 (race condition in NativeTransportService 
shutdown) (CASSANDRA-12472)
 + * If CF has no clustering columns, any row cache is full partition cache 
(CASSANDRA-12499)
 + * Correct log message for statistics of offheap memtable flush 
(CASSANDRA-12776)
 + * Explicitly set locale for string validation 
(CASSANDRA-12541,CASSANDRA-12542,CASSANDRA-12543,CASSANDRA-12545)
 +Merged from 2.2:
+  * Fix purgeability of tombstones with max timestamp (CASSANDRA-12792)
   * Fail repair if participant dies during sync or anticompaction 
(CASSANDRA-12901)
   * cqlsh COPY: unprotected pk values before converting them if not using 
prepared statements (CASSANDRA-12863)
   * Fix Util.spinAssertEquals (CASSANDRA-12283)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb41380c/src/java/org/apache/cassandra/db/ReadCommand.java

[10/13] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

2016-11-18 Thread blambov

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/eb41380c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/eb41380c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/eb41380c

Branch: refs/heads/trunk
Commit: eb41380cc27277e34edf2c74f535588fd1382a9a
Parents: 14f36fc 7d2fdfe
Author: Branimir Lambov 
Authored: Fri Nov 18 12:35:32 2016 +0200
Committer: Branimir Lambov 
Committed: Fri Nov 18 12:36:26 2016 +0200

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/db/ReadCommand.java|   5 +-
 .../db/compaction/CompactionController.java |  50 ---
 .../db/compaction/CompactionIterator.java   |  22 +--
 .../db/compaction/CompactionManager.java|   5 +-
 .../db/compaction/SSTableSplitter.java  |   5 +-
 .../cassandra/db/compaction/Upgrader.java   |   5 +-
 .../cassandra/db/compaction/Verifier.java   |   5 +-
 .../cassandra/db/partitions/PurgeFunction.java  |   6 +-
 .../db/compaction/CompactionControllerTest.java |  21 ++-
 .../db/compaction/CompactionsPurgeTest.java | 138 ++-
 11 files changed, 213 insertions(+), 50 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb41380c/CHANGES.txt
--
diff --cc CHANGES.txt
index efc681d,54dc4b5..8a3ac65
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,42 -1,5 +1,43 @@@
 -2.2.9
 +3.0.11
 + * Prevent reloading of logback.xml from UDF sandbox (CASSANDRA-12535)
 +
 +3.0.10
 + * Disallow offheap_buffers memtable allocation (CASSANDRA-11039)
 + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283)
 + * Pass root cause to CorruptBlockException when uncompression failed 
(CASSANDRA-12889)
 + * Fix partition count log during compaction (CASSANDRA-12184)
 + * Batch with multiple conditional updates for the same partition causes 
AssertionError (CASSANDRA-12867)
 + * Make AbstractReplicationStrategy extendable from outside its package 
(CASSANDRA-12788)
 + * Fix CommitLogTest.testDeleteIfNotDirty (CASSANDRA-12854)
 + * Don't tell users to turn off consistent rangemovements during rebuild. 
(CASSANDRA-12296)
 + * Avoid deadlock due to materialized view lock contention (CASSANDRA-12689)
 + * Fix for KeyCacheCqlTest flakiness (CASSANDRA-12801)
 + * Include SSTable filename in compacting large row message (CASSANDRA-12384)
 + * Fix potential socket leak (CASSANDRA-12329, CASSANDRA-12330)
 + * Fix ViewTest.testCompaction (CASSANDRA-12789)
 + * Improve avg aggregate functions (CASSANDRA-12417)
 + * Preserve quoted reserved keyword column names in MV creation 
(CASSANDRA-11803)
 + * nodetool stopdaemon errors out (CASSANDRA-12646)
 + * Split materialized view mutations on build to prevent OOM (CASSANDRA-12268)
 + * mx4j does not work in 3.0.8 (CASSANDRA-12274)
 + * Abort cqlsh copy-from in case of no answer after prolonged period of time 
(CASSANDRA-12740)
 + * Avoid sstable corrupt exception due to dropped static column 
(CASSANDRA-12582)
 + * Make stress use client mode to avoid checking commit log size on startup 
(CASSANDRA-12478)
 + * Fix exceptions with new vnode allocation (CASSANDRA-12715)
 + * Unify drain and shutdown processes (CASSANDRA-12509)
 + * Fix NPE in ComponentOfSlice.isEQ() (CASSANDRA-12706)
 + * Fix failure in LogTransactionTest (CASSANDRA-12632)
 + * Fix potentially incomplete non-frozen UDT values when querying with the
 +   full primary key specified (CASSANDRA-12605)
 + * Skip writing MV mutations to commitlog on mutation.applyUnsafe() 
(CASSANDRA-11670)
 + * Establish consistent distinction between non-existing partition and NULL 
value for LWTs on static columns (CASSANDRA-12060)
 + * Extend ColumnIdentifier.internedInstances key to include the type that 
generated the byte buffer (CASSANDRA-12516)
 + * Backport CASSANDRA-10756 (race condition in NativeTransportService 
shutdown) (CASSANDRA-12472)
 + * If CF has no clustering columns, any row cache is full partition cache 
(CASSANDRA-12499)
 + * Correct log message for statistics of offheap memtable flush 
(CASSANDRA-12776)
 + * Explicitly set locale for string validation 
(CASSANDRA-12541,CASSANDRA-12542,CASSANDRA-12543,CASSANDRA-12545)
 +Merged from 2.2:
+  * Fix purgeability of tombstones with max timestamp (CASSANDRA-12792)
   * Fail repair if participant dies during sync or anticompaction 
(CASSANDRA-12901)
   * cqlsh COPY: unprotected pk values before converting them if not using 
prepared statements (CASSANDRA-12863)
   * Fix Util.spinAssertEquals (CASSANDRA-12283)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb41380c/src/java/org/apache/cassandra/db/ReadCommand.java

1 2 >

1 - 100 of 104 matches

Mail list logo