from:"slebresne"

[cassandra] branch trunk updated (c26269b -> 9a3ca00)

2020-11-27 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from c26269b  ninja fixing wrongly committed circle-ci
 new 2d0b168  Fix serial read/non-applying CAS linearizability
 new 080280d  Merge commit '2d0b16804785660e8515aca9944784fb3733c619' into 
cassandra-3.11
 new 9a3ca00  Merge commit '080280dc0177da6176dd4ba970e5a35aa7e2a729' into 
trunk

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|   1 +
 NEWS.txt   |   8 +
 .../org/apache/cassandra/service/StorageProxy.java | 352 +++
 .../org/apache/cassandra/service/paxos/Commit.java |   6 +
 .../cassandra/service/paxos/PrepareCallback.java   |  12 +-
 .../cassandra/distributed/impl/Instance.java   |  83 +++
 .../apache/cassandra/distributed/test/CASTest.java | 688 +
 7 files changed, 1034 insertions(+), 116 deletions(-)
 create mode 100644 
test/distributed/org/apache/cassandra/distributed/test/CASTest.java


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 01/01: Merge commit '2d0b16804785660e8515aca9944784fb3733c619' into cassandra-3.11

2020-11-27 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 080280dc0177da6176dd4ba970e5a35aa7e2a729
Merge: 7ffb2fe 2d0b168
Author: Sylvain Lebresne 
AuthorDate: Fri Nov 27 17:05:48 2020 +0100

Merge commit '2d0b16804785660e8515aca9944784fb3733c619' into cassandra-3.11

 CHANGES.txt|   1 +
 NEWS.txt   |   8 +
 .../org/apache/cassandra/service/StorageProxy.java | 239 +--
 .../org/apache/cassandra/service/paxos/Commit.java |   6 +
 .../apache/cassandra/service/paxos/PaxosState.java |   3 -
 .../cassandra/service/paxos/PrepareCallback.java   |  12 +-
 .../cassandra/distributed/impl/Instance.java   | 113 ++--
 .../apache/cassandra/distributed/test/CASTest.java | 684 +
 8 files changed, 949 insertions(+), 117 deletions(-)

diff --cc CHANGES.txt
index dd554c2,d6f406d..c3c5f02
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,7 -1,5 +1,8 @@@
 -3.0.24:
 +3.11.10
 + * Rate limit validation compactions using compaction_throughput_mb_per_sec 
(CASSANDRA-16161)
 + * SASI's `max_compaction_flush_memory_in_mb` settings over 100GB revert to 
default of 1GB (CASSANDRA-16071)
 +Merged from 3.0:
+  * Fix serial read/non-applying CAS linearizability (CASSANDRA-12126)
   * Avoid potential NPE in JVMStabilityInspector (CASSANDRA-16294)
   * Improved check of num_tokens against the length of initial_token 
(CASSANDRA-14477)
   * Fix a race condition on ColumnFamilyStore and TableMetrics 
(CASSANDRA-16228)
diff --cc NEWS.txt
index 3af2150,6cc5e84..c5a3439
--- a/NEWS.txt
+++ b/NEWS.txt
@@@ -42,10 -42,19 +42,18 @@@ restore snapshots created with the prev
  'sstableloader' tool. You can upgrade the file format of your snapshots
  using the provided 'sstableupgrade' tool.
  
 -3.0.24
 -==
 -
 +3.11.10
 +=
  Upgrading
  -
+ - This release fix a correctness issue with SERIAL reads, and LWT writes 
that do not apply.
+   Unfortunately, this fix has a performance impact on read performance at 
the SERIAL or
+   LOCAL_SERIAL consistency levels. For heavy users of such SERIAL reads, 
the performance
+   impact may be noticeable and may also result in an increased of 
timeouts. For that
+   reason, a opt-in system property has been added to disable the fix:
+ -Dcassandra.unsafe.disable-serial-reads-linearizability=true
+   Use this flag at your own risk as it revert SERIAL reads to the 
incorrect behavior of
+   previous versions. See CASSANDRA-12126 for details.
  - In cassandra.yaml, when using vnodes num_tokens must be defined if 
initial_token is defined.
If it is not defined, or not equal to the numbers of tokens defined in 
initial_tokens,
the node will not start. See CASSANDRA-14477 for details.
diff --cc src/java/org/apache/cassandra/service/StorageProxy.java
index b1e0696,91dd991..d6f713e
--- a/src/java/org/apache/cassandra/service/StorageProxy.java
+++ b/src/java/org/apache/cassandra/service/StorageProxy.java
@@@ -169,11 -169,15 +174,21 @@@ public class StorageProxy implements St
  }
  };
  
 +for(ConsistencyLevel level : ConsistencyLevel.values())
 +{
 +readMetricsMap.put(level, new ClientRequestMetrics("Read-" + 
level.name()));
 +writeMetricsMap.put(level, new ClientRequestMetrics("Write-" + 
level.name()));
 +}
++
+ if (disableSerialReadLinearizability)
+ {
+ logger.warn("This node was started with -D{}. SERIAL (and 
LOCAL_SERIAL) reads coordinated by this node " +
+ "will not offer linearizability (see CASSANDRA-12126 
for details on what this mean) with " +
+ "respect to other SERIAL operations. Please note 
that, with this flag, SERIAL reads will be " +
+ "slower than QUORUM reads, yet offer no more 
guarantee. This flag should only be used in " +
+ "the restricted case of upgrading from a 
pre-CASSANDRA-12126 version, and only if you " +
+ "understand the tradeoff.", 
DISABLE_SERIAL_READ_LINEARIZABILITY_KEY);
+ }
  }
  
  /**
@@@ -223,31 -227,16 +238,17 @@@
CASRequest request,
ConsistencyLevel consistencyForPaxos,
ConsistencyLevel consistencyForCommit,
 -  ClientState state)
 +  ClientState state,
 +  long queryStartNanoTime)
  throws UnavailableException, IsBootstrappingException, 
RequestFailureException, RequestTimeoutException, InvalidRequestException
  {
 -final long start = System.nanoTim

[cassandra] 01/01: Merge commit '080280dc0177da6176dd4ba970e5a35aa7e2a729' into trunk

2020-11-27 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 9a3ca008bad2a7bfa887a8ba2d119fdc4369ba08
Merge: c26269b 080280d
Author: Sylvain Lebresne 
AuthorDate: Fri Nov 27 17:07:15 2020 +0100

Merge commit '080280dc0177da6176dd4ba970e5a35aa7e2a729' into trunk

 CHANGES.txt|   1 +
 NEWS.txt   |   8 +
 .../org/apache/cassandra/service/StorageProxy.java | 352 +++
 .../org/apache/cassandra/service/paxos/Commit.java |   6 +
 .../cassandra/service/paxos/PrepareCallback.java   |  12 +-
 .../cassandra/distributed/impl/Instance.java   |  83 +++
 .../apache/cassandra/distributed/test/CASTest.java | 688 +
 7 files changed, 1034 insertions(+), 116 deletions(-)

diff --cc CHANGES.txt
index 92094dc,c3c5f02..56843d7
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,20 -1,8 +1,21 @@@
 -3.11.10
 - * Rate limit validation compactions using compaction_throughput_mb_per_sec 
(CASSANDRA-16161)
 +4.0-beta4
 + * Update jctools dependency to 3.1.0 (CASSANDRA-16255)
 + * 'SSLEngine closed already' exception on failed outbound connection 
(CASSANDRA-16277)
 + * Drain and/or shutdown might throw because of slow messaging service 
shutdown (CASSANDRA-16276)
 + * Upgrade JNA to 5.6.0, dropping support for <=glibc-2.6 systems 
(CASSANDRA-16212)
 + * Add saved Host IDs to TokenMetadata at startup (CASSANDRA-16246)
 + * Ensure that CacheMetrics.requests is picked up by the metric reporter 
(CASSANDRA-16228)
 + * Add a ratelimiter to snapshot creation and deletion (CASSANDRA-13019)
 + * Produce consistent tombstone for reads to avoid digest mistmatch 
(CASSANDRA-15369)
 + * Fix SSTableloader issue when restoring a table named backups 
(CASSANDRA-16235)
 + * Invalid serialized size for responses caused by increasing message time by 
1ms which caused extra bytes in size calculation (CASSANDRA-16103)
 + * Throw BufferOverflowException from DataOutputBuffer for better visibility 
(CASSANDRA-16214)
 + * TLS connections to the storage port on a node without server encryption 
configured causes java.io.IOException accessing missing keystore 
(CASSANDRA-16144)
 + * Internode messaging catches OOMs and does not rethrow (CASSANDRA-15214)
 +Merged from 3.11:
   * SASI's `max_compaction_flush_memory_in_mb` settings over 100GB revert to 
default of 1GB (CASSANDRA-16071)
  Merged from 3.0:
+  * Fix serial read/non-applying CAS linearizability (CASSANDRA-12126)
   * Avoid potential NPE in JVMStabilityInspector (CASSANDRA-16294)
   * Improved check of num_tokens against the length of initial_token 
(CASSANDRA-14477)
   * Fix a race condition on ColumnFamilyStore and TableMetrics 
(CASSANDRA-16228)
diff --cc NEWS.txt
index 9774d00,c5a3439..d02f2f0
--- a/NEWS.txt
+++ b/NEWS.txt
@@@ -33,256 -42,21 +33,264 @@@ restore snapshots created with the prev
  'sstableloader' tool. You can upgrade the file format of your snapshots
  using the provided 'sstableupgrade' tool.
  
 +4.0
 +===
 +
 +New features
 +
 +- Nodes will now bootstrap all intra-cluster connections at startup by 
default and wait
 +  10 seconds for the all but one node in the local data center to be 
connected and marked
 +  UP in gossip. This prevents nodes from coordinating requests and 
failing because they
 +  aren't able to connect to the cluster fast enough. 
block_for_peers_timeout_in_secs in
 +  cassandra.yaml can be used to configure how long to wait (or whether to 
wait at all)
 +  and block_for_peers_in_remote_dcs can be used to also block on all but 
one node in
 +  each remote DC as well. See CASSANDRA-14297 and CASSANDRA-13993 for 
more information.
 +- *Experimental* support for Transient Replication and Cheap Quorums 
introduced by CASSANDRA-14404
 +  The intended audience for this functionality is expert users of 
Cassandra who are prepared
 +  to validate every aspect of the database for their application and 
deployment practices. Future
 +  releases of Cassandra will make this feature suitable for a wider 
audience.
 +- *Experimental* support for Java 11 has been added. JVM options that 
differ between or are
 +  specific for Java 8 and 11 have been moved from jvm.options into 
jvm8.options and jvm11.options.
 +  IMPORTANT: Running C* on Java 11 is *experimental* and do it at your 
own risk.
 +- LCS now respects the max_threshold parameter when compacting - this was 
hard coded to 32
 +  before, but now it is possible to do bigger compactions when compacting 
from L0 to L1.
 +  This also applies to STCS-compactions in L0 - if there are more than 32 
sstables in L0
 +  we will compact at most max_threshold sstables in an L0 STCS 
compaction. See CASSANDRA-14388
 +  for more information.
 +- There is now an option to automatically upgrade sstables af

[cassandra] branch cassandra-3.11 updated (7ffb2fe -> 080280d)

2020-11-27 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a change to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from 7ffb2fe  Merge branch 'cassandra-3.0' into cassandra-3.11
 new 2d0b168  Fix serial read/non-applying CAS linearizability
 new 080280d  Merge commit '2d0b16804785660e8515aca9944784fb3733c619' into 
cassandra-3.11

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|   1 +
 NEWS.txt   |   8 +
 .../org/apache/cassandra/service/StorageProxy.java | 239 +--
 .../org/apache/cassandra/service/paxos/Commit.java |   6 +
 .../apache/cassandra/service/paxos/PaxosState.java |   3 -
 .../cassandra/service/paxos/PrepareCallback.java   |  12 +-
 .../cassandra/distributed/impl/Instance.java   | 113 ++--
 .../apache/cassandra/distributed/test/CASTest.java | 684 +
 8 files changed, 949 insertions(+), 117 deletions(-)
 create mode 100644 
test/distributed/org/apache/cassandra/distributed/test/CASTest.java


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch cassandra-3.0 updated: Fix serial read/non-applying CAS linearizability

2020-11-27 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/cassandra-3.0 by this push:
 new 2d0b168  Fix serial read/non-applying CAS linearizability
2d0b168 is described below

commit 2d0b16804785660e8515aca9944784fb3733c619
Author: Sylvain Lebresne 
AuthorDate: Wed May 20 14:56:02 2020 +0200

Fix serial read/non-applying CAS linearizability

Before this patch, a SERIAL read or a non-applying CAS replay any
in-progress commit by calling `beginAndRepairPaxos`, but only a quorum
of nodes is contacted, so a minority of nodes could have an unfinished
in-progress proposal in their Paxos state. If such in-progress proposal
is not replayed by a SERIAL read/non-applying CAS, it should never be
replayed by any following operation as that would break serializability,
but nothing was done to avoid this.

This patch ensures that both a SERIAL read or a non-applying CAS commit
an empty update before succeeding. This ensures that no prior incomplete
in-progress proposal can be replayed (such proposal will be discarded as
older than the last committed ballot).

As this fix has a performance impact on SERIAL reads, a flag is provided
to disable the new code (even if this is discouraged by a warning).

Patch by Sylvain Lebresne, reviewed by Benjamin Lerer for CASSANDRA-12126
---
 CHANGES.txt|   1 +
 NEWS.txt   |   8 +
 .../org/apache/cassandra/service/StorageProxy.java | 231 +--
 .../org/apache/cassandra/service/paxos/Commit.java |   6 +
 .../apache/cassandra/service/paxos/PaxosState.java |   3 -
 .../cassandra/service/paxos/PrepareCallback.java   |  12 +-
 .../cassandra/distributed/impl/Instance.java   | 112 ++--
 .../apache/cassandra/distributed/test/CASTest.java | 679 +
 8 files changed, 939 insertions(+), 113 deletions(-)

diff --git a/CHANGES.txt b/CHANGES.txt
index 546fd98..d6f406d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.24:
+ * Fix serial read/non-applying CAS linearizability (CASSANDRA-12126)
  * Avoid potential NPE in JVMStabilityInspector (CASSANDRA-16294)
  * Improved check of num_tokens against the length of initial_token 
(CASSANDRA-14477)
  * Fix a race condition on ColumnFamilyStore and TableMetrics (CASSANDRA-16228)
diff --git a/NEWS.txt b/NEWS.txt
index 7034c2c..6cc5e84 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -47,6 +47,14 @@ using the provided 'sstableupgrade' tool.
 
 Upgrading
 -
+- This release fix a correctness issue with SERIAL reads, and LWT writes 
that do not apply.
+  Unfortunately, this fix has a performance impact on read performance at 
the SERIAL or
+  LOCAL_SERIAL consistency levels. For heavy users of such SERIAL reads, 
the performance
+  impact may be noticeable and may also result in an increased of 
timeouts. For that
+  reason, a opt-in system property has been added to disable the fix:
+-Dcassandra.unsafe.disable-serial-reads-linearizability=true
+  Use this flag at your own risk as it revert SERIAL reads to the 
incorrect behavior of
+  previous versions. See CASSANDRA-12126 for details.
 - In cassandra.yaml, when using vnodes num_tokens must be defined if 
initial_token is defined.
   If it is not defined, or not equal to the numbers of tokens defined in 
initial_tokens,
   the node will not start. See CASSANDRA-14477 for details.
diff --git a/src/java/org/apache/cassandra/service/StorageProxy.java 
b/src/java/org/apache/cassandra/service/StorageProxy.java
index c7888c4..91dd991 100644
--- a/src/java/org/apache/cassandra/service/StorageProxy.java
+++ b/src/java/org/apache/cassandra/service/StorageProxy.java
@@ -24,6 +24,7 @@ import java.util.*;
 import java.util.concurrent.*;
 import java.util.concurrent.atomic.AtomicInteger;
 import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Supplier;
 
 import com.google.common.base.Predicate;
 import com.google.common.annotations.VisibleForTesting;
@@ -109,6 +110,10 @@ public class StorageProxy implements StorageProxyMBean
  */
 private static final int MAX_CONCURRENT_RANGE_REQUESTS = Math.max(1, 
Integer.getInteger("cassandra.max_concurrent_range_requests", 
FBUtilities.getAvailableProcessors() * 10));
 
+private static final String DISABLE_SERIAL_READ_LINEARIZABILITY_KEY = 
"cassandra.unsafe.disable-serial-reads-linearizability";
+private static final boolean disableSerialReadLinearizability =
+
Boolean.parseBoolean(System.getProperty(DISABLE_SERIAL_READ_LINEARIZABILITY_KEY,
 "false"));
+
 private StorageProxy()
 {
 }
@@ -163,6 +168,16 @@ public class StorageProxy implements StorageProxyMBean

[cassandra] 01/01: Merge branch 'cassandra-3.0' into cassandra-3.11

2020-11-02 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 833ba83c155871247092d6783d026c27582cde7b
Merge: a32881e f106ef0
Author: Sylvain Lebresne 
AuthorDate: Mon Nov 2 11:37:43 2020 +0100

Merge branch 'cassandra-3.0' into cassandra-3.11

 CHANGES.txt |  2 ++
 .../apache/cassandra/cql3/statements/SelectStatement.java   | 13 ++---
 .../cassandra/cql3/validation/ThriftIntegrationTest.java|  4 ++--
 3 files changed, 14 insertions(+), 5 deletions(-)

diff --cc CHANGES.txt
index b4caada,80f5d58..2501fcb
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,11 -1,5 +1,13 @@@
 +3.11.9
 + * Synchronize Keyspace instance store/clear (CASSANDRA-16210)
 + * Fix ColumnFilter to avoid querying cells of unselected complex columns 
(CASSANDRA-15977)
 + * Fix memory leak in CompressedChunkReader (CASSANDRA-15880)
 + * Don't attempt value skipping with mixed version cluster (CASSANDRA-15833)
 + * Avoid failing compactions with very large partitions (CASSANDRA-15164)
 + * Make sure LCS handles duplicate sstable added/removed notifications 
correctly (CASSANDRA-14103)
 +Merged from 3.0:
+ 3.0.23:
+  * Fix invalid cell value skipping when reading from disk (CASSANDRA-16223)
   * Prevent invoking enable/disable gossip when not in NORMAL (CASSANDRA-16146)
   * Fix OOM when terminating repair session (CASSANDRA-15902)
   * Avoid marking shutting down nodes as up after receiving gossip shutdown 
message (CASSANDRA-16094)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch cassandra-3.11 updated (a32881e -> 833ba83)

2020-11-02 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a change to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from a32881e  Merge branch 'cassandra-3.0' into cassandra-3.11
 new f106ef0  Fix invalid cell value skipping when reading from disk
 new 833ba83  Merge branch 'cassandra-3.0' into cassandra-3.11

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt |  2 ++
 .../apache/cassandra/cql3/statements/SelectStatement.java   | 13 ++---
 .../cassandra/cql3/validation/ThriftIntegrationTest.java|  4 ++--
 3 files changed, 14 insertions(+), 5 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch trunk updated (17cf2e7 -> cc28eba)

2020-11-02 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from 17cf2e7  Merge branch 'cassandra-3.11' into trunk
 new f106ef0  Fix invalid cell value skipping when reading from disk
 new cc28eba  Merge commit 'f106ef0697e172492b0343462c593edb703f2ac8' into 
trunk

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 01/01: Merge commit 'f106ef0697e172492b0343462c593edb703f2ac8' into trunk

2020-11-02 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit cc28eba205341396bf009095ce04f6e3c953e1bd
Merge: 17cf2e7 f106ef0
Author: Sylvain Lebresne 
AuthorDate: Mon Nov 2 11:39:14 2020 +0100

Merge commit 'f106ef0697e172492b0343462c593edb703f2ac8' into trunk



-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch cassandra-3.0 updated: Fix invalid cell value skipping when reading from disk

2020-11-02 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/cassandra-3.0 by this push:
 new f106ef0  Fix invalid cell value skipping when reading from disk
f106ef0 is described below

commit f106ef0697e172492b0343462c593edb703f2ac8
Author: jacek-lewandowski 
AuthorDate: Thu Oct 22 22:36:24 2020 +0200

Fix invalid cell value skipping when reading from disk

patch by Jacek Lewandowski; reviewed by Sylvain Lebresne for CASSANDRA-16223
---
 CHANGES.txt |  1 +
 .../apache/cassandra/cql3/statements/SelectStatement.java   | 13 ++---
 .../cassandra/cql3/validation/ThriftIntegrationTest.java|  4 ++--
 3 files changed, 13 insertions(+), 5 deletions(-)

diff --git a/CHANGES.txt b/CHANGES.txt
index bfc0a81..80f5d58 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.23:
+ * Fix invalid cell value skipping when reading from disk (CASSANDRA-16223)
  * Prevent invoking enable/disable gossip when not in NORMAL (CASSANDRA-16146)
  * Fix OOM when terminating repair session (CASSANDRA-15902)
  * Avoid marking shutting down nodes as up after receiving gossip shutdown 
message (CASSANDRA-16094)
diff --git a/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java 
b/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java
index 30c4458..046cc93 100644
--- a/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java
@@ -514,12 +514,12 @@ public class SelectStatement implements CQLStatement
 // Note that we use the total limit for every key, which is 
potentially inefficient.
 // However, IN + LIMIT is not a very sensible choice.
 List commands = new 
ArrayList<>(keys.size());
+ColumnFilter columnFilter = createColumnFilter(options);
 for (ByteBuffer key : keys)
 {
 QueryProcessor.validateKey(key);
 DecoratedKey dk = cfm.decorateKey(ByteBufferUtil.clone(key));
-ColumnFilter cf = (cfm.isSuper() && cfm.isDense()) ? 
SuperColumnCompatibility.getColumnFilter(cfm, options, 
restrictions.getSuperColumnRestrictions()) : queriedColumns;
-commands.add(SinglePartitionReadCommand.create(cfm, nowInSec, cf, 
rowFilter, limit, dk, filter));
+commands.add(SinglePartitionReadCommand.create(cfm, nowInSec, 
columnFilter, rowFilter, limit, dk, filter));
 }
 
 return new SinglePartitionReadCommand.Group(commands, limit);
@@ -581,7 +581,7 @@ public class SelectStatement implements CQLStatement
 return ReadQuery.EMPTY;
 
 PartitionRangeReadCommand command =
-PartitionRangeReadCommand.create(false, cfm, nowInSec, 
queriedColumns, rowFilter, limit, new DataRange(keyBounds, 
clusteringIndexFilter));
+PartitionRangeReadCommand.create(false, cfm, nowInSec, 
createColumnFilter(options), rowFilter, limit, new DataRange(keyBounds, 
clusteringIndexFilter));
 
 // If there's a secondary index that the command can use, have it 
validate the request parameters.
 command.maybeValidateIndex();
@@ -870,6 +870,13 @@ public class SelectStatement implements CQLStatement
 Collections.sort(cqlRows.rows, orderingComparator);
 }
 
+private ColumnFilter createColumnFilter(QueryOptions options)
+{
+return (cfm.isSuper() && cfm.isDense())
+   ? SuperColumnCompatibility.getColumnFilter(cfm, options, 
restrictions.getSuperColumnRestrictions())
+   : queriedColumns;
+}
+
 public static class RawStatement extends CFStatement
 {
 public final Parameters parameters;
diff --git 
a/test/unit/org/apache/cassandra/cql3/validation/ThriftIntegrationTest.java 
b/test/unit/org/apache/cassandra/cql3/validation/ThriftIntegrationTest.java
index def489e..7741b80 100644
--- a/test/unit/org/apache/cassandra/cql3/validation/ThriftIntegrationTest.java
+++ b/test/unit/org/apache/cassandra/cql3/validation/ThriftIntegrationTest.java
@@ -936,7 +936,7 @@ public class ThriftIntegrationTest extends ThriftCQLTester
 
 private void flushAll()
 {
-for (String cfName : new String[]{ currentTable(), 
currentSparseTable(), currentCounterTable() })
-Keyspace.open(KEYSPACE).getColumnFamilyStore(cfName);
+for (String cfName : new String[]{ currentTable(), 
currentSparseTable(), currentCounterTable(), currentDenseTable() })
+
Keyspace.open(KEYSPACE).getColumnFamilyStore(cfName).forceBlockingFlush();
 }
 }


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk

2020-08-17 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit efce6b39fb557314fad0cb56b07a1919d26b84fc
Merge: 139ad44 ecd23f1
Author: Sylvain Lebresne 
AuthorDate: Mon Aug 17 11:39:24 2020 +0200

Merge branch 'cassandra-3.11' into trunk

 CHANGES.txt|  1 +
 .../cassandra/db/SinglePartitionReadCommand.java   | 25 --
 .../db/compaction/AbstractCompactionStrategy.java  |  5 -
 .../db/compaction/CompactionStrategyManager.java   |  7 --
 .../compaction/SizeTieredCompactionStrategy.java   |  6 --
 5 files changed, 1 insertion(+), 43 deletions(-)

diff --cc CHANGES.txt
index 58239c9,a6bc9d9..7298e0c
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,20 -1,14 +1,21 @@@
 -3.11.8
 +4.0-beta2
 + * Make TimestampSerializer accept fractional seconds of varying precision 
(CASSANDRA-15976)
 + * Improve cassandra-stress logging when using a profile file that doesn't 
exist (CASSANDRA-14425)
 + * Improve logging for socket connection/disconnection (CASSANDRA-15980)
 + * Throw FSWriteError upon write failures in order to apply DiskFailurePolicy 
(CASSANDRA-15928)
 + * Forbid altering UDTs used in partition keys (CASSANDRA-15933)
 + * Fix version parsing logic when upgrading from 3.0 (CASSANDRA-15973)
 + * Optimize NoSpamLogger use in hot paths (CASSANDRA-15766)
 + * Verify sstable components on startup (CASSANDRA-15945)
 +Merged from 3.11:
   * Fix short read protection for GROUP BY queries (CASSANDRA-15459)
 + * stop_paranoid disk failure policy is ignored on CorruptSSTableException 
after node is up (CASSANDRA-15191)
   * Frozen RawTuple is not annotated with frozen in the toString method 
(CASSANDRA-15857)
  Merged from 3.0:
+  * Remove broken 'defrag-on-read' optimization (CASSANDRA-15432)
   * Check for endpoint collision with hibernating nodes (CASSANDRA-14599)
   * Operational improvements and hardening for replica filtering protection 
(CASSANDRA-15907)
 - * stop_paranoid disk failure policy is ignored on CorruptSSTableException 
after node is up (CASSANDRA-15191)
 - * Forbid altering UDTs used in partition keys (CASSANDRA-15933)
   * Fix empty/null json string representation (CASSANDRA-15896)
 - * 3.x fails to start if commit log has range tombstones from a column which 
is also deleted (CASSANDRA-15970)
  Merged from 2.2:
   * Fix CQL parsing of collections when the column type is reversed 
(CASSANDRA-15814)
  
diff --cc src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
index 6fbe523,c5de444..e581be5
--- a/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
+++ b/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
@@@ -28,10 -28,15 +28,9 @@@ import com.google.common.collect.Sets
  import org.apache.cassandra.cache.IRowCacheEntry;
  import org.apache.cassandra.cache.RowCacheKey;
  import org.apache.cassandra.cache.RowCacheSentinel;
--import org.apache.cassandra.concurrent.Stage;
 -import org.apache.cassandra.concurrent.StageManager;
 -import org.apache.cassandra.config.CFMetaData;
 -import org.apache.cassandra.config.ColumnDefinition;
  import org.apache.cassandra.config.DatabaseDescriptor;
 -import org.apache.cassandra.db.lifecycle.*;
  import org.apache.cassandra.db.filter.*;
 +import org.apache.cassandra.db.lifecycle.*;
  import org.apache.cassandra.db.partitions.*;
  import org.apache.cassandra.db.rows.*;
  import org.apache.cassandra.db.transform.RTBoundValidator;
@@@ -889,11 -992,8 +887,8 @@@ public class SinglePartitionReadComman
  if (iter.isEmpty())
  continue;
  
- if (sstable.isRepaired())
- onlyUnrepaired = false;
- 
  result = add(
 -RTBoundValidator.validate(isForThrift() ? 
ThriftResultsMerger.maybeWrap(iter, nowInSec()) : iter, 
RTBoundValidator.Stage.SSTABLE, false),
 +RTBoundValidator.validate(iter, 
RTBoundValidator.Stage.SSTABLE, false),
  result,
  filter,
  sstable.isRepaired()
@@@ -907,29 -1007,9 +902,9 @@@
  return EmptyIterators.unfilteredRow(metadata(), partitionKey(), 
false);
  
  DecoratedKey key = result.partitionKey();
 -
cfs.metric.samplers.get(TableMetrics.Sampler.READS).addSample(key.getKey(), 
key.hashCode(), 1);
 -StorageHook.instance.reportRead(cfs.metadata.cfId, partitionKey());
 +cfs.metric.topReadPartitionFrequency.addSample(key.getKey(), 1);
 +StorageHook.instance.reportRead(cfs.metadata.id, partitionKey());
  
- // "hoist up" the requested data into a more recent sstable
- if (metricsCollector.getMergedSSTables() > 
cfs.getMinimumCompactionThreshold()
- && onlyUnrepaired
- && !cfs.isAutoCompactionDisabled()
- &&

[cassandra] branch trunk updated (139ad44 -> efce6b3)

2020-08-17 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from 139ad44  Update docs regarding timestamp precision
 new e2ecdf2  Remove broken "defragment-on-read" optimization
 new ecd23f1  Merge commit 'e2ecdf268a82fa3ac0f4c9fe77ab35bca33cc72a' into 
cassandra-3.11
 new efce6b3  Merge branch 'cassandra-3.11' into trunk

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|  1 +
 .../cassandra/db/SinglePartitionReadCommand.java   | 25 --
 .../db/compaction/AbstractCompactionStrategy.java  |  5 -
 .../db/compaction/CompactionStrategyManager.java   |  7 --
 .../compaction/SizeTieredCompactionStrategy.java   |  6 --
 5 files changed, 1 insertion(+), 43 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch cassandra-3.11 updated (2beebbb -> ecd23f1)

2020-08-17 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a change to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from 2beebbb  Merge branch 'cassandra-3.0' into cassandra-3.11
 new e2ecdf2  Remove broken "defragment-on-read" optimization
 new ecd23f1  Merge commit 'e2ecdf268a82fa3ac0f4c9fe77ab35bca33cc72a' into 
cassandra-3.11

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|  1 +
 .../cassandra/db/SinglePartitionReadCommand.java   | 24 --
 .../db/compaction/AbstractCompactionStrategy.java  |  5 -
 .../db/compaction/CompactionStrategyManager.java   |  7 ---
 .../compaction/SizeTieredCompactionStrategy.java   |  6 --
 5 files changed, 1 insertion(+), 42 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 01/01: Merge commit 'e2ecdf268a82fa3ac0f4c9fe77ab35bca33cc72a' into cassandra-3.11

2020-08-17 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit ecd23f1da5894511cccac6c8445f962f3b73f733
Merge: 2beebbb e2ecdf2
Author: Sylvain Lebresne 
AuthorDate: Mon Aug 17 11:32:46 2020 +0200

Merge commit 'e2ecdf268a82fa3ac0f4c9fe77ab35bca33cc72a' into cassandra-3.11

 CHANGES.txt|  1 +
 .../cassandra/db/SinglePartitionReadCommand.java   | 24 --
 .../db/compaction/AbstractCompactionStrategy.java  |  5 -
 .../db/compaction/CompactionStrategyManager.java   |  7 ---
 .../compaction/SizeTieredCompactionStrategy.java   |  6 --
 5 files changed, 1 insertion(+), 42 deletions(-)

diff --cc CHANGES.txt
index 5168acb,9b4f8c3..a6bc9d9
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,7 -1,5 +1,8 @@@
 -3.0.22:
 +3.11.8
 + * Fix short read protection for GROUP BY queries (CASSANDRA-15459)
 + * Frozen RawTuple is not annotated with frozen in the toString method 
(CASSANDRA-15857)
 +Merged from 3.0:
+  * Remove broken 'defrag-on-read' optimization (CASSANDRA-15432)
   * Check for endpoint collision with hibernating nodes (CASSANDRA-14599)
   * Operational improvements and hardening for replica filtering protection 
(CASSANDRA-15907)
   * stop_paranoid disk failure policy is ignored on CorruptSSTableException 
after node is up (CASSANDRA-15191)
diff --cc src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
index a820a89,ca4e8e3..c5de444
--- a/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
+++ b/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
@@@ -918,8 -882,7 +918,7 @@@ public class SinglePartitionReadComman
  }
  
  /* add the SSTables on disk */
 -Collections.sort(view.sstables, SSTableReader.maxTimestampComparator);
 +Collections.sort(view.sstables, SSTableReader.maxTimestampDescending);
- boolean onlyUnrepaired = true;
  // read sorted sstables
  SSTableReadMetricsCollector metricsCollector = new 
SSTableReadMetricsCollector();
  for (SSTableReader sstable : view.sstables)
@@@ -1012,28 -967,7 +1008,8 @@@
  
  DecoratedKey key = result.partitionKey();
  
cfs.metric.samplers.get(TableMetrics.Sampler.READS).addSample(key.getKey(), 
key.hashCode(), 1);
 +StorageHook.instance.reportRead(cfs.metadata.cfId, partitionKey());
  
- // "hoist up" the requested data into a more recent sstable
- if (metricsCollector.getMergedSSTables() > 
cfs.getMinimumCompactionThreshold()
- && onlyUnrepaired
- && !cfs.isAutoCompactionDisabled()
- && cfs.getCompactionStrategyManager().shouldDefragment())
- {
- // !!WARNING!!   if we stop copying our data to a heap-managed 
object,
- //   we will need to track the lifetime of this 
mutation as well
- Tracing.trace("Defragmenting requested data");
- 
- try (UnfilteredRowIterator iter = 
result.unfilteredIterator(columnFilter(), Slices.ALL, false))
- {
- final Mutation mutation = new 
Mutation(PartitionUpdate.fromIterator(iter, columnFilter()));
- StageManager.getStage(Stage.MUTATION).execute(() -> {
- // skipping commitlog and index updates is fine since 
we're just de-fragmenting existing data
- Keyspace.open(mutation.getKeyspaceName()).apply(mutation, 
false, false);
- });
- }
- }
- 
  return result.unfilteredIterator(columnFilter(), Slices.ALL, 
clusteringIndexFilter().isReversed());
  }
  
diff --cc 
src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java
index 86170a1,5d4a254..d486679
--- a/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java
@@@ -102,11 -64,8 +102,10 @@@ public class CompactionStrategyManager 
  
  If a user changes the local compaction strategy and then later ALTERs 
a compaction parameter,
  we will use the new compaction parameters.
 - */
 -private CompactionParams schemaCompactionParams;
 + **/
 +private volatile CompactionParams schemaCompactionParams;
- private boolean shouldDefragment;
 +private boolean supportsEarlyOpen;
 +private int fanout;
  
  public CompactionStrategyManager(ColumnFamilyStore cfs)
  {
@@@ -206,28 -129,13 +205,27 @@@
  
  private void startup()
  {
 -for (SSTableReader sstable : cfs.getSSTables(SSTableSet.CANONICAL))
 +writeLock.lock();
 +try
  {
 -if (sstable.openReason != SSTableReader.OpenReason.EARLY)
 -getCompactionStrategyFor(sstable).addSSTable(sstable);
 +

[cassandra] branch cassandra-3.0 updated: Remove broken "defragment-on-read" optimization

2020-08-17 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/cassandra-3.0 by this push:
 new e2ecdf2  Remove broken "defragment-on-read" optimization
e2ecdf2 is described below

commit e2ecdf268a82fa3ac0f4c9fe77ab35bca33cc72a
Author: Sylvain Lebresne 
AuthorDate: Wed Aug 12 16:29:43 2020 +0200

Remove broken "defragment-on-read" optimization

The read path for names queries has had a "defragment-on-read"
optimization for a while whereby if too many sstables are hit by the
read, the result is written back into memtable, in the hope that later
reads will only read that newly written data in a single sstable (or at
least fewer).

The principle of that optimisation does not work however as data is
written back with the same timestamp as it originally has and that means
future reads cannot know to skip older sstables (at least with the
metadata we currently store).

As such, this optimisation never saved anything and in fact added load.

The patch removes that broken code.

Patch by Sylvain Lebresne, reviewed by Aleksey Yeschenko for
CASSANDRA-15432
---
 CHANGES.txt|  1 +
 .../cassandra/db/SinglePartitionReadCommand.java   | 28 --
 .../db/compaction/AbstractCompactionStrategy.java  |  5 
 .../db/compaction/CompactionStrategyManager.java   |  6 -
 .../compaction/SizeTieredCompactionStrategy.java   |  6 -
 5 files changed, 1 insertion(+), 45 deletions(-)

diff --git a/CHANGES.txt b/CHANGES.txt
index 7d4b7a9..9b4f8c3 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.22:
+ * Remove broken 'defrag-on-read' optimization (CASSANDRA-15432)
  * Check for endpoint collision with hibernating nodes (CASSANDRA-14599)
  * Operational improvements and hardening for replica filtering protection 
(CASSANDRA-15907)
  * stop_paranoid disk failure policy is ignored on CorruptSSTableException 
after node is up (CASSANDRA-15191)
diff --git a/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java 
b/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
index 841c3b9..ca4e8e3 100644
--- a/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
+++ b/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
@@ -883,7 +883,6 @@ public class SinglePartitionReadCommand extends ReadCommand
 
 /* add the SSTables on disk */
 Collections.sort(view.sstables, SSTableReader.maxTimestampComparator);
-boolean onlyUnrepaired = true;
 // read sorted sstables
 SSTableReadMetricsCollector metricsCollector = new 
SSTableReadMetricsCollector();
 for (SSTableReader sstable : view.sstables)
@@ -952,9 +951,6 @@ public class SinglePartitionReadCommand extends ReadCommand
 if (iter.isEmpty())
 continue;
 
-if (sstable.isRepaired())
-onlyUnrepaired = false;
-
 result = add(
 RTBoundValidator.validate(isForThrift() ? 
ThriftResultsMerger.maybeWrap(iter, nowInSec()) : iter, 
RTBoundValidator.Stage.SSTABLE, false),
 result,
@@ -972,30 +968,6 @@ public class SinglePartitionReadCommand extends ReadCommand
 DecoratedKey key = result.partitionKey();
 
cfs.metric.samplers.get(TableMetrics.Sampler.READS).addSample(key.getKey(), 
key.hashCode(), 1);
 
-// "hoist up" the requested data into a more recent sstable
-if (metricsCollector.getMergedSSTables() > 
cfs.getMinimumCompactionThreshold()
-&& onlyUnrepaired
-&& !cfs.isAutoCompactionDisabled()
-&& cfs.getCompactionStrategyManager().shouldDefragment())
-{
-// !!WARNING!!   if we stop copying our data to a heap-managed 
object,
-//   we will need to track the lifetime of this 
mutation as well
-Tracing.trace("Defragmenting requested data");
-
-try (UnfilteredRowIterator iter = 
result.unfilteredIterator(columnFilter(), Slices.ALL, false))
-{
-final Mutation mutation = new 
Mutation(PartitionUpdate.fromIterator(iter));
-StageManager.getStage(Stage.MUTATION).execute(new Runnable()
-{
-public void run()
-{
-// skipping commitlog and index updates is fine since 
we're just de-fragmenting existing data
-
Keyspace.open(mutation.getKeyspaceName()).apply(mutation, false, false);
-}
-});
-}
-}
-
 return result.unfilteredIterator(columnFilter(), Slices.ALL, 
clus

[cassandra-dtest] branch master updated: Add test for the upgrade of KEYS 2i to 4.0 (CASSANDRA-15906)

2020-08-12 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-dtest.git


The following commit(s) were added to refs/heads/master by this push:
 new f25832a  Add test for the upgrade of KEYS 2i to 4.0 (CASSANDRA-15906)
f25832a is described below

commit f25832aef72d62e9f00a45c54b2cdc17dc9f2d7b
Author: Sylvain Lebresne 
AuthorDate: Thu Jun 25 17:52:14 2020 +0200

Add test for the upgrade of KEYS 2i to 4.0 (CASSANDRA-15906)
---
 upgrade_tests/thrift_upgrade_test.py | 195 ++-
 1 file changed, 190 insertions(+), 5 deletions(-)

diff --git a/upgrade_tests/thrift_upgrade_test.py 
b/upgrade_tests/thrift_upgrade_test.py
index b427660..8846625 100644
--- a/upgrade_tests/thrift_upgrade_test.py
+++ b/upgrade_tests/thrift_upgrade_test.py
@@ -2,15 +2,17 @@ import itertools
 import pytest
 import logging
 
-from cassandra.query import dict_factory
+from cassandra.query import dict_factory, SimpleStatement
 
 from dtest import RUN_STATIC_UPGRADE_MATRIX, Tester
 from thrift_bindings.thrift010 import Cassandra
 from thrift_bindings.thrift010.Cassandra import (Column, ColumnDef,
-   ColumnParent, ConsistencyLevel,
-   SlicePredicate, SliceRange)
+ ColumnParent, 
ConsistencyLevel,
+ IndexType,
+ SlicePredicate, SliceRange)
 from thrift_test import _i64, get_thrift_client
-from tools.assertions import assert_length_equal, assert_lists_of_dicts_equal
+from tools.assertions import (assert_all, assert_length_equal,
+  assert_lists_of_dicts_equal)
 from tools.misc import wait_for_agreement, add_skip
 from .upgrade_base import UpgradeTester
 from .upgrade_manifest import build_upgrade_pairs
@@ -187,7 +189,6 @@ def _validate_dense_thrift(client, cf='dense_super_1'):
 assert cosc.super_column.columns[0].name == _i64(100)
 assert cosc.super_column.columns[0].value == 'value1'.encode()
 
-
 @pytest.mark.upgrade_test
 class TestUpgradeSuperColumnsThrough(Tester):
 def upgrade_to_version(self, tag, nodes=None):
@@ -373,6 +374,190 @@ class TestUpgradeSuperColumnsThrough(Tester):
 
 
 @pytest.mark.upgrade_test
+@since('4')
+class TestUpgradeTo40(Tester):
+"""
+Thrift is dead in 4.0. However, we still want to ensure users that used 
thrift
+ in 3.0 or earlier have an upgrade path to 4.0 and this class provides 
tests
+ cases for this.
+
+ Note that we don't want to run this if the "current" version (the one 
we're
+ upgrading to) is not 4.0 or more, as the tests makes assumptions on that.
+"""
+def prepare(self, start_version, num_nodes=1, rf=1):
+"""
+Prepare the test, starting a cluster on the initial version, creating
+a keyspace (named 'ks') and returning a CQL and a thrift connection to
+the first node (and set on the created keyspace).
+
+:param start_version: the version to set the node at initially.
+:param num_nodes: the number of nodes to use.
+:param rf: replication factor for the keyspace created.
+:return: a pair (cql, thrift) of a CQL connection and an open thrift
+connection to the first node in the cluster.
+"""
+self.cluster.set_install_dir(version=start_version)
+self.fixture_dtest_setup.reinitialize_cluster_for_different_version()
+
+self.cluster.populate(num_nodes)
+for node in self.cluster.nodelist():
+node.set_configuration_options(values={'start_rpc': 'true'})
+
+self.cluster.start()
+logger.debug("Started node on %s", start_version)
+
+node = self.cluster.nodelist()[0]
+cql = self.patient_cql_connection(node)
+
+cql.execute("CREATE KEYSPACE ks WITH replication = {{ 'class': 
'SimpleStrategy', 'replication_factor': '{}' }}".format(rf))
+cql.execute("USE ks")
+
+host, port = node.network_interfaces['thrift']
+thrift = get_thrift_client(host, port)
+thrift.transport.open()
+thrift.set_keyspace('ks')
+return cql, thrift
+
+def _connect(self, node):
+connection = self.patient_cql_connection(node)
+connection.execute("USE ks")
+return connection
+
+def _do_rolling_upgrade(self, after_each_upgrade=None):
+"""
+Upgrade all the nodes in the cluster to the "current" version (so 4.0+)
+in a rolling fashion.
+
+:param after_each_upgrade: if not None, a function that is called with 
2
+  arguments (in that order):
+  - the index of the node we just upgrade

[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk

2020-07-01 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit dc3705a517e70ea8e975d79bee1664e4abd7d779
Merge: eebb9e0 c5b2a37
Author: Sylvain Lebresne 
AuthorDate: Wed Jul 1 14:57:05 2020 +0200

Merge branch 'cassandra-3.11' into trunk



-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch trunk updated (eebb9e0 -> dc3705a)

2020-07-01 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from eebb9e0  Merge branch 'cassandra-3.11' into trunk
 new 64c80f4  Accepts 2i queries with only static columns for KEYS index
 new c5b2a37  Merge commit '64c80f4ef89f0cc88c15febed8c01eb07ae0a84e' into 
cassandra-3.11
 new dc3705a  Merge branch 'cassandra-3.11' into trunk

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 01/01: Merge commit '64c80f4ef89f0cc88c15febed8c01eb07ae0a84e' into cassandra-3.11

2020-07-01 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit c5b2a37904b5373bd7eb418865c9770b8cc9d505
Merge: 9251b81 64c80f4
Author: Sylvain Lebresne 
AuthorDate: Wed Jul 1 14:56:50 2020 +0200

Merge commit '64c80f4ef89f0cc88c15febed8c01eb07ae0a84e' into cassandra-3.11



-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch cassandra-3.11 updated (9251b81 -> c5b2a37)

2020-07-01 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a change to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from 9251b81  Fix cqlsh output when fetching all rows in batch mode
 new 64c80f4  Accepts 2i queries with only static columns for KEYS index
 new c5b2a37  Merge commit '64c80f4ef89f0cc88c15febed8c01eb07ae0a84e' into 
cassandra-3.11

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch cassandra-3.0 updated: Accepts 2i queries with only static columns for KEYS index

2020-07-01 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/cassandra-3.0 by this push:
 new 64c80f4  Accepts 2i queries with only static columns for KEYS index
64c80f4 is described below

commit 64c80f4ef89f0cc88c15febed8c01eb07ae0a84e
Author: Sylvain Lebresne 
AuthorDate: Mon Jun 29 16:27:01 2020 +0200

Accepts 2i queries with only static columns for KEYS index

KEYS index can originally only be created on compact tables, which
don't have static columns. However, through DROP COMPACT STORAGE,
a previously compact table with a KEYS index will have its columns all
be static, and queries of the 2i KEYS was rejected.

This patch ensure the queries does went through in that case.

Patch by Sylvain Lebresne, reviewed by Brandon Williams for
  CASSANDRA-15906
---
 CHANGES.txt |  1 +
 .../cql3/restrictions/StatementRestrictions.java| 17 ++---
 2 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/CHANGES.txt b/CHANGES.txt
index 7e1a4fd..d94041b 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.21
+ * Fix broken KEYS 2i queries after DROP COMPACT STORAGE (CASSANDRA-15906)
  * Add token to tombstone warning and error messages (CASSANDRA-15890)
  * Fixed range read concurrency factor computation and capped as 10 times tpc 
cores (CASSANDRA-15752)
  * Catch exception on bootstrap resume and init native transport 
(CASSANDRA-15863)
diff --git 
a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java 
b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
index 84c6958..d7d6f48 100644
--- a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
+++ b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
@@ -21,6 +21,7 @@ import java.nio.ByteBuffer;
 import java.util.*;
 
 import com.google.common.base.Joiner;
+import com.google.common.collect.Iterables;
 import com.google.common.collect.Iterators;
 
 import org.apache.cassandra.config.CFMetaData;
@@ -37,6 +38,7 @@ import 
org.apache.cassandra.exceptions.InvalidRequestException;
 import org.apache.cassandra.index.Index;
 import org.apache.cassandra.index.SecondaryIndexManager;
 import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.schema.IndexMetadata;
 import org.apache.cassandra.utils.ByteBufferUtil;
 import org.apache.cassandra.utils.btree.BTreeSet;
 
@@ -838,9 +840,18 @@ public final class StatementRestrictions
"Select on indexed columns and with IN clause for the 
PRIMARY KEY are not supported");
 // When the user only select static columns, the intent is that we 
don't query the whole partition but just
 // the static parts. But 1) we don't have an easy way to do that with 
2i and 2) since we don't support index on
-// static columns
-// so far, 2i means that you've restricted a non static column, so the 
query is somewhat non-sensical.
-checkFalse(selectsOnlyStaticColumns, "Queries using 2ndary indexes 
don't support selecting only static columns");
+// static columns so far, 2i means that you've restricted a non static 
column, so the query is somewhat
+// non-sensical.
+// Note: an exception is if the index is a KEYS one. Which can happen 
if the user had a KEYS index on
+// a compact table, and subsequently DROP COMPACT STORAGE on that 
table. After which, the KEYS index will still
+// work, but queries will effectively be only on now-static columns 
and we should let this work.
+checkFalse(selectsOnlyStaticColumns && !hasKeysIndex(cfm),
+   "Queries using 2ndary indexes don't support selecting only 
static columns");
+}
+
+private boolean hasKeysIndex(CFMetaData cfm)
+{
+return Iterables.any(cfm.getIndexes(), i -> i.kind == 
IndexMetadata.Kind.KEYS);
 }
 
 /**


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk

2020-05-27 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 44c4e1f7874c962309a853fa2434a9dd4ae16aeb
Merge: e690e29 ebfd052
Author: Sylvain Lebresne 
AuthorDate: Wed May 27 17:18:43 2020 +0200

Merge branch 'cassandra-3.11' into trunk



-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch trunk updated (e690e29 -> 44c4e1f)

2020-05-27 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from e690e29  Add docs section on configuring the Jenkins master to create 
the "Cassandra" category throttle.
 new 8358e19  Fix legacy handling of RangeTombstone with collection ones
 new c8a2834  Fix LegacyLayout handling of non-selected collection 
tombstones
 new ebfd052  Merge commit 'c8a2834606d683ba9945e9cc11bdb4207ce269d1' into 
cassandra-3.11
 new 44c4e1f  Merge branch 'cassandra-3.11' into trunk

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 01/01: Merge commit 'c8a2834606d683ba9945e9cc11bdb4207ce269d1' into cassandra-3.11

2020-05-27 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit ebfd05254f84000f71fa018650632d24d3761f07
Merge: 3cda9d7 c8a2834
Author: Sylvain Lebresne 
AuthorDate: Wed May 27 17:12:44 2020 +0200

Merge commit 'c8a2834606d683ba9945e9cc11bdb4207ce269d1' into cassandra-3.11

 CHANGES.txt|   1 +
 src/java/org/apache/cassandra/db/LegacyLayout.java | 105 +
 .../cassandra/db/UnfilteredDeserializer.java   | 129 -
 .../upgrade/MixedModeRangeTombstoneTest.java   |  73 
 .../org/apache/cassandra/db/LegacyLayoutTest.java  | 102 +---
 5 files changed, 340 insertions(+), 70 deletions(-)

diff --cc CHANGES.txt
index 11515c4,46b3f56..a809016
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,7 -1,5 +1,8 @@@
 -3.0.21
 +3.11.7
 + * Fix CQL formatting of read command restrictions for slow query log 
(CASSANDRA-15503)
 + * Allow sstableloader to use SSL on the native port (CASSANDRA-14904)
 +Merged from 3.0:
+  * Fix duplicated row on 2.x upgrades when multi-rows range tombstones 
interact with collection ones (CASSANDRA-15805)
   * Rely on snapshotted session infos on StreamResultFuture.maybeComplete to 
avoid race conditions (CASSANDRA-15667)
   * EmptyType doesn't override writeValue so could attempt to write bytes when 
expected not to (CASSANDRA-15790)
   * Fix index queries on partition key columns when some partitions contains 
only static data (CASSANDRA-13666)
diff --cc src/java/org/apache/cassandra/db/LegacyLayout.java
index 4ec0c30,8492de5..b28c72a
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@@ -1891,9 -1934,9 +1936,9 @@@ public abstract class LegacyLayou
  if ((start.collectionName == null) != (stop.collectionName == 
null))
  {
  if (start.collectionName == null)
- stop = new LegacyBound(stop.bound, stop.isStatic, null);
 -stop = new 
LegacyBound(Slice.Bound.inclusiveEndOf(stop.bound.values), stop.isStatic, null);
++stop = new 
LegacyBound(ClusteringBound.inclusiveEndOf(stop.bound.values), stop.isStatic, 
null);
  else
- start = new LegacyBound(start.bound, start.isStatic, 
null);
 -start = new 
LegacyBound(Slice.Bound.inclusiveStartOf(start.bound.values), start.isStatic, 
null);
++start = new 
LegacyBound(ClusteringBound.inclusiveStartOf(start.bound.values), 
start.isStatic, null);
  }
  else if (!Objects.equals(start.collectionName, 
stop.collectionName))
  {
@@@ -1920,11 -1963,21 +1965,21 @@@
  return new LegacyRangeTombstone(newStart, stop, deletionTime);
  }
  
 -public LegacyRangeTombstone withNewStart(Slice.Bound newStart)
++public LegacyRangeTombstone withNewStart(ClusteringBound newStart)
+ {
+ return withNewStart(new LegacyBound(newStart, start.isStatic, 
null));
+ }
+ 
  public LegacyRangeTombstone withNewEnd(LegacyBound newStop)
  {
  return new LegacyRangeTombstone(start, newStop, deletionTime);
  }
  
 -public LegacyRangeTombstone withNewEnd(Slice.Bound newEnd)
++public LegacyRangeTombstone withNewEnd(ClusteringBound newEnd)
+ {
+ return withNewEnd(new LegacyBound(newEnd, stop.isStatic, null));
+ }
+ 
  public boolean isCell()
  {
  return false;
diff --cc src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
index cdcde2e,2d270bc..262b333
--- a/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
+++ b/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
@@@ -480,19 -480,9 +480,10 @@@ public abstract class UnfilteredDeseria
  this.helper = helper;
  this.grouper = new LegacyLayout.CellGrouper(metadata, helper);
  this.tombstoneTracker = new 
TombstoneTracker(partitionDeletion);
- this.atoms = new AtomIterator(atomReader);
- }
- 
- private boolean isRow(LegacyLayout.LegacyAtom atom)
- {
- if (atom.isCell())
- return true;
- 
- LegacyLayout.LegacyRangeTombstone tombstone = 
atom.asRangeTombstone();
- return tombstone.isCollectionTombstone() || 
tombstone.isRowDeletion(metadata);
+ this.atoms = new AtomIterator(atomReader, metadata);
  }
  
 +
  public boolean hasNext()
  {
  // Note that we loop on next == null because 
TombstoneTracker.openNew() could return null below or the atom might be 
shadowed.
@@@ -540,13 -530,57 +531,57

[cassandra] branch cassandra-3.11 updated (3cda9d7 -> ebfd052)

2020-05-27 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a change to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from 3cda9d7  Merge branch cassandra-3.0 into cassandra-3.11
 new 8358e19  Fix legacy handling of RangeTombstone with collection ones
 new c8a2834  Fix LegacyLayout handling of non-selected collection 
tombstones
 new ebfd052  Merge commit 'c8a2834606d683ba9945e9cc11bdb4207ce269d1' into 
cassandra-3.11

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|   1 +
 src/java/org/apache/cassandra/db/LegacyLayout.java | 105 +
 .../cassandra/db/UnfilteredDeserializer.java   | 129 -
 .../upgrade/MixedModeRangeTombstoneTest.java   |  73 
 .../org/apache/cassandra/db/LegacyLayoutTest.java  | 102 +---
 5 files changed, 340 insertions(+), 70 deletions(-)
 create mode 100644 
test/distributed/org/apache/cassandra/distributed/upgrade/MixedModeRangeTombstoneTest.java


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 01/02: Fix legacy handling of RangeTombstone with collection ones

2020-05-27 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 8358e19840d352475a5831d130ff3c43a11f2f4e
Author: Sylvain Lebresne 
AuthorDate: Fri May 8 18:12:55 2020 +0200

Fix legacy handling of RangeTombstone with collection ones

When a multi-row range tombstone interacts with a a collection tombstone
within one of a covered row, the resulting range tombstone in the legacy
format will start in the middle of the row and extend past said row and
it needs special handling.

Before this commit, the code deserializing that RT was making it
artificially start at the end of the row (in which the collection
tombstone is), but that means that when `LegacyLayout.CellGrouper`
encountered it, it decided the row was finished, even if it was not,
leading to potential row duplication.

The patch solves this by:
1. making that problematic tombstone start at the beginning of the row
instead of its end (to avoid code deciding the row is over).
2. modify `UnfilteredDeserializer` to 'split' that range tombstone into
a row tombstone for the row it covers, which is handled as a normal row
tombstone, and push the rest of the range tombstone (that starts after
the row and extends to the original end of the RT) to be handled after
that row is fully "grouped".

The patch also removes the possibility of getting an empty row from
`LegacyLayout#getNextRow` to avoid theoretical problems with that.

Patch by Sylvain Lebresne; reviewed by Marcus Eriksson & Aleksey
Yeschenko for CASSANDRA-15805
---
 CHANGES.txt|   1 +
 src/java/org/apache/cassandra/db/LegacyLayout.java |  99 
 .../cassandra/db/UnfilteredDeserializer.java   | 129 -
 .../upgrade/MixedModeRangeTombstoneTest.java   |  73 
 4 files changed, 252 insertions(+), 50 deletions(-)

diff --git a/CHANGES.txt b/CHANGES.txt
index cdb9ad0..46b3f56 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.21
+ * Fix duplicated row on 2.x upgrades when multi-rows range tombstones 
interact with collection ones (CASSANDRA-15805)
  * Rely on snapshotted session infos on StreamResultFuture.maybeComplete to 
avoid race conditions (CASSANDRA-15667)
  * EmptyType doesn't override writeValue so could attempt to write bytes when 
expected not to (CASSANDRA-15790)
  * Fix index queries on partition key columns when some partitions contains 
only static data (CASSANDRA-13666)
diff --git a/src/java/org/apache/cassandra/db/LegacyLayout.java 
b/src/java/org/apache/cassandra/db/LegacyLayout.java
index 37cc935..39dd54a 100644
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@ -1115,7 +1115,7 @@ public abstract class LegacyLayout
 return true;
 }
 
-private static Comparator legacyAtomComparator(CFMetaData 
metadata)
+static Comparator legacyAtomComparator(CFMetaData metadata)
 {
 return (o1, o2) ->
 {
@@ -1373,8 +1373,24 @@ public abstract class LegacyLayout
 this.hasValidCells = false;
 }
 
+/**
+ * Try adding the provided atom to the currently grouped row.
+ *
+ * @param atom the new atom to try to add. This must be a "row" 
atom, that is either a cell or a legacy
+ * range tombstone that covers only one row (row deletion) 
or a subset of it (collection
+ * deletion). Meaning that legacy range tombstone covering 
multiple rows (that should be handled as
+ * legit range tombstone in the new storage engine) should 
be handled separately. Atoms should also
+ * be provided in proper clustering order.
+ * @return {@code true} if the provided atom has been "consumed" by 
this grouper (this does _not_ mean the
+ *  atom has been "used" by the grouper as the grouper will 
skip some shadowed atoms for instance, just
+ *  that {@link #getRow()} shouldn't be called just yet if 
there is more atom in the atom iterator we're
+ *  grouping). {@code false} otherwise, that is if the row 
currently built by this grouper is done
+ *  _without_ the provided atom being "consumed" (and so 
{@link #getRow()} should be called and the
+ *  grouper resetted, after which the provided atom should be 
provided again).
+ */
 public boolean addAtom(LegacyAtom atom)
 {
+assert atom.isRowAtom(metadata) : "Unexpected non in-row legacy 
range tombstone " + atom;
 return atom.isCell()
  ? addCell(atom.asCell())
  : addRangeTombstone(atom

[cassandra] 02/02: Fix LegacyLayout handling of non-selected collection tombstones

2020-05-27 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit c8a2834606d683ba9945e9cc11bdb4207ce269d1
Author: Sylvain Lebresne 
AuthorDate: Wed May 13 11:44:08 2020 +0200

Fix LegacyLayout handling of non-selected collection tombstones

If a collection tombstone is not included by a query, it can be ignored,
but it currently made `LegacyLayout.CellGrouper#addCollectionTombstone`
return `false`, which made it stop the current row, which is incorrect
(this can potentially lead to a duplicate row). This patch changes it
to return `true`.

Patch by Sylvain Lebresne; reviewed by Marcus Eriksson & Aleksey
Yeschenko for CASSANDRA-15805
---
 src/java/org/apache/cassandra/db/LegacyLayout.java |   6 +-
 .../org/apache/cassandra/db/LegacyLayoutTest.java  | 102 +
 2 files changed, 88 insertions(+), 20 deletions(-)

diff --git a/src/java/org/apache/cassandra/db/LegacyLayout.java 
b/src/java/org/apache/cassandra/db/LegacyLayout.java
index 39dd54a..8492de5 100644
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@ -1537,8 +1537,12 @@ public abstract class LegacyLayout
 
 private boolean addCollectionTombstone(LegacyRangeTombstone tombstone)
 {
+// If the collection tombstone is not included in the query (which 
technically would only apply to thrift
+// queries since CQL one "fetch" everything), we can skip it (so 
return), but we're problably still within
+// the current row so we return `true`. Technically, it is 
possible that tombstone belongs to another row
+// that the row currently grouped, but as we ignore it, returning 
`true` is ok in that case too.
 if (!helper.includes(tombstone.start.collectionName))
-return false; // see CASSANDRA-13109
+return true; // see CASSANDRA-13109
 
 // The helper needs to be informed about the current complex 
column identifier before
 // it can perform the comparison between the recorded drop time 
and the RT deletion time.
diff --git a/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java 
b/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java
index 0bb2459..f0d2a02 100644
--- a/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java
+++ b/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java
@@ -24,18 +24,19 @@ import java.nio.file.Files;
 import java.nio.file.Path;
 import java.nio.file.Paths;
 
+import org.apache.cassandra.db.LegacyLayout.CellGrouper;
+import org.apache.cassandra.db.LegacyLayout.LegacyBound;
+import org.apache.cassandra.db.LegacyLayout.LegacyCell;
+import org.apache.cassandra.db.LegacyLayout.LegacyRangeTombstone;
 import org.apache.cassandra.db.filter.ColumnFilter;
 import org.apache.cassandra.db.marshal.MapType;
 import org.apache.cassandra.db.marshal.UTF8Type;
-import org.apache.cassandra.db.partitions.ImmutableBTreePartition;
 import org.apache.cassandra.db.rows.BufferCell;
 import org.apache.cassandra.db.rows.Cell;
 import org.apache.cassandra.db.rows.RowIterator;
-import org.apache.cassandra.db.rows.Rows;
 import org.apache.cassandra.db.rows.SerializationHelper;
 import org.apache.cassandra.db.rows.UnfilteredRowIterator;
 import org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer;
-import org.apache.cassandra.db.rows.UnfilteredRowIterators;
 import org.apache.cassandra.db.transform.FilteredRows;
 import org.apache.cassandra.exceptions.ConfigurationException;
 import org.apache.cassandra.io.util.DataInputBuffer;
@@ -62,10 +63,10 @@ import org.apache.cassandra.db.rows.BTreeRow;
 import org.apache.cassandra.db.rows.Row;
 import org.apache.cassandra.dht.Murmur3Partitioner;
 import org.apache.cassandra.schema.KeyspaceParams;
-import org.apache.cassandra.utils.ByteBufferUtil;
 import org.apache.cassandra.utils.Hex;
 
 import static org.apache.cassandra.net.MessagingService.VERSION_21;
+import static org.apache.cassandra.utils.ByteBufferUtil.bytes;
 import static org.junit.Assert.*;
 
 public class LegacyLayoutTest
@@ -98,7 +99,7 @@ public class LegacyLayoutTest
 builder.addComplexDeletion(b, new DeletionTime(1L, 1));
 Row row = builder.build();
 
-ByteBuffer key = ByteBufferUtil.bytes(1);
+ByteBuffer key = bytes(1);
 PartitionUpdate upd = PartitionUpdate.singleRowUpdate(table, key, row);
 
 LegacyLayout.LegacyUnfilteredPartition p = 
LegacyLayout.fromUnfilteredRowIterator(null, upd.unfilteredIterator());
@@ -216,7 +217,7 @@ public class LegacyLayoutTest
 builder.addCell(new BufferCell(v, 1L, Cell.NO_TTL, 
Cell.NO_DELETION_TIME, Int32Serializer.instance.serialize(1), null));
 Row row = builder.build();
 
-DecoratedKey pk = table.decorateKey(ByteBufferUtil.bytes(1));
+D

[cassandra] branch cassandra-3.0 updated (a4b6deb -> c8a2834)

2020-05-27 Thread slebresne

This is an automated email from the ASF dual-hosted git repository.

slebresne pushed a change to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from a4b6deb  Rely on snapshotted session infos on 
StreamResultFuture.maybeComplete to avoid races
 new 8358e19  Fix legacy handling of RangeTombstone with collection ones
 new c8a2834  Fix LegacyLayout handling of non-selected collection 
tombstones

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|   1 +
 src/java/org/apache/cassandra/db/LegacyLayout.java | 105 +
 .../cassandra/db/UnfilteredDeserializer.java   | 129 -
 .../upgrade/MixedModeRangeTombstoneTest.java   |  73 
 .../org/apache/cassandra/db/LegacyLayoutTest.java  | 102 +---
 5 files changed, 340 insertions(+), 70 deletions(-)
 create mode 100644 
test/distributed/org/apache/cassandra/distributed/upgrade/MixedModeRangeTombstoneTest.java


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[4/6] cassandra git commit: Merge commit 'a4b8d1e9bc67eb0eff159b5b35902cab19e3fc11' into cassandra-3.11

2018-07-05 Thread slebresne

Merge commit 'a4b8d1e9bc67eb0eff159b5b35902cab19e3fc11' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/942fe3fb
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/942fe3fb
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/942fe3fb

Branch: refs/heads/trunk
Commit: 942fe3fb3d81b0f78164ab35c174130c2395a402
Parents: 49e63c2 a4b8d1e
Author: Sylvain Lebresne 
Authored: Thu Jul 5 09:27:51 2018 +0200
Committer: Sylvain Lebresne 
Committed: Thu Jul 5 09:27:51 2018 +0200

--

--



-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[5/6] cassandra git commit: Merge commit 'a4b8d1e9bc67eb0eff159b5b35902cab19e3fc11' into cassandra-3.11

2018-07-05 Thread slebresne

Merge commit 'a4b8d1e9bc67eb0eff159b5b35902cab19e3fc11' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/942fe3fb
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/942fe3fb
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/942fe3fb

Branch: refs/heads/cassandra-3.11
Commit: 942fe3fb3d81b0f78164ab35c174130c2395a402
Parents: 49e63c2 a4b8d1e
Author: Sylvain Lebresne 
Authored: Thu Jul 5 09:27:51 2018 +0200
Committer: Sylvain Lebresne 
Committed: Thu Jul 5 09:27:51 2018 +0200

--

--



-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[2/6] cassandra git commit: Fix build

2018-07-05 Thread slebresne

Fix build


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a4b8d1e9
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a4b8d1e9
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a4b8d1e9

Branch: refs/heads/cassandra-3.11
Commit: a4b8d1e9bc67eb0eff159b5b35902cab19e3fc11
Parents: 93012e4
Author: Sylvain Lebresne 
Authored: Thu Jul 5 09:24:17 2018 +0200
Committer: Sylvain Lebresne 
Committed: Thu Jul 5 09:24:17 2018 +0200

--
 test/unit/org/apache/cassandra/db/CounterCellTest.java | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a4b8d1e9/test/unit/org/apache/cassandra/db/CounterCellTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/CounterCellTest.java 
b/test/unit/org/apache/cassandra/db/CounterCellTest.java
index 8859fc6..b4c7b2a 100644
--- a/test/unit/org/apache/cassandra/db/CounterCellTest.java
+++ b/test/unit/org/apache/cassandra/db/CounterCellTest.java
@@ -32,7 +32,6 @@ import org.apache.cassandra.config.ColumnDefinition;
 import org.apache.cassandra.db.rows.BTreeRow;
 import org.apache.cassandra.db.rows.BufferCell;
 import org.apache.cassandra.db.rows.Cell;
-import org.apache.cassandra.db.rows.CellPath;
 import org.apache.cassandra.db.rows.Cells;
 import org.apache.cassandra.db.context.CounterContext;
 import org.apache.cassandra.db.rows.Row;
@@ -292,10 +291,10 @@ public class CounterCellTest
 ColumnFamilyStore cfs = 
Keyspace.open(KEYSPACE1).getColumnFamilyStore(COUNTER1);
 
 ColumnDefinition emptyColDef = 
cfs.metadata.getColumnDefinition(ByteBufferUtil.bytes("val2"));
-BufferCell emptyCell = BufferCell.live(emptyColDef, 0, 
ByteBuffer.allocate(0));
+BufferCell emptyCell = BufferCell.live(cfs.metadata, emptyColDef, 0, 
ByteBuffer.allocate(0));
 
 Row.Builder builder = BTreeRow.unsortedBuilder(0);
-
builder.newRow(Clustering.make(AsciiSerializer.instance.serialize("test")));
+builder.newRow(new 
Clustering(AsciiSerializer.instance.serialize("test")));
 builder.addCell(emptyCell);
 Row row = builder.build();
 


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[3/6] cassandra git commit: Fix build

2018-07-05 Thread slebresne

Fix build


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a4b8d1e9
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a4b8d1e9
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a4b8d1e9

Branch: refs/heads/trunk
Commit: a4b8d1e9bc67eb0eff159b5b35902cab19e3fc11
Parents: 93012e4
Author: Sylvain Lebresne 
Authored: Thu Jul 5 09:24:17 2018 +0200
Committer: Sylvain Lebresne 
Committed: Thu Jul 5 09:24:17 2018 +0200

--
 test/unit/org/apache/cassandra/db/CounterCellTest.java | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a4b8d1e9/test/unit/org/apache/cassandra/db/CounterCellTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/CounterCellTest.java 
b/test/unit/org/apache/cassandra/db/CounterCellTest.java
index 8859fc6..b4c7b2a 100644
--- a/test/unit/org/apache/cassandra/db/CounterCellTest.java
+++ b/test/unit/org/apache/cassandra/db/CounterCellTest.java
@@ -32,7 +32,6 @@ import org.apache.cassandra.config.ColumnDefinition;
 import org.apache.cassandra.db.rows.BTreeRow;
 import org.apache.cassandra.db.rows.BufferCell;
 import org.apache.cassandra.db.rows.Cell;
-import org.apache.cassandra.db.rows.CellPath;
 import org.apache.cassandra.db.rows.Cells;
 import org.apache.cassandra.db.context.CounterContext;
 import org.apache.cassandra.db.rows.Row;
@@ -292,10 +291,10 @@ public class CounterCellTest
 ColumnFamilyStore cfs = 
Keyspace.open(KEYSPACE1).getColumnFamilyStore(COUNTER1);
 
 ColumnDefinition emptyColDef = 
cfs.metadata.getColumnDefinition(ByteBufferUtil.bytes("val2"));
-BufferCell emptyCell = BufferCell.live(emptyColDef, 0, 
ByteBuffer.allocate(0));
+BufferCell emptyCell = BufferCell.live(cfs.metadata, emptyColDef, 0, 
ByteBuffer.allocate(0));
 
 Row.Builder builder = BTreeRow.unsortedBuilder(0);
-
builder.newRow(Clustering.make(AsciiSerializer.instance.serialize("test")));
+builder.newRow(new 
Clustering(AsciiSerializer.instance.serialize("test")));
 builder.addCell(emptyCell);
 Row row = builder.build();
 


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[1/6] cassandra git commit: Fix build

2018-07-05 Thread slebresne

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 93012e43e -> a4b8d1e9b
  refs/heads/cassandra-3.11 49e63c273 -> 942fe3fb3
  refs/heads/trunk 243c371f4 -> 9e3b88646


Fix build


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a4b8d1e9
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a4b8d1e9
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a4b8d1e9

Branch: refs/heads/cassandra-3.0
Commit: a4b8d1e9bc67eb0eff159b5b35902cab19e3fc11
Parents: 93012e4
Author: Sylvain Lebresne 
Authored: Thu Jul 5 09:24:17 2018 +0200
Committer: Sylvain Lebresne 
Committed: Thu Jul 5 09:24:17 2018 +0200

--
 test/unit/org/apache/cassandra/db/CounterCellTest.java | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a4b8d1e9/test/unit/org/apache/cassandra/db/CounterCellTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/CounterCellTest.java 
b/test/unit/org/apache/cassandra/db/CounterCellTest.java
index 8859fc6..b4c7b2a 100644
--- a/test/unit/org/apache/cassandra/db/CounterCellTest.java
+++ b/test/unit/org/apache/cassandra/db/CounterCellTest.java
@@ -32,7 +32,6 @@ import org.apache.cassandra.config.ColumnDefinition;
 import org.apache.cassandra.db.rows.BTreeRow;
 import org.apache.cassandra.db.rows.BufferCell;
 import org.apache.cassandra.db.rows.Cell;
-import org.apache.cassandra.db.rows.CellPath;
 import org.apache.cassandra.db.rows.Cells;
 import org.apache.cassandra.db.context.CounterContext;
 import org.apache.cassandra.db.rows.Row;
@@ -292,10 +291,10 @@ public class CounterCellTest
 ColumnFamilyStore cfs = 
Keyspace.open(KEYSPACE1).getColumnFamilyStore(COUNTER1);
 
 ColumnDefinition emptyColDef = 
cfs.metadata.getColumnDefinition(ByteBufferUtil.bytes("val2"));
-BufferCell emptyCell = BufferCell.live(emptyColDef, 0, 
ByteBuffer.allocate(0));
+BufferCell emptyCell = BufferCell.live(cfs.metadata, emptyColDef, 0, 
ByteBuffer.allocate(0));
 
 Row.Builder builder = BTreeRow.unsortedBuilder(0);
-
builder.newRow(Clustering.make(AsciiSerializer.instance.serialize("test")));
+builder.newRow(new 
Clustering(AsciiSerializer.instance.serialize("test")));
 builder.addCell(emptyCell);
 Row row = builder.build();
 


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2018-07-05 Thread slebresne

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9e3b8864
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9e3b8864
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9e3b8864

Branch: refs/heads/trunk
Commit: 9e3b88646ff4f97601baf40be7fb708ebb3f7fed
Parents: 243c371 942fe3f
Author: Sylvain Lebresne 
Authored: Thu Jul 5 09:30:03 2018 +0200
Committer: Sylvain Lebresne 
Committed: Thu Jul 5 09:30:03 2018 +0200

--

--



-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2018-07-04 Thread slebresne

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/49e63c27
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/49e63c27
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/49e63c27

Branch: refs/heads/cassandra-3.11
Commit: 49e63c273e973c0f8ab3769caae6f82c18449f44
Parents: c3a66ab 93012e4
Author: Sylvain Lebresne 
Authored: Wed Jul 4 11:51:34 2018 +0200
Committer: Sylvain Lebresne 
Committed: Wed Jul 4 11:51:34 2018 +0200

--
 CHANGES.txt |  1 +
 .../cassandra/db/context/CounterContext.java|  3 +++
 .../apache/cassandra/db/CounterCellTest.java| 23 
 3 files changed, 27 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/49e63c27/CHANGES.txt
--
diff --cc CHANGES.txt
index 951b0c0,00944dc..32ccda3
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,23 -1,6 +1,24 @@@
 -3.0.17
 +3.11.3
 + * Validate supported column type with SASI analyzer (CASSANDRA-13669)
 + * Remove BTree.Builder Recycler to reduce memory usage (CASSANDRA-13929)
 + * Reduce nodetool GC thread count (CASSANDRA-14475)
 + * Fix New SASI view creation during Index Redistribution (CASSANDRA-14055)
 + * Remove string formatting lines from BufferPool hot path (CASSANDRA-14416)
 + * Update metrics to 3.1.5 (CASSANDRA-12924)
 + * Detect OpenJDK jvm type and architecture (CASSANDRA-12793)
 + * Don't use guava collections in the non-system keyspace jmx attributes 
(CASSANDRA-12271)
 + * Allow existing nodes to use all peers in shadow round (CASSANDRA-13851)
 + * Fix cqlsh to read connection.ssl cqlshrc option again (CASSANDRA-14299)
 + * Downgrade log level to trace for CommitLogSegmentManager (CASSANDRA-14370)
 + * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891)
 + * Serialize empty buffer as empty string for json output format 
(CASSANDRA-14245)
 + * Allow logging implementation to be interchanged for embedded testing 
(CASSANDRA-13396)
 + * SASI tokenizer for simple delimiter based entries (CASSANDRA-14247)
 + * Fix Loss of digits when doing CAST from varint/bigint to decimal 
(CASSANDRA-14170)
 + * RateBasedBackPressure unnecessarily invokes a lock on the Guava 
RateLimiter (CASSANDRA-14163)
 + * Fix wildcard GROUP BY queries (CASSANDRA-14209)
 +Merged from 3.0:
+  * Fix potential IndexOutOfBoundsException with counters (CASSANDRA-14167)
 - * Restore resumable hints delivery, backport CASSANDRA-11960 
(CASSANDRA-14419)
   * Always close RT markers returned by ReadCommand#executeLocally() 
(CASSANDRA-14515)
   * Reverse order queries with range tombstones can cause data loss 
(CASSANDRA-14513)
   * Fix regression of lagging commitlog flush log message (CASSANDRA-14451)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/49e63c27/test/unit/org/apache/cassandra/db/CounterCellTest.java
--
diff --cc test/unit/org/apache/cassandra/db/CounterCellTest.java
index 74599c3,8859fc6..5208cb2
--- a/test/unit/org/apache/cassandra/db/CounterCellTest.java
+++ b/test/unit/org/apache/cassandra/db/CounterCellTest.java
@@@ -29,12 -29,16 +29,15 @@@ import org.junit.Test
  
  import org.apache.cassandra.SchemaLoader;
  import org.apache.cassandra.config.ColumnDefinition;
+ import org.apache.cassandra.db.rows.BTreeRow;
  import org.apache.cassandra.db.rows.BufferCell;
  import org.apache.cassandra.db.rows.Cell;
 -import org.apache.cassandra.db.rows.CellPath;
  import org.apache.cassandra.db.rows.Cells;
  import org.apache.cassandra.db.context.CounterContext;
+ import org.apache.cassandra.db.rows.Row;
  import org.apache.cassandra.exceptions.ConfigurationException;
  import org.apache.cassandra.schema.KeyspaceParams;
+ import org.apache.cassandra.serializers.AsciiSerializer;
  import org.apache.cassandra.utils.*;
  
  import static org.junit.Assert.*;


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[3/6] cassandra git commit: Ignore empty Counter cells on digest calculation.

2018-07-04 Thread slebresne

Ignore empty Counter cells on digest calculation.

patch by Francisco FernÃ¡ndez CastaÃ±o; reviewed by Sylvain Lebresne for 
CASSANDRA-14167


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/93012e43
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/93012e43
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/93012e43

Branch: refs/heads/trunk
Commit: 93012e43ea03411456c20e7cec1ede73ff46b7d2
Parents: c498258
Author: Fransisco Fernandez Castano 
Authored: Sun Apr 22 13:08:26 2018 +0200
Committer: Sylvain Lebresne 
Committed: Wed Jul 4 11:50:41 2018 +0200

--
 CHANGES.txt |  1 +
 .../cassandra/db/context/CounterContext.java|  3 +++
 .../apache/cassandra/db/CounterCellTest.java| 23 
 3 files changed, 27 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/93012e43/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index ee95718..00944dc 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.17
+ * Fix potential IndexOutOfBoundsException with counters (CASSANDRA-14167)
  * Restore resumable hints delivery, backport CASSANDRA-11960 (CASSANDRA-14419)
  * Always close RT markers returned by ReadCommand#executeLocally() 
(CASSANDRA-14515)
  * Reverse order queries with range tombstones can cause data loss 
(CASSANDRA-14513)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/93012e43/src/java/org/apache/cassandra/db/context/CounterContext.java
--
diff --git a/src/java/org/apache/cassandra/db/context/CounterContext.java 
b/src/java/org/apache/cassandra/db/context/CounterContext.java
index 29e5cfc..b402464 100644
--- a/src/java/org/apache/cassandra/db/context/CounterContext.java
+++ b/src/java/org/apache/cassandra/db/context/CounterContext.java
@@ -692,6 +692,9 @@ public class CounterContext
  */
 public void updateDigest(MessageDigest message, ByteBuffer context)
 {
+// context can be empty due to the optimization from CASSANDRA-10657
+if (!context.hasRemaining())
+return;
 ByteBuffer dup = context.duplicate();
 dup.position(context.position() + headerLength(context));
 message.update(dup);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/93012e43/test/unit/org/apache/cassandra/db/CounterCellTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/CounterCellTest.java 
b/test/unit/org/apache/cassandra/db/CounterCellTest.java
index a8ddfcc..8859fc6 100644
--- a/test/unit/org/apache/cassandra/db/CounterCellTest.java
+++ b/test/unit/org/apache/cassandra/db/CounterCellTest.java
@@ -29,13 +29,16 @@ import org.junit.Test;
 
 import org.apache.cassandra.SchemaLoader;
 import org.apache.cassandra.config.ColumnDefinition;
+import org.apache.cassandra.db.rows.BTreeRow;
 import org.apache.cassandra.db.rows.BufferCell;
 import org.apache.cassandra.db.rows.Cell;
 import org.apache.cassandra.db.rows.CellPath;
 import org.apache.cassandra.db.rows.Cells;
 import org.apache.cassandra.db.context.CounterContext;
+import org.apache.cassandra.db.rows.Row;
 import org.apache.cassandra.exceptions.ConfigurationException;
 import org.apache.cassandra.schema.KeyspaceParams;
+import org.apache.cassandra.serializers.AsciiSerializer;
 import org.apache.cassandra.utils.*;
 
 import static org.junit.Assert.*;
@@ -281,4 +284,24 @@ public class CounterCellTest
 
 assert Arrays.equals(digest1.digest(), digest2.digest());
 }
+
+@Test
+public void testDigestWithEmptyCells() throws Exception
+{
+// For DB-1881
+ColumnFamilyStore cfs = 
Keyspace.open(KEYSPACE1).getColumnFamilyStore(COUNTER1);
+
+ColumnDefinition emptyColDef = 
cfs.metadata.getColumnDefinition(ByteBufferUtil.bytes("val2"));
+BufferCell emptyCell = BufferCell.live(emptyColDef, 0, 
ByteBuffer.allocate(0));
+
+Row.Builder builder = BTreeRow.unsortedBuilder(0);
+
builder.newRow(Clustering.make(AsciiSerializer.instance.serialize("test")));
+builder.addCell(emptyCell);
+Row row = builder.build();
+
+MessageDigest digest = MessageDigest.getInstance("md5");
+row.digest(digest);
+assertNotNull(digest.digest());
+}
+
 }


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[1/6] cassandra git commit: Ignore empty Counter cells on digest calculation.

2018-07-04 Thread slebresne

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 c4982587b -> 93012e43e
  refs/heads/cassandra-3.11 c3a66ab1e -> 49e63c273
  refs/heads/trunk d500100a4 -> 243c371f4


Ignore empty Counter cells on digest calculation.

patch by Francisco FernÃ¡ndez CastaÃ±o; reviewed by Sylvain Lebresne for 
CASSANDRA-14167


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/93012e43
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/93012e43
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/93012e43

Branch: refs/heads/cassandra-3.0
Commit: 93012e43ea03411456c20e7cec1ede73ff46b7d2
Parents: c498258
Author: Fransisco Fernandez Castano 
Authored: Sun Apr 22 13:08:26 2018 +0200
Committer: Sylvain Lebresne 
Committed: Wed Jul 4 11:50:41 2018 +0200

--
 CHANGES.txt |  1 +
 .../cassandra/db/context/CounterContext.java|  3 +++
 .../apache/cassandra/db/CounterCellTest.java| 23 
 3 files changed, 27 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/93012e43/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index ee95718..00944dc 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.17
+ * Fix potential IndexOutOfBoundsException with counters (CASSANDRA-14167)
  * Restore resumable hints delivery, backport CASSANDRA-11960 (CASSANDRA-14419)
  * Always close RT markers returned by ReadCommand#executeLocally() 
(CASSANDRA-14515)
  * Reverse order queries with range tombstones can cause data loss 
(CASSANDRA-14513)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/93012e43/src/java/org/apache/cassandra/db/context/CounterContext.java
--
diff --git a/src/java/org/apache/cassandra/db/context/CounterContext.java 
b/src/java/org/apache/cassandra/db/context/CounterContext.java
index 29e5cfc..b402464 100644
--- a/src/java/org/apache/cassandra/db/context/CounterContext.java
+++ b/src/java/org/apache/cassandra/db/context/CounterContext.java
@@ -692,6 +692,9 @@ public class CounterContext
  */
 public void updateDigest(MessageDigest message, ByteBuffer context)
 {
+// context can be empty due to the optimization from CASSANDRA-10657
+if (!context.hasRemaining())
+return;
 ByteBuffer dup = context.duplicate();
 dup.position(context.position() + headerLength(context));
 message.update(dup);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/93012e43/test/unit/org/apache/cassandra/db/CounterCellTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/CounterCellTest.java 
b/test/unit/org/apache/cassandra/db/CounterCellTest.java
index a8ddfcc..8859fc6 100644
--- a/test/unit/org/apache/cassandra/db/CounterCellTest.java
+++ b/test/unit/org/apache/cassandra/db/CounterCellTest.java
@@ -29,13 +29,16 @@ import org.junit.Test;
 
 import org.apache.cassandra.SchemaLoader;
 import org.apache.cassandra.config.ColumnDefinition;
+import org.apache.cassandra.db.rows.BTreeRow;
 import org.apache.cassandra.db.rows.BufferCell;
 import org.apache.cassandra.db.rows.Cell;
 import org.apache.cassandra.db.rows.CellPath;
 import org.apache.cassandra.db.rows.Cells;
 import org.apache.cassandra.db.context.CounterContext;
+import org.apache.cassandra.db.rows.Row;
 import org.apache.cassandra.exceptions.ConfigurationException;
 import org.apache.cassandra.schema.KeyspaceParams;
+import org.apache.cassandra.serializers.AsciiSerializer;
 import org.apache.cassandra.utils.*;
 
 import static org.junit.Assert.*;
@@ -281,4 +284,24 @@ public class CounterCellTest
 
 assert Arrays.equals(digest1.digest(), digest2.digest());
 }
+
+@Test
+public void testDigestWithEmptyCells() throws Exception
+{
+// For DB-1881
+ColumnFamilyStore cfs = 
Keyspace.open(KEYSPACE1).getColumnFamilyStore(COUNTER1);
+
+ColumnDefinition emptyColDef = 
cfs.metadata.getColumnDefinition(ByteBufferUtil.bytes("val2"));
+BufferCell emptyCell = BufferCell.live(emptyColDef, 0, 
ByteBuffer.allocate(0));
+
+Row.Builder builder = BTreeRow.unsortedBuilder(0);
+
builder.newRow(Clustering.make(AsciiSerializer.instance.serialize("test")));
+builder.addCell(emptyCell);
+Row row = builder.build();
+
+MessageDigest digest = MessageDigest.getInstance("md5");
+row.digest(digest);
+assertNotNull(digest.digest());
+}
+
 }


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail:

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2018-07-04 Thread slebresne

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/49e63c27
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/49e63c27
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/49e63c27

Branch: refs/heads/trunk
Commit: 49e63c273e973c0f8ab3769caae6f82c18449f44
Parents: c3a66ab 93012e4
Author: Sylvain Lebresne 
Authored: Wed Jul 4 11:51:34 2018 +0200
Committer: Sylvain Lebresne 
Committed: Wed Jul 4 11:51:34 2018 +0200

--
 CHANGES.txt |  1 +
 .../cassandra/db/context/CounterContext.java|  3 +++
 .../apache/cassandra/db/CounterCellTest.java| 23 
 3 files changed, 27 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/49e63c27/CHANGES.txt
--
diff --cc CHANGES.txt
index 951b0c0,00944dc..32ccda3
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,23 -1,6 +1,24 @@@
 -3.0.17
 +3.11.3
 + * Validate supported column type with SASI analyzer (CASSANDRA-13669)
 + * Remove BTree.Builder Recycler to reduce memory usage (CASSANDRA-13929)
 + * Reduce nodetool GC thread count (CASSANDRA-14475)
 + * Fix New SASI view creation during Index Redistribution (CASSANDRA-14055)
 + * Remove string formatting lines from BufferPool hot path (CASSANDRA-14416)
 + * Update metrics to 3.1.5 (CASSANDRA-12924)
 + * Detect OpenJDK jvm type and architecture (CASSANDRA-12793)
 + * Don't use guava collections in the non-system keyspace jmx attributes 
(CASSANDRA-12271)
 + * Allow existing nodes to use all peers in shadow round (CASSANDRA-13851)
 + * Fix cqlsh to read connection.ssl cqlshrc option again (CASSANDRA-14299)
 + * Downgrade log level to trace for CommitLogSegmentManager (CASSANDRA-14370)
 + * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891)
 + * Serialize empty buffer as empty string for json output format 
(CASSANDRA-14245)
 + * Allow logging implementation to be interchanged for embedded testing 
(CASSANDRA-13396)
 + * SASI tokenizer for simple delimiter based entries (CASSANDRA-14247)
 + * Fix Loss of digits when doing CAST from varint/bigint to decimal 
(CASSANDRA-14170)
 + * RateBasedBackPressure unnecessarily invokes a lock on the Guava 
RateLimiter (CASSANDRA-14163)
 + * Fix wildcard GROUP BY queries (CASSANDRA-14209)
 +Merged from 3.0:
+  * Fix potential IndexOutOfBoundsException with counters (CASSANDRA-14167)
 - * Restore resumable hints delivery, backport CASSANDRA-11960 
(CASSANDRA-14419)
   * Always close RT markers returned by ReadCommand#executeLocally() 
(CASSANDRA-14515)
   * Reverse order queries with range tombstones can cause data loss 
(CASSANDRA-14513)
   * Fix regression of lagging commitlog flush log message (CASSANDRA-14451)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/49e63c27/test/unit/org/apache/cassandra/db/CounterCellTest.java
--
diff --cc test/unit/org/apache/cassandra/db/CounterCellTest.java
index 74599c3,8859fc6..5208cb2
--- a/test/unit/org/apache/cassandra/db/CounterCellTest.java
+++ b/test/unit/org/apache/cassandra/db/CounterCellTest.java
@@@ -29,12 -29,16 +29,15 @@@ import org.junit.Test
  
  import org.apache.cassandra.SchemaLoader;
  import org.apache.cassandra.config.ColumnDefinition;
+ import org.apache.cassandra.db.rows.BTreeRow;
  import org.apache.cassandra.db.rows.BufferCell;
  import org.apache.cassandra.db.rows.Cell;
 -import org.apache.cassandra.db.rows.CellPath;
  import org.apache.cassandra.db.rows.Cells;
  import org.apache.cassandra.db.context.CounterContext;
+ import org.apache.cassandra.db.rows.Row;
  import org.apache.cassandra.exceptions.ConfigurationException;
  import org.apache.cassandra.schema.KeyspaceParams;
+ import org.apache.cassandra.serializers.AsciiSerializer;
  import org.apache.cassandra.utils.*;
  
  import static org.junit.Assert.*;


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2018-07-04 Thread slebresne

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/243c371f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/243c371f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/243c371f

Branch: refs/heads/trunk
Commit: 243c371f4881be554439731cb809680a37c875ac
Parents: d500100 49e63c2
Author: Sylvain Lebresne 
Authored: Wed Jul 4 12:10:05 2018 +0200
Committer: Sylvain Lebresne 
Committed: Wed Jul 4 12:10:05 2018 +0200

--
 CHANGES.txt |  1 +
 .../cassandra/db/context/CounterContext.java|  3 +++
 .../apache/cassandra/db/CounterCellTest.java| 24 +++-
 3 files changed, 27 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/243c371f/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/243c371f/src/java/org/apache/cassandra/db/context/CounterContext.java
--
diff --cc src/java/org/apache/cassandra/db/context/CounterContext.java
index 9b34231,b402464..29dc3f0
--- a/src/java/org/apache/cassandra/db/context/CounterContext.java
+++ b/src/java/org/apache/cassandra/db/context/CounterContext.java
@@@ -690,11 -690,14 +690,14 @@@ public class CounterContex
   * nodes. This means in particular that we always have:
   *  updateDigest(ctx) == updateDigest(clearAllLocal(ctx))
   */
 -public void updateDigest(MessageDigest message, ByteBuffer context)
 +public void updateDigest(Hasher hasher, ByteBuffer context)
  {
+ // context can be empty due to the optimization from CASSANDRA-10657
+ if (!context.hasRemaining())
+ return;
  ByteBuffer dup = context.duplicate();
  dup.position(context.position() + headerLength(context));
 -message.update(dup);
 +HashingUtils.updateBytes(hasher, dup);
  }
  
  /**

http://git-wip-us.apache.org/repos/asf/cassandra/blob/243c371f/test/unit/org/apache/cassandra/db/CounterCellTest.java
--
diff --cc test/unit/org/apache/cassandra/db/CounterCellTest.java
index b410427,5208cb2..0ee59e1
--- a/test/unit/org/apache/cassandra/db/CounterCellTest.java
+++ b/test/unit/org/apache/cassandra/db/CounterCellTest.java
@@@ -27,8 -28,8 +27,8 @@@ import org.junit.BeforeClass
  import org.junit.Test;
  
  import org.apache.cassandra.SchemaLoader;
- import org.apache.cassandra.net.MessagingService;
 -import org.apache.cassandra.config.ColumnDefinition;
 +import org.apache.cassandra.schema.ColumnMetadata;
+ import org.apache.cassandra.db.rows.BTreeRow;
  import org.apache.cassandra.db.rows.BufferCell;
  import org.apache.cassandra.db.rows.Cell;
  import org.apache.cassandra.db.rows.Cells;
@@@ -272,12 -275,32 +274,32 @@@ public class CounterCellTes
  
  Cell original = createCounterCellFromContext(cfs, col, state, 5);
  
 -ColumnDefinition cDef = cfs.metadata.getColumnDefinition(col);
 +ColumnMetadata cDef = cfs.metadata().getColumn(col);
  Cell cleared = BufferCell.live(cDef, 5, 
CounterContext.instance().clearAllLocal(state.context));
  
 -original.digest(digest1);
 -cleared.digest(digest2);
 +original.digest(hasher1);
 +cleared.digest(hasher2);
  
 -assert Arrays.equals(digest1.digest(), digest2.digest());
 +Assert.assertEquals(hasher1.hash(), hasher2.hash());
  }
+ 
+ @Test
+ public void testDigestWithEmptyCells() throws Exception
+ {
+ // For DB-1881
+ ColumnFamilyStore cfs = 
Keyspace.open(KEYSPACE1).getColumnFamilyStore(COUNTER1);
+ 
 -ColumnDefinition emptyColDef = 
cfs.metadata.getColumnDefinition(ByteBufferUtil.bytes("val2"));
++ColumnMetadata emptyColDef = 
cfs.metadata().getColumn(ByteBufferUtil.bytes("val2"));
+ BufferCell emptyCell = BufferCell.live(emptyColDef, 0, 
ByteBuffer.allocate(0));
+ 
+ Row.Builder builder = BTreeRow.unsortedBuilder(0);
+ 
builder.newRow(Clustering.make(AsciiSerializer.instance.serialize("test")));
+ builder.addCell(emptyCell);
+ Row row = builder.build();
+ 
 -MessageDigest digest = MessageDigest.getInstance("md5");
 -row.digest(digest);
 -assertNotNull(digest.digest());
++Hasher hasher = HashingUtils.CURRENT_HASH_FUNCTION.newHasher();
++row.digest(hasher);
++assertNotNull(hasher.hash());
+ }
+ 
  }


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail:

[2/6] cassandra git commit: Ignore empty Counter cells on digest calculation.

2018-07-04 Thread slebresne

Ignore empty Counter cells on digest calculation.

patch by Francisco FernÃ¡ndez CastaÃ±o; reviewed by Sylvain Lebresne for 
CASSANDRA-14167


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/93012e43
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/93012e43
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/93012e43

Branch: refs/heads/cassandra-3.11
Commit: 93012e43ea03411456c20e7cec1ede73ff46b7d2
Parents: c498258
Author: Fransisco Fernandez Castano 
Authored: Sun Apr 22 13:08:26 2018 +0200
Committer: Sylvain Lebresne 
Committed: Wed Jul 4 11:50:41 2018 +0200

--
 CHANGES.txt |  1 +
 .../cassandra/db/context/CounterContext.java|  3 +++
 .../apache/cassandra/db/CounterCellTest.java| 23 
 3 files changed, 27 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/93012e43/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index ee95718..00944dc 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.17
+ * Fix potential IndexOutOfBoundsException with counters (CASSANDRA-14167)
  * Restore resumable hints delivery, backport CASSANDRA-11960 (CASSANDRA-14419)
  * Always close RT markers returned by ReadCommand#executeLocally() 
(CASSANDRA-14515)
  * Reverse order queries with range tombstones can cause data loss 
(CASSANDRA-14513)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/93012e43/src/java/org/apache/cassandra/db/context/CounterContext.java
--
diff --git a/src/java/org/apache/cassandra/db/context/CounterContext.java 
b/src/java/org/apache/cassandra/db/context/CounterContext.java
index 29e5cfc..b402464 100644
--- a/src/java/org/apache/cassandra/db/context/CounterContext.java
+++ b/src/java/org/apache/cassandra/db/context/CounterContext.java
@@ -692,6 +692,9 @@ public class CounterContext
  */
 public void updateDigest(MessageDigest message, ByteBuffer context)
 {
+// context can be empty due to the optimization from CASSANDRA-10657
+if (!context.hasRemaining())
+return;
 ByteBuffer dup = context.duplicate();
 dup.position(context.position() + headerLength(context));
 message.update(dup);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/93012e43/test/unit/org/apache/cassandra/db/CounterCellTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/CounterCellTest.java 
b/test/unit/org/apache/cassandra/db/CounterCellTest.java
index a8ddfcc..8859fc6 100644
--- a/test/unit/org/apache/cassandra/db/CounterCellTest.java
+++ b/test/unit/org/apache/cassandra/db/CounterCellTest.java
@@ -29,13 +29,16 @@ import org.junit.Test;
 
 import org.apache.cassandra.SchemaLoader;
 import org.apache.cassandra.config.ColumnDefinition;
+import org.apache.cassandra.db.rows.BTreeRow;
 import org.apache.cassandra.db.rows.BufferCell;
 import org.apache.cassandra.db.rows.Cell;
 import org.apache.cassandra.db.rows.CellPath;
 import org.apache.cassandra.db.rows.Cells;
 import org.apache.cassandra.db.context.CounterContext;
+import org.apache.cassandra.db.rows.Row;
 import org.apache.cassandra.exceptions.ConfigurationException;
 import org.apache.cassandra.schema.KeyspaceParams;
+import org.apache.cassandra.serializers.AsciiSerializer;
 import org.apache.cassandra.utils.*;
 
 import static org.junit.Assert.*;
@@ -281,4 +284,24 @@ public class CounterCellTest
 
 assert Arrays.equals(digest1.digest(), digest2.digest());
 }
+
+@Test
+public void testDigestWithEmptyCells() throws Exception
+{
+// For DB-1881
+ColumnFamilyStore cfs = 
Keyspace.open(KEYSPACE1).getColumnFamilyStore(COUNTER1);
+
+ColumnDefinition emptyColDef = 
cfs.metadata.getColumnDefinition(ByteBufferUtil.bytes("val2"));
+BufferCell emptyCell = BufferCell.live(emptyColDef, 0, 
ByteBuffer.allocate(0));
+
+Row.Builder builder = BTreeRow.unsortedBuilder(0);
+
builder.newRow(Clustering.make(AsciiSerializer.instance.serialize("test")));
+builder.addCell(emptyCell);
+Row row = builder.build();
+
+MessageDigest digest = MessageDigest.getInstance("md5");
+row.digest(digest);
+assertNotNull(digest.digest());
+}
+
 }


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[2/6] cassandra git commit: Fix serialized size of DataLimits

2017-11-18 Thread slebresne

Fix serialized size of DataLimits

patch by Sylvain Lebresne; reviewed by Jason Brown for CASSANDRA-14057


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/eb05025c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/eb05025c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/eb05025c

Branch: refs/heads/cassandra-3.11
Commit: eb05025c0a768241ea61fd86db9a88cfd8f6e93e
Parents: f1e850a
Author: Sylvain Lebresne 
Authored: Fri Nov 17 15:46:58 2017 +0100
Committer: Sylvain Lebresne 
Committed: Sat Nov 18 14:16:42 2017 +0100

--
 CHANGES.txt | 1 +
 src/java/org/apache/cassandra/db/TypeSizes.java | 6 ++
 2 files changed, 7 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb05025c/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index a3c43fd..6c50a3f 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * Fix serialized size of DataLimits (CASSANDRA-14057)
  * Add flag to allow dropping oversized read repair mutations (CASSANDRA-13975)
  * Fix SSTableLoader logger message (CASSANDRA-14003)
  * Fix repair race that caused gossip to block (CASSANDRA-13849)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb05025c/src/java/org/apache/cassandra/db/TypeSizes.java
--
diff --git a/src/java/org/apache/cassandra/db/TypeSizes.java 
b/src/java/org/apache/cassandra/db/TypeSizes.java
index 6543025..b47e300 100644
--- a/src/java/org/apache/cassandra/db/TypeSizes.java
+++ b/src/java/org/apache/cassandra/db/TypeSizes.java
@@ -28,6 +28,7 @@ public final class TypeSizes
 private TypeSizes(){}
 
 private static final int BOOL_SIZE = 1;
+private static final int BYTE_SIZE = 1;
 private static final int SHORT_SIZE = 2;
 private static final int INT_SIZE = 4;
 private static final int LONG_SIZE = 8;
@@ -78,6 +79,11 @@ public final class TypeSizes
 return BOOL_SIZE;
 }
 
+public static int sizeof(byte value)
+{
+return BYTE_SIZE;
+}
+
 public static int sizeof(short value)
 {
 return SHORT_SIZE;


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-11-18 Thread slebresne

Merge branch 'cassandra-3.0' into cassandra-3.11

* cassandra-3.0:
  Fix serialized size of DataLimits


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d2bc37f4
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d2bc37f4
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d2bc37f4

Branch: refs/heads/trunk
Commit: d2bc37f43db14661a159406e421985a2484d5387
Parents: 9ee44db eb05025
Author: Sylvain Lebresne 
Authored: Sat Nov 18 14:17:43 2017 +0100
Committer: Sylvain Lebresne 
Committed: Sat Nov 18 14:17:43 2017 +0100

--
 CHANGES.txt | 1 +
 src/java/org/apache/cassandra/db/TypeSizes.java | 6 ++
 2 files changed, 7 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d2bc37f4/CHANGES.txt
--
diff --cc CHANGES.txt
index a1a1a37,6c50a3f..3fc934e
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,9 -1,5 +1,10 @@@
 -3.0.16
 +3.11.2
 + * Add asm jar to build.xml for maven builds (CASSANDRA-11193)
 + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
 + * Update jackson JSON jars (CASSANDRA-13949)
 + * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
 +Merged from 3.0:
+  * Fix serialized size of DataLimits (CASSANDRA-14057)
   * Add flag to allow dropping oversized read repair mutations 
(CASSANDRA-13975)
   * Fix SSTableLoader logger message (CASSANDRA-14003)
   * Fix repair race that caused gossip to block (CASSANDRA-13849)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2017-11-18 Thread slebresne

Merge branch 'cassandra-3.11' into trunk

* cassandra-3.11:
  Fix serialized size of DataLimits


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/da410153
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/da410153
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/da410153

Branch: refs/heads/trunk
Commit: da410153ed250e03080fd48cd0f57baee481412d
Parents: f1f6ed6 d2bc37f
Author: Sylvain Lebresne 
Authored: Sat Nov 18 14:17:58 2017 +0100
Committer: Sylvain Lebresne 
Committed: Sat Nov 18 14:17:58 2017 +0100

--
 CHANGES.txt | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/da410153/CHANGES.txt
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[3/6] cassandra git commit: Fix serialized size of DataLimits

2017-11-18 Thread slebresne

Fix serialized size of DataLimits

patch by Sylvain Lebresne; reviewed by Jason Brown for CASSANDRA-14057


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/eb05025c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/eb05025c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/eb05025c

Branch: refs/heads/trunk
Commit: eb05025c0a768241ea61fd86db9a88cfd8f6e93e
Parents: f1e850a
Author: Sylvain Lebresne 
Authored: Fri Nov 17 15:46:58 2017 +0100
Committer: Sylvain Lebresne 
Committed: Sat Nov 18 14:16:42 2017 +0100

--
 CHANGES.txt | 1 +
 src/java/org/apache/cassandra/db/TypeSizes.java | 6 ++
 2 files changed, 7 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb05025c/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index a3c43fd..6c50a3f 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * Fix serialized size of DataLimits (CASSANDRA-14057)
  * Add flag to allow dropping oversized read repair mutations (CASSANDRA-13975)
  * Fix SSTableLoader logger message (CASSANDRA-14003)
  * Fix repair race that caused gossip to block (CASSANDRA-13849)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb05025c/src/java/org/apache/cassandra/db/TypeSizes.java
--
diff --git a/src/java/org/apache/cassandra/db/TypeSizes.java 
b/src/java/org/apache/cassandra/db/TypeSizes.java
index 6543025..b47e300 100644
--- a/src/java/org/apache/cassandra/db/TypeSizes.java
+++ b/src/java/org/apache/cassandra/db/TypeSizes.java
@@ -28,6 +28,7 @@ public final class TypeSizes
 private TypeSizes(){}
 
 private static final int BOOL_SIZE = 1;
+private static final int BYTE_SIZE = 1;
 private static final int SHORT_SIZE = 2;
 private static final int INT_SIZE = 4;
 private static final int LONG_SIZE = 8;
@@ -78,6 +79,11 @@ public final class TypeSizes
 return BOOL_SIZE;
 }
 
+public static int sizeof(byte value)
+{
+return BYTE_SIZE;
+}
+
 public static int sizeof(short value)
 {
 return SHORT_SIZE;


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-11-18 Thread slebresne

Merge branch 'cassandra-3.0' into cassandra-3.11

* cassandra-3.0:
  Fix serialized size of DataLimits


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d2bc37f4
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d2bc37f4
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d2bc37f4

Branch: refs/heads/cassandra-3.11
Commit: d2bc37f43db14661a159406e421985a2484d5387
Parents: 9ee44db eb05025
Author: Sylvain Lebresne 
Authored: Sat Nov 18 14:17:43 2017 +0100
Committer: Sylvain Lebresne 
Committed: Sat Nov 18 14:17:43 2017 +0100

--
 CHANGES.txt | 1 +
 src/java/org/apache/cassandra/db/TypeSizes.java | 6 ++
 2 files changed, 7 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d2bc37f4/CHANGES.txt
--
diff --cc CHANGES.txt
index a1a1a37,6c50a3f..3fc934e
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,9 -1,5 +1,10 @@@
 -3.0.16
 +3.11.2
 + * Add asm jar to build.xml for maven builds (CASSANDRA-11193)
 + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
 + * Update jackson JSON jars (CASSANDRA-13949)
 + * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
 +Merged from 3.0:
+  * Fix serialized size of DataLimits (CASSANDRA-14057)
   * Add flag to allow dropping oversized read repair mutations 
(CASSANDRA-13975)
   * Fix SSTableLoader logger message (CASSANDRA-14003)
   * Fix repair race that caused gossip to block (CASSANDRA-13849)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[1/6] cassandra git commit: Fix serialized size of DataLimits

2017-11-18 Thread slebresne

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 f1e850a49 -> eb05025c0
  refs/heads/cassandra-3.11 9ee44db49 -> d2bc37f43
  refs/heads/trunk f1f6ed609 -> da410153e


Fix serialized size of DataLimits

patch by Sylvain Lebresne; reviewed by Jason Brown for CASSANDRA-14057


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/eb05025c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/eb05025c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/eb05025c

Branch: refs/heads/cassandra-3.0
Commit: eb05025c0a768241ea61fd86db9a88cfd8f6e93e
Parents: f1e850a
Author: Sylvain Lebresne 
Authored: Fri Nov 17 15:46:58 2017 +0100
Committer: Sylvain Lebresne 
Committed: Sat Nov 18 14:16:42 2017 +0100

--
 CHANGES.txt | 1 +
 src/java/org/apache/cassandra/db/TypeSizes.java | 6 ++
 2 files changed, 7 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb05025c/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index a3c43fd..6c50a3f 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * Fix serialized size of DataLimits (CASSANDRA-14057)
  * Add flag to allow dropping oversized read repair mutations (CASSANDRA-13975)
  * Fix SSTableLoader logger message (CASSANDRA-14003)
  * Fix repair race that caused gossip to block (CASSANDRA-13849)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb05025c/src/java/org/apache/cassandra/db/TypeSizes.java
--
diff --git a/src/java/org/apache/cassandra/db/TypeSizes.java 
b/src/java/org/apache/cassandra/db/TypeSizes.java
index 6543025..b47e300 100644
--- a/src/java/org/apache/cassandra/db/TypeSizes.java
+++ b/src/java/org/apache/cassandra/db/TypeSizes.java
@@ -28,6 +28,7 @@ public final class TypeSizes
 private TypeSizes(){}
 
 private static final int BOOL_SIZE = 1;
+private static final int BYTE_SIZE = 1;
 private static final int SHORT_SIZE = 2;
 private static final int INT_SIZE = 4;
 private static final int LONG_SIZE = 8;
@@ -78,6 +79,11 @@ public final class TypeSizes
 return BOOL_SIZE;
 }
 
+public static int sizeof(byte value)
+{
+return BYTE_SIZE;
+}
+
 public static int sizeof(short value)
 {
 return SHORT_SIZE;


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[04/12] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-10-06 Thread slebresne

Merge branch 'cassandra-3.0' into cassandra-3.11

* cassandra-3.0:
  Fix handling of cells for removed column when reading legacy sstables


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/15276844
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/15276844
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/15276844

Branch: refs/heads/cassandra-3.11
Commit: 1527684462f903be6481324e401dc018da7e0e39
Parents: 710657d 5378ba2
Author: Sylvain Lebresne 
Authored: Fri Oct 6 16:16:25 2017 +0200
Committer: Sylvain Lebresne 
Committed: Fri Oct 6 16:16:25 2017 +0200

--
 .../org/apache/cassandra/db/LegacyLayout.java   | 70 +---
 .../cassandra/db/UnfilteredDeserializer.java| 29 +---
 2 files changed, 67 insertions(+), 32 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/15276844/src/java/org/apache/cassandra/db/LegacyLayout.java
--
diff --cc src/java/org/apache/cassandra/db/LegacyLayout.java
index 77aa59a,3ba96a6..9679a32
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@@ -1021,29 -1037,36 +1038,36 @@@ public abstract class LegacyLayou
  };
  }
  
- public static LegacyAtom readLegacyAtom(CFMetaData metadata, 
DataInputPlus in, boolean readAllAsDynamic) throws IOException
+ public static LegacyAtom readLegacyAtom(CFMetaData metadata, 
DataInputPlus in, boolean readAllAsDynamic)
+ throws IOException, UnknownColumnException
  {
- while (true)
- {
- ByteBuffer cellname = ByteBufferUtil.readWithShortLength(in);
- if (!cellname.hasRemaining())
- return null; // END_OF_ROW
+ ByteBuffer cellname = ByteBufferUtil.readWithShortLength(in);
+ if (!cellname.hasRemaining())
+ return null; // END_OF_ROW
  
- try
- {
- int b = in.readUnsignedByte();
- return (b & RANGE_TOMBSTONE_MASK) != 0
- ? readLegacyRangeTombstoneBody(metadata, in, cellname)
- : readLegacyCellBody(metadata, in, cellname, b, 
SerializationHelper.Flag.LOCAL, readAllAsDynamic);
- }
- catch (UnknownColumnException e)
- {
- // We can get there if we read a cell for a dropped column, 
and ff that is the case,
- // then simply ignore the cell is fine. But also not that we 
ignore if it's the
- // system keyspace because for those table we actually remove 
columns without registering
- // them in the dropped columns
- assert 
metadata.ksName.equals(SchemaConstants.SYSTEM_KEYSPACE_NAME) || 
metadata.getDroppedColumnDefinition(e.columnName) != null : e.getMessage();
- }
+ try
+ {
+ int b = in.readUnsignedByte();
+ return (b & RANGE_TOMBSTONE_MASK) != 0
+? readLegacyRangeTombstoneBody(metadata, in, cellname)
+: readLegacyCellBody(metadata, in, cellname, b, 
SerializationHelper.Flag.LOCAL, readAllAsDynamic);
+ }
+ catch (UnknownColumnException e)
+ {
+ // We legitimately can get here in 2 cases:
+ // 1) for system tables, because we've unceremoniously removed 
columns (without registering them as dropped)
+ // 2) for dropped columns.
+ // In any other case, there is a mismatch between the schema and 
the data, and we complain loudly in
+ // that case. Note that if we are in a legit case of an unknown 
column, we want to simply skip that cell,
+ // but we don't do this here and re-throw the exception because 
the calling code sometimes has to know
+ // about this happening. This does mean code calling this method 
should handle this case properly.
 -if (!metadata.ksName.equals(SystemKeyspace.NAME) && 
metadata.getDroppedColumnDefinition(e.columnName) == null)
++if (!metadata.ksName.equals(SchemaConstants.SYSTEM_KEYSPACE_NAME) 
&& metadata.getDroppedColumnDefinition(e.columnName) == null)
+ throw new IllegalStateException(String.format("Got cell for 
unknown column %s in sstable of %s.%s: " +
+   "This suggest a 
problem with the schema which doesn't list " +
+   "this column. 
Even if that column was dropped, it should have " +
+   "been listed as 
such", metadata.ksName, metadata.cfName,

[06/12] cassandra git commit: Merge commit '1527684462f903be6481324e401dc018da7e0e39' into trunk

2017-10-06 Thread slebresne

Merge commit '1527684462f903be6481324e401dc018da7e0e39' into trunk

* commit '1527684462f903be6481324e401dc018da7e0e39':
  Fix handling of cells for removed column when reading legacy sstables


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9815c466
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9815c466
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9815c466

Branch: refs/heads/trunk
Commit: 9815c46679ad14b038f0314a9b9611e099750724
Parents: cfee3e9 1527684
Author: Sylvain Lebresne 
Authored: Fri Oct 6 16:16:53 2017 +0200
Committer: Sylvain Lebresne 
Committed: Fri Oct 6 16:16:53 2017 +0200

--

--



-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[11/12] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-10-06 Thread slebresne

Merge branch 'cassandra-3.0' into cassandra-3.11

* cassandra-3.0:
  Ninja-fix changelog


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3d09901b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3d09901b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3d09901b

Branch: refs/heads/trunk
Commit: 3d09901b4a0e67ce162283f85c34b1fe3c55dce3
Parents: 1527684 8a424ce
Author: Sylvain Lebresne 
Authored: Fri Oct 6 16:18:36 2017 +0200
Committer: Sylvain Lebresne 
Committed: Fri Oct 6 16:18:36 2017 +0200

--
 CHANGES.txt | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3d09901b/CHANGES.txt
--
diff --cc CHANGES.txt
index e231bf2,6977be7..879397b
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,22 -1,8 +1,23 @@@
 -3.0.16
 +3.11.2
 +Merged from 3.0:
+  * Mishandling of cells for removed/dropped columns when reading legacy files 
(CASSANDRA-13939)
   * Deserialise sstable metadata in nodetool verify (CASSANDRA-13922)
  
 -3.0.15
 +
 +3.11.1
 + * Fix the computation of cdc_total_space_in_mb for exabyte filesystems 
(CASSANDRA-13808)
 + * AbstractTokenTreeBuilder#serializedSize returns wrong value when there is 
a single leaf and overflow collisions (CASSANDRA-13869)
 + * Add a compaction option to TWCS to ignore sstables overlapping checks 
(CASSANDRA-13418)
 + * BTree.Builder memory leak (CASSANDRA-13754)
 + * Revert CASSANDRA-10368 of supporting non-pk column filtering due to 
correctness (CASSANDRA-13798)
 + * Add a skip read validation flag to cassandra-stress (CASSANDRA-13772)
 + * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
 + * Better bootstrap failure message when blocked by (potential) range 
movement (CASSANDRA-13744)
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
   * Improve TRUNCATE performance (CASSANDRA-13909)
   * Implement short read protection on partition boundaries (CASSANDRA-13595)
   * Fix ISE thrown by UPI.Serializer.hasNext() for some SELECT queries 
(CASSANDRA-13911)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[12/12] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2017-10-06 Thread slebresne

Merge branch 'cassandra-3.11' into trunk

* cassandra-3.11:
  Ninja-fix changelog


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9663c009
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9663c009
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9663c009

Branch: refs/heads/trunk
Commit: 9663c0093840a352faa08e3ffda6160745df75b8
Parents: 9815c46 3d09901
Author: Sylvain Lebresne 
Authored: Fri Oct 6 16:18:46 2017 +0200
Committer: Sylvain Lebresne 
Committed: Fri Oct 6 16:18:46 2017 +0200

--
 CHANGES.txt | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9663c009/CHANGES.txt
--
diff --cc CHANGES.txt
index 90c6b9c,879397b..eeed794
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,155 -1,6 +1,156 @@@
 +4.0
 + * Checksum sstable metadata (CASSANDRA-13321)
 + * Expose recent histograms in JmxHistograms (CASSANDRA-13642)
 + * Fix buffer length comparison when decompressing in netty-based streaming 
(CASSANDRA-13899)
 + * Properly close StreamCompressionInputStream to release any ByteBuf 
(CASSANDRA-13906)
 + * Add SERIAL and LOCAL_SERIAL support for cassandra-stress (CASSANDRA-13925)
 + * LCS needlessly checks for L0 STCS candidates multiple times 
(CASSANDRA-12961)
 + * Correctly close netty channels when a stream session ends (CASSANDRA-13905)
 + * Update lz4 to 1.4.0 (CASSANDRA-13741)
 + * Optimize Paxos prepare and propose stage for local requests 
(CASSANDRA-13862)
 + * Throttle base partitions during MV repair streaming to prevent OOM 
(CASSANDRA-13299)
 + * Use compaction threshold for STCS in L0 (CASSANDRA-13861)
 + * Fix problem with min_compress_ratio: 1 and disallow ratio < 1 
(CASSANDRA-13703)
 + * Add extra information to SASI timeout exception (CASSANDRA-13677)
 + * Add incremental repair support for --hosts, --force, and subrange repair 
(CASSANDRA-13818)
 + * Rework CompactionStrategyManager.getScanners synchronization 
(CASSANDRA-13786)
 + * Add additional unit tests for batch behavior, TTLs, Timestamps 
(CASSANDRA-13846)
 + * Add keyspace and table name in schema validation exception 
(CASSANDRA-13845)
 + * Emit metrics whenever we hit tombstone failures and warn thresholds 
(CASSANDRA-13771)
 + * Make netty EventLoopGroups daemon threads (CASSANDRA-13837)
 + * Race condition when closing stream sessions (CASSANDRA-13852)
 + * NettyFactoryTest is failing in trunk on macOS (CASSANDRA-13831)
 + * Allow changing log levels via nodetool for related classes 
(CASSANDRA-12696)
 + * Add stress profile yaml with LWT (CASSANDRA-7960)
 + * Reduce memory copies and object creations when acting on ByteBufs 
(CASSANDRA-13789)
 + * Simplify mx4j configuration (Cassandra-13578)
 + * Fix trigger example on 4.0 (CASSANDRA-13796)
 + * Force minumum timeout value (CASSANDRA-9375)
 + * Use netty for streaming (CASSANDRA-12229)
 + * Use netty for internode messaging (CASSANDRA-8457)
 + * Add bytes repaired/unrepaired to nodetool tablestats (CASSANDRA-13774)
 + * Don't delete incremental repair sessions if they still have sstables 
(CASSANDRA-13758)
 + * Fix pending repair manager index out of bounds check (CASSANDRA-13769)
 + * Don't use RangeFetchMapCalculator when RF=1 (CASSANDRA-13576)
 + * Don't optimise trivial ranges in RangeFetchMapCalculator (CASSANDRA-13664)
 + * Use an ExecutorService for repair commands instead of new 
Thread(..).start() (CASSANDRA-13594)
 + * Fix race / ref leak in anticompaction (CASSANDRA-13688)
 + * Expose tasks queue length via JMX (CASSANDRA-12758)
 + * Fix race / ref leak in PendingRepairManager (CASSANDRA-13751)
 + * Enable ppc64le runtime as unsupported architecture (CASSANDRA-13615)
 + * Improve sstablemetadata output (CASSANDRA-11483)
 + * Support for migrating legacy users to roles has been dropped 
(CASSANDRA-13371)
 + * Introduce error metrics for repair (CASSANDRA-13387)
 + * Refactoring to primitive functional interfaces in AuthCache 
(CASSANDRA-13732)
 + * Update metrics to 3.1.5 (CASSANDRA-13648)
 + * batch_size_warn_threshold_in_kb can now be set at runtime (CASSANDRA-13699)
 + * Avoid always rebuilding secondary indexes at startup (CASSANDRA-13725)
 + * Upgrade JMH from 1.13 to 1.19 (CASSANDRA-13727)
 + * Upgrade SLF4J from 1.7.7 to 1.7.25 (CASSANDRA-12996)
 + * Default for start_native_transport now true if not set in config 
(CASSANDRA-13656)
 + * Don't add localhost to the graph when calculating where to stream from 
(CASSANDRA-13583)
 + * Make CDC availability more deterministic via hard-linking (CASSANDRA-12148)
 + * Allow skipping equality-restricted clustering columns in ORDER BY clause 
(CASSANDRA-10271)
 + * Use common nowInSec for validation compactions (CASSANDRA-13671)
 + *

[07/12] cassandra git commit: Ninja-fix changelog

2017-10-06 Thread slebresne

Ninja-fix changelog


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8a424cef
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8a424cef
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8a424cef

Branch: refs/heads/cassandra-3.0
Commit: 8a424cef32270e75b3a4565793b2d436e6df181f
Parents: 5378ba2
Author: Sylvain Lebresne 
Authored: Fri Oct 6 16:18:07 2017 +0200
Committer: Sylvain Lebresne 
Committed: Fri Oct 6 16:18:07 2017 +0200

--
 CHANGES.txt | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/8a424cef/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index df05f7f..6977be7 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,6 +1,6 @@
 3.0.16
-  * Deserialise sstable metadata in nodetool verify (CASSANDRA-13922)
-
+ * Mishandling of cells for removed/dropped columns when reading legacy files 
(CASSANDRA-13939)
+ * Deserialise sstable metadata in nodetool verify (CASSANDRA-13922)
 
 3.0.15
  * Improve TRUNCATE performance (CASSANDRA-13909)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[10/12] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-10-06 Thread slebresne

Merge branch 'cassandra-3.0' into cassandra-3.11

* cassandra-3.0:
  Ninja-fix changelog


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3d09901b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3d09901b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3d09901b

Branch: refs/heads/cassandra-3.11
Commit: 3d09901b4a0e67ce162283f85c34b1fe3c55dce3
Parents: 1527684 8a424ce
Author: Sylvain Lebresne 
Authored: Fri Oct 6 16:18:36 2017 +0200
Committer: Sylvain Lebresne 
Committed: Fri Oct 6 16:18:36 2017 +0200

--
 CHANGES.txt | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3d09901b/CHANGES.txt
--
diff --cc CHANGES.txt
index e231bf2,6977be7..879397b
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,22 -1,8 +1,23 @@@
 -3.0.16
 +3.11.2
 +Merged from 3.0:
+  * Mishandling of cells for removed/dropped columns when reading legacy files 
(CASSANDRA-13939)
   * Deserialise sstable metadata in nodetool verify (CASSANDRA-13922)
  
 -3.0.15
 +
 +3.11.1
 + * Fix the computation of cdc_total_space_in_mb for exabyte filesystems 
(CASSANDRA-13808)
 + * AbstractTokenTreeBuilder#serializedSize returns wrong value when there is 
a single leaf and overflow collisions (CASSANDRA-13869)
 + * Add a compaction option to TWCS to ignore sstables overlapping checks 
(CASSANDRA-13418)
 + * BTree.Builder memory leak (CASSANDRA-13754)
 + * Revert CASSANDRA-10368 of supporting non-pk column filtering due to 
correctness (CASSANDRA-13798)
 + * Add a skip read validation flag to cassandra-stress (CASSANDRA-13772)
 + * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
 + * Better bootstrap failure message when blocked by (potential) range 
movement (CASSANDRA-13744)
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
   * Improve TRUNCATE performance (CASSANDRA-13909)
   * Implement short read protection on partition boundaries (CASSANDRA-13595)
   * Fix ISE thrown by UPI.Serializer.hasNext() for some SELECT queries 
(CASSANDRA-13911)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[05/12] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-10-06 Thread slebresne

Merge branch 'cassandra-3.0' into cassandra-3.11

* cassandra-3.0:
  Fix handling of cells for removed column when reading legacy sstables


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/15276844
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/15276844
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/15276844

Branch: refs/heads/trunk
Commit: 1527684462f903be6481324e401dc018da7e0e39
Parents: 710657d 5378ba2
Author: Sylvain Lebresne 
Authored: Fri Oct 6 16:16:25 2017 +0200
Committer: Sylvain Lebresne 
Committed: Fri Oct 6 16:16:25 2017 +0200

--
 .../org/apache/cassandra/db/LegacyLayout.java   | 70 +---
 .../cassandra/db/UnfilteredDeserializer.java| 29 +---
 2 files changed, 67 insertions(+), 32 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/15276844/src/java/org/apache/cassandra/db/LegacyLayout.java
--
diff --cc src/java/org/apache/cassandra/db/LegacyLayout.java
index 77aa59a,3ba96a6..9679a32
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@@ -1021,29 -1037,36 +1038,36 @@@ public abstract class LegacyLayou
  };
  }
  
- public static LegacyAtom readLegacyAtom(CFMetaData metadata, 
DataInputPlus in, boolean readAllAsDynamic) throws IOException
+ public static LegacyAtom readLegacyAtom(CFMetaData metadata, 
DataInputPlus in, boolean readAllAsDynamic)
+ throws IOException, UnknownColumnException
  {
- while (true)
- {
- ByteBuffer cellname = ByteBufferUtil.readWithShortLength(in);
- if (!cellname.hasRemaining())
- return null; // END_OF_ROW
+ ByteBuffer cellname = ByteBufferUtil.readWithShortLength(in);
+ if (!cellname.hasRemaining())
+ return null; // END_OF_ROW
  
- try
- {
- int b = in.readUnsignedByte();
- return (b & RANGE_TOMBSTONE_MASK) != 0
- ? readLegacyRangeTombstoneBody(metadata, in, cellname)
- : readLegacyCellBody(metadata, in, cellname, b, 
SerializationHelper.Flag.LOCAL, readAllAsDynamic);
- }
- catch (UnknownColumnException e)
- {
- // We can get there if we read a cell for a dropped column, 
and ff that is the case,
- // then simply ignore the cell is fine. But also not that we 
ignore if it's the
- // system keyspace because for those table we actually remove 
columns without registering
- // them in the dropped columns
- assert 
metadata.ksName.equals(SchemaConstants.SYSTEM_KEYSPACE_NAME) || 
metadata.getDroppedColumnDefinition(e.columnName) != null : e.getMessage();
- }
+ try
+ {
+ int b = in.readUnsignedByte();
+ return (b & RANGE_TOMBSTONE_MASK) != 0
+? readLegacyRangeTombstoneBody(metadata, in, cellname)
+: readLegacyCellBody(metadata, in, cellname, b, 
SerializationHelper.Flag.LOCAL, readAllAsDynamic);
+ }
+ catch (UnknownColumnException e)
+ {
+ // We legitimately can get here in 2 cases:
+ // 1) for system tables, because we've unceremoniously removed 
columns (without registering them as dropped)
+ // 2) for dropped columns.
+ // In any other case, there is a mismatch between the schema and 
the data, and we complain loudly in
+ // that case. Note that if we are in a legit case of an unknown 
column, we want to simply skip that cell,
+ // but we don't do this here and re-throw the exception because 
the calling code sometimes has to know
+ // about this happening. This does mean code calling this method 
should handle this case properly.
 -if (!metadata.ksName.equals(SystemKeyspace.NAME) && 
metadata.getDroppedColumnDefinition(e.columnName) == null)
++if (!metadata.ksName.equals(SchemaConstants.SYSTEM_KEYSPACE_NAME) 
&& metadata.getDroppedColumnDefinition(e.columnName) == null)
+ throw new IllegalStateException(String.format("Got cell for 
unknown column %s in sstable of %s.%s: " +
+   "This suggest a 
problem with the schema which doesn't list " +
+   "this column. 
Even if that column was dropped, it should have " +
+   "been listed as 
such", metadata.ksName, metadata.cfName,

[01/12] cassandra git commit: Fix handling of cells for removed column when reading legacy sstables

2017-10-06 Thread slebresne

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 c07441251 -> 8a424cef3
  refs/heads/cassandra-3.11 710657d82 -> 3d09901b4
  refs/heads/trunk cfee3e93b -> 9663c0093


Fix handling of cells for removed column when reading legacy sstables

Patch by Sylvain Lebresne; reviewed by Robert Stupp for CASSANDRA-13939


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5378ba27
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5378ba27
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5378ba27

Branch: refs/heads/cassandra-3.0
Commit: 5378ba27ff66b4f39e34d613f99b0f2eddf7990d
Parents: c074412
Author: Sylvain Lebresne 
Authored: Thu Oct 5 10:30:56 2017 +0200
Committer: Sylvain Lebresne 
Committed: Fri Oct 6 16:12:56 2017 +0200

--
 .../org/apache/cassandra/db/LegacyLayout.java   | 70 +---
 .../cassandra/db/UnfilteredDeserializer.java| 29 +---
 2 files changed, 67 insertions(+), 32 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5378ba27/src/java/org/apache/cassandra/db/LegacyLayout.java
--
diff --git a/src/java/org/apache/cassandra/db/LegacyLayout.java 
b/src/java/org/apache/cassandra/db/LegacyLayout.java
index 40b9fd3..3ba96a6 100644
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@ -649,7 +649,7 @@ public abstract class LegacyLayout
 
 boolean foundOne = false;
 LegacyAtom atom;
-while ((atom = readLegacyAtom(metadata, in, false)) != null)
+while ((atom = readLegacyAtomSkippingUnknownColumn(metadata,in)) != 
null)
 {
 if (atom.isCell())
 {
@@ -672,6 +672,23 @@ public abstract class LegacyLayout
 return foundOne ? builder.build() : Rows.EMPTY_STATIC_ROW;
 }
 
+private static LegacyAtom readLegacyAtomSkippingUnknownColumn(CFMetaData 
metadata, DataInputPlus in)
+throws IOException
+{
+while (true)
+{
+try
+{
+return readLegacyAtom(metadata, in, false);
+}
+catch (UnknownColumnException e)
+{
+// Simply skip, as the method name implies.
+}
+}
+
+}
+
 private static Row getNextRow(CellGrouper grouper, PeekingIterator cells)
 {
 if (!cells.hasNext())
@@ -1020,29 +1037,36 @@ public abstract class LegacyLayout
 };
 }
 
-public static LegacyAtom readLegacyAtom(CFMetaData metadata, DataInputPlus 
in, boolean readAllAsDynamic) throws IOException
+public static LegacyAtom readLegacyAtom(CFMetaData metadata, DataInputPlus 
in, boolean readAllAsDynamic)
+throws IOException, UnknownColumnException
 {
-while (true)
-{
-ByteBuffer cellname = ByteBufferUtil.readWithShortLength(in);
-if (!cellname.hasRemaining())
-return null; // END_OF_ROW
-
-try
-{
-int b = in.readUnsignedByte();
-return (b & RANGE_TOMBSTONE_MASK) != 0
-? readLegacyRangeTombstoneBody(metadata, in, cellname)
-: readLegacyCellBody(metadata, in, cellname, b, 
SerializationHelper.Flag.LOCAL, readAllAsDynamic);
-}
-catch (UnknownColumnException e)
-{
-// We can get there if we read a cell for a dropped column, 
and ff that is the case,
-// then simply ignore the cell is fine. But also not that we 
ignore if it's the
-// system keyspace because for those table we actually remove 
columns without registering
-// them in the dropped columns
-assert metadata.ksName.equals(SystemKeyspace.NAME) || 
metadata.getDroppedColumnDefinition(e.columnName) != null : e.getMessage();
-}
+ByteBuffer cellname = ByteBufferUtil.readWithShortLength(in);
+if (!cellname.hasRemaining())
+return null; // END_OF_ROW
+
+try
+{
+int b = in.readUnsignedByte();
+return (b & RANGE_TOMBSTONE_MASK) != 0
+   ? readLegacyRangeTombstoneBody(metadata, in, cellname)
+   : readLegacyCellBody(metadata, in, cellname, b, 
SerializationHelper.Flag.LOCAL, readAllAsDynamic);
+}
+catch (UnknownColumnException e)
+{
+// We legitimately can get here in 2 cases:
+// 1) for system tables, because we've unceremoniously removed 
columns (without registering them as dropped)
+// 2) for dropped columns.
+// In any other case, there is a mismatch

[03/12] cassandra git commit: Fix handling of cells for removed column when reading legacy sstables

2017-10-06 Thread slebresne

Fix handling of cells for removed column when reading legacy sstables

Patch by Sylvain Lebresne; reviewed by Robert Stupp for CASSANDRA-13939


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5378ba27
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5378ba27
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5378ba27

Branch: refs/heads/trunk
Commit: 5378ba27ff66b4f39e34d613f99b0f2eddf7990d
Parents: c074412
Author: Sylvain Lebresne 
Authored: Thu Oct 5 10:30:56 2017 +0200
Committer: Sylvain Lebresne 
Committed: Fri Oct 6 16:12:56 2017 +0200

--
 .../org/apache/cassandra/db/LegacyLayout.java   | 70 +---
 .../cassandra/db/UnfilteredDeserializer.java| 29 +---
 2 files changed, 67 insertions(+), 32 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5378ba27/src/java/org/apache/cassandra/db/LegacyLayout.java
--
diff --git a/src/java/org/apache/cassandra/db/LegacyLayout.java 
b/src/java/org/apache/cassandra/db/LegacyLayout.java
index 40b9fd3..3ba96a6 100644
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@ -649,7 +649,7 @@ public abstract class LegacyLayout
 
 boolean foundOne = false;
 LegacyAtom atom;
-while ((atom = readLegacyAtom(metadata, in, false)) != null)
+while ((atom = readLegacyAtomSkippingUnknownColumn(metadata,in)) != 
null)
 {
 if (atom.isCell())
 {
@@ -672,6 +672,23 @@ public abstract class LegacyLayout
 return foundOne ? builder.build() : Rows.EMPTY_STATIC_ROW;
 }
 
+private static LegacyAtom readLegacyAtomSkippingUnknownColumn(CFMetaData 
metadata, DataInputPlus in)
+throws IOException
+{
+while (true)
+{
+try
+{
+return readLegacyAtom(metadata, in, false);
+}
+catch (UnknownColumnException e)
+{
+// Simply skip, as the method name implies.
+}
+}
+
+}
+
 private static Row getNextRow(CellGrouper grouper, PeekingIterator cells)
 {
 if (!cells.hasNext())
@@ -1020,29 +1037,36 @@ public abstract class LegacyLayout
 };
 }
 
-public static LegacyAtom readLegacyAtom(CFMetaData metadata, DataInputPlus 
in, boolean readAllAsDynamic) throws IOException
+public static LegacyAtom readLegacyAtom(CFMetaData metadata, DataInputPlus 
in, boolean readAllAsDynamic)
+throws IOException, UnknownColumnException
 {
-while (true)
-{
-ByteBuffer cellname = ByteBufferUtil.readWithShortLength(in);
-if (!cellname.hasRemaining())
-return null; // END_OF_ROW
-
-try
-{
-int b = in.readUnsignedByte();
-return (b & RANGE_TOMBSTONE_MASK) != 0
-? readLegacyRangeTombstoneBody(metadata, in, cellname)
-: readLegacyCellBody(metadata, in, cellname, b, 
SerializationHelper.Flag.LOCAL, readAllAsDynamic);
-}
-catch (UnknownColumnException e)
-{
-// We can get there if we read a cell for a dropped column, 
and ff that is the case,
-// then simply ignore the cell is fine. But also not that we 
ignore if it's the
-// system keyspace because for those table we actually remove 
columns without registering
-// them in the dropped columns
-assert metadata.ksName.equals(SystemKeyspace.NAME) || 
metadata.getDroppedColumnDefinition(e.columnName) != null : e.getMessage();
-}
+ByteBuffer cellname = ByteBufferUtil.readWithShortLength(in);
+if (!cellname.hasRemaining())
+return null; // END_OF_ROW
+
+try
+{
+int b = in.readUnsignedByte();
+return (b & RANGE_TOMBSTONE_MASK) != 0
+   ? readLegacyRangeTombstoneBody(metadata, in, cellname)
+   : readLegacyCellBody(metadata, in, cellname, b, 
SerializationHelper.Flag.LOCAL, readAllAsDynamic);
+}
+catch (UnknownColumnException e)
+{
+// We legitimately can get here in 2 cases:
+// 1) for system tables, because we've unceremoniously removed 
columns (without registering them as dropped)
+// 2) for dropped columns.
+// In any other case, there is a mismatch between the schema and 
the data, and we complain loudly in
+// that case. Note that if we are in a legit case of an unknown 
column, we want to simply skip that cell,
+// but

[08/12] cassandra git commit: Ninja-fix changelog

2017-10-06 Thread slebresne

Ninja-fix changelog


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8a424cef
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8a424cef
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8a424cef

Branch: refs/heads/trunk
Commit: 8a424cef32270e75b3a4565793b2d436e6df181f
Parents: 5378ba2
Author: Sylvain Lebresne 
Authored: Fri Oct 6 16:18:07 2017 +0200
Committer: Sylvain Lebresne 
Committed: Fri Oct 6 16:18:07 2017 +0200

--
 CHANGES.txt | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/8a424cef/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index df05f7f..6977be7 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,6 +1,6 @@
 3.0.16
-  * Deserialise sstable metadata in nodetool verify (CASSANDRA-13922)
-
+ * Mishandling of cells for removed/dropped columns when reading legacy files 
(CASSANDRA-13939)
+ * Deserialise sstable metadata in nodetool verify (CASSANDRA-13922)
 
 3.0.15
  * Improve TRUNCATE performance (CASSANDRA-13909)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[02/12] cassandra git commit: Fix handling of cells for removed column when reading legacy sstables

2017-10-06 Thread slebresne

Fix handling of cells for removed column when reading legacy sstables

Patch by Sylvain Lebresne; reviewed by Robert Stupp for CASSANDRA-13939


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5378ba27
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5378ba27
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5378ba27

Branch: refs/heads/cassandra-3.11
Commit: 5378ba27ff66b4f39e34d613f99b0f2eddf7990d
Parents: c074412
Author: Sylvain Lebresne 
Authored: Thu Oct 5 10:30:56 2017 +0200
Committer: Sylvain Lebresne 
Committed: Fri Oct 6 16:12:56 2017 +0200

--
 .../org/apache/cassandra/db/LegacyLayout.java   | 70 +---
 .../cassandra/db/UnfilteredDeserializer.java| 29 +---
 2 files changed, 67 insertions(+), 32 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5378ba27/src/java/org/apache/cassandra/db/LegacyLayout.java
--
diff --git a/src/java/org/apache/cassandra/db/LegacyLayout.java 
b/src/java/org/apache/cassandra/db/LegacyLayout.java
index 40b9fd3..3ba96a6 100644
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@ -649,7 +649,7 @@ public abstract class LegacyLayout
 
 boolean foundOne = false;
 LegacyAtom atom;
-while ((atom = readLegacyAtom(metadata, in, false)) != null)
+while ((atom = readLegacyAtomSkippingUnknownColumn(metadata,in)) != 
null)
 {
 if (atom.isCell())
 {
@@ -672,6 +672,23 @@ public abstract class LegacyLayout
 return foundOne ? builder.build() : Rows.EMPTY_STATIC_ROW;
 }
 
+private static LegacyAtom readLegacyAtomSkippingUnknownColumn(CFMetaData 
metadata, DataInputPlus in)
+throws IOException
+{
+while (true)
+{
+try
+{
+return readLegacyAtom(metadata, in, false);
+}
+catch (UnknownColumnException e)
+{
+// Simply skip, as the method name implies.
+}
+}
+
+}
+
 private static Row getNextRow(CellGrouper grouper, PeekingIterator cells)
 {
 if (!cells.hasNext())
@@ -1020,29 +1037,36 @@ public abstract class LegacyLayout
 };
 }
 
-public static LegacyAtom readLegacyAtom(CFMetaData metadata, DataInputPlus 
in, boolean readAllAsDynamic) throws IOException
+public static LegacyAtom readLegacyAtom(CFMetaData metadata, DataInputPlus 
in, boolean readAllAsDynamic)
+throws IOException, UnknownColumnException
 {
-while (true)
-{
-ByteBuffer cellname = ByteBufferUtil.readWithShortLength(in);
-if (!cellname.hasRemaining())
-return null; // END_OF_ROW
-
-try
-{
-int b = in.readUnsignedByte();
-return (b & RANGE_TOMBSTONE_MASK) != 0
-? readLegacyRangeTombstoneBody(metadata, in, cellname)
-: readLegacyCellBody(metadata, in, cellname, b, 
SerializationHelper.Flag.LOCAL, readAllAsDynamic);
-}
-catch (UnknownColumnException e)
-{
-// We can get there if we read a cell for a dropped column, 
and ff that is the case,
-// then simply ignore the cell is fine. But also not that we 
ignore if it's the
-// system keyspace because for those table we actually remove 
columns without registering
-// them in the dropped columns
-assert metadata.ksName.equals(SystemKeyspace.NAME) || 
metadata.getDroppedColumnDefinition(e.columnName) != null : e.getMessage();
-}
+ByteBuffer cellname = ByteBufferUtil.readWithShortLength(in);
+if (!cellname.hasRemaining())
+return null; // END_OF_ROW
+
+try
+{
+int b = in.readUnsignedByte();
+return (b & RANGE_TOMBSTONE_MASK) != 0
+   ? readLegacyRangeTombstoneBody(metadata, in, cellname)
+   : readLegacyCellBody(metadata, in, cellname, b, 
SerializationHelper.Flag.LOCAL, readAllAsDynamic);
+}
+catch (UnknownColumnException e)
+{
+// We legitimately can get here in 2 cases:
+// 1) for system tables, because we've unceremoniously removed 
columns (without registering them as dropped)
+// 2) for dropped columns.
+// In any other case, there is a mismatch between the schema and 
the data, and we complain loudly in
+// that case. Note that if we are in a legit case of an unknown 
column, we want to simply skip that cell,
+

[09/12] cassandra git commit: Ninja-fix changelog

2017-10-06 Thread slebresne

Ninja-fix changelog


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8a424cef
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8a424cef
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8a424cef

Branch: refs/heads/cassandra-3.11
Commit: 8a424cef32270e75b3a4565793b2d436e6df181f
Parents: 5378ba2
Author: Sylvain Lebresne 
Authored: Fri Oct 6 16:18:07 2017 +0200
Committer: Sylvain Lebresne 
Committed: Fri Oct 6 16:18:07 2017 +0200

--
 CHANGES.txt | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/8a424cef/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index df05f7f..6977be7 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,6 +1,6 @@
 3.0.16
-  * Deserialise sstable metadata in nodetool verify (CASSANDRA-13922)
-
+ * Mishandling of cells for removed/dropped columns when reading legacy files 
(CASSANDRA-13939)
+ * Deserialise sstable metadata in nodetool verify (CASSANDRA-13922)
 
 3.0.15
  * Improve TRUNCATE performance (CASSANDRA-13909)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[1/6] cassandra git commit: Fix sstable reader to support range-tombstone-marker for multi-slices

2017-09-20 Thread slebresne

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 35e32f20b -> 975c3d81b
  refs/heads/cassandra-3.11 594f1c1df -> 66115139a
  refs/heads/trunk eb7669215 -> 9a6247482


Fix sstable reader to support range-tombstone-marker for multi-slices

patch by Zhao Yang; reviewed by Sylvain Lebresne for CASSANDRA-13787


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/975c3d81
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/975c3d81
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/975c3d81

Branch: refs/heads/cassandra-3.0
Commit: 975c3d81b67e9c1e1dcefdda3f90e8edf6be5efa
Parents: 35e32f2
Author: Zhao Yang 
Authored: Wed Aug 23 16:15:25 2017 +0800
Committer: Sylvain Lebresne 
Committed: Wed Sep 20 15:09:49 2017 +0200

--
 CHANGES.txt |   1 +
 .../columniterator/AbstractSSTableIterator.java |   7 -
 .../db/columniterator/SSTableIterator.java  |   8 +-
 .../columniterator/SSTableReversedIterator.java |   2 +-
 .../org/apache/cassandra/cql3/ViewTest.java |  49 +++
 .../db/SinglePartitionSliceCommandTest.java | 145 ++-
 6 files changed, 197 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/975c3d81/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 74e70e1..2d11a3e 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Fix sstable reader to support range-tombstone-marker for multi-slices 
(CASSANDRA-13787)
  * Fix short read protection for tables with no clustering columns 
(CASSANDRA-13880)
  * Make isBuilt volatile in PartitionUpdate (CASSANDRA-13619)
  * Prevent integer overflow of timestamps in CellTest and RowsTest 
(CASSANDRA-13866)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/975c3d81/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
--
diff --git 
a/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java 
b/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
index c61b6aa..f9e6545 100644
--- 
a/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
+++ 
b/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
@@ -329,13 +329,6 @@ abstract class AbstractSSTableIterator implements 
SliceableUnfilteredRowIterator
 openMarker = marker.isOpen(false) ? marker.openDeletionTime(false) 
: null;
 }
 
-protected DeletionTime getAndClearOpenMarker()
-{
-DeletionTime toReturn = openMarker;
-openMarker = null;
-return toReturn;
-}
-
 public boolean hasNext() 
 {
 try

http://git-wip-us.apache.org/repos/asf/cassandra/blob/975c3d81/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
--
diff --git 
a/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java 
b/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
index ff91871..47f85ac 100644
--- a/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
+++ b/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
@@ -173,14 +173,14 @@ public class SSTableIterator extends 
AbstractSSTableIterator
 if (next != null)
 return true;
 
-// If we have an open marker, we should close it before finishing
+// for current slice, no data read from deserialization
+sliceDone = true;
+// If we have an open marker, we should not close it, there could 
be more slices
 if (openMarker != null)
 {
-next = new RangeTombstoneBoundMarker(end, 
getAndClearOpenMarker());
+next = new RangeTombstoneBoundMarker(end, openMarker);
 return true;
 }
-
-sliceDone = true; // not absolutely necessary but accurate and 
cheap
 return false;
 }
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/975c3d81/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
--
diff --git 
a/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java 
b/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
index b12ed67..76d8c4d 100644
--- 
a/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
+++

[3/6] cassandra git commit: Fix sstable reader to support range-tombstone-marker for multi-slices

2017-09-20 Thread slebresne

Fix sstable reader to support range-tombstone-marker for multi-slices

patch by Zhao Yang; reviewed by Sylvain Lebresne for CASSANDRA-13787


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/975c3d81
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/975c3d81
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/975c3d81

Branch: refs/heads/trunk
Commit: 975c3d81b67e9c1e1dcefdda3f90e8edf6be5efa
Parents: 35e32f2
Author: Zhao Yang 
Authored: Wed Aug 23 16:15:25 2017 +0800
Committer: Sylvain Lebresne 
Committed: Wed Sep 20 15:09:49 2017 +0200

--
 CHANGES.txt |   1 +
 .../columniterator/AbstractSSTableIterator.java |   7 -
 .../db/columniterator/SSTableIterator.java  |   8 +-
 .../columniterator/SSTableReversedIterator.java |   2 +-
 .../org/apache/cassandra/cql3/ViewTest.java |  49 +++
 .../db/SinglePartitionSliceCommandTest.java | 145 ++-
 6 files changed, 197 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/975c3d81/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 74e70e1..2d11a3e 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Fix sstable reader to support range-tombstone-marker for multi-slices 
(CASSANDRA-13787)
  * Fix short read protection for tables with no clustering columns 
(CASSANDRA-13880)
  * Make isBuilt volatile in PartitionUpdate (CASSANDRA-13619)
  * Prevent integer overflow of timestamps in CellTest and RowsTest 
(CASSANDRA-13866)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/975c3d81/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
--
diff --git 
a/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java 
b/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
index c61b6aa..f9e6545 100644
--- 
a/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
+++ 
b/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
@@ -329,13 +329,6 @@ abstract class AbstractSSTableIterator implements 
SliceableUnfilteredRowIterator
 openMarker = marker.isOpen(false) ? marker.openDeletionTime(false) 
: null;
 }
 
-protected DeletionTime getAndClearOpenMarker()
-{
-DeletionTime toReturn = openMarker;
-openMarker = null;
-return toReturn;
-}
-
 public boolean hasNext() 
 {
 try

http://git-wip-us.apache.org/repos/asf/cassandra/blob/975c3d81/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
--
diff --git 
a/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java 
b/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
index ff91871..47f85ac 100644
--- a/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
+++ b/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
@@ -173,14 +173,14 @@ public class SSTableIterator extends 
AbstractSSTableIterator
 if (next != null)
 return true;
 
-// If we have an open marker, we should close it before finishing
+// for current slice, no data read from deserialization
+sliceDone = true;
+// If we have an open marker, we should not close it, there could 
be more slices
 if (openMarker != null)
 {
-next = new RangeTombstoneBoundMarker(end, 
getAndClearOpenMarker());
+next = new RangeTombstoneBoundMarker(end, openMarker);
 return true;
 }
-
-sliceDone = true; // not absolutely necessary but accurate and 
cheap
 return false;
 }
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/975c3d81/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
--
diff --git 
a/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java 
b/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
index b12ed67..76d8c4d 100644
--- 
a/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
+++ 
b/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
@@ -240,7 +240,7 @@ public class SSTableReversedIterator extends 
AbstractSSTableIterator
 // skipFirstIteratedItem (this is the last item of the block, 
but

[5/6] cassandra git commit: Merge commit '975c3d81b67e9c1e1dcefdda3f90e8edf6be5efa' into cassandra-3.11

2017-09-20 Thread slebresne

Merge commit '975c3d81b67e9c1e1dcefdda3f90e8edf6be5efa' into cassandra-3.11

* commit '975c3d81b67e9c1e1dcefdda3f90e8edf6be5efa':
  Fix sstable reader to support range-tombstone-marker for multi-slices


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/66115139
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/66115139
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/66115139

Branch: refs/heads/cassandra-3.11
Commit: 66115139addfb2bb6e26fa85e4225a1178d2e99c
Parents: 594f1c1 975c3d8
Author: Sylvain Lebresne 
Authored: Wed Sep 20 15:11:58 2017 +0200
Committer: Sylvain Lebresne 
Committed: Wed Sep 20 15:12:40 2017 +0200

--
 CHANGES.txt |   1 +
 .../columniterator/AbstractSSTableIterator.java |   7 -
 .../db/columniterator/SSTableIterator.java  |   8 +-
 .../columniterator/SSTableReversedIterator.java |   2 +-
 .../org/apache/cassandra/cql3/ViewTest.java |  49 +++
 .../db/SinglePartitionSliceCommandTest.java | 145 ++-
 6 files changed, 197 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/66115139/CHANGES.txt
--
diff --cc CHANGES.txt
index 01955f6,2d11a3e..39270e5
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,15 -1,5 +1,16 @@@
 -3.0.15
 +3.11.1
 + * AbstractTokenTreeBuilder#serializedSize returns wrong value when there is 
a single leaf and overflow collisions (CASSANDRA-13869)
 + * Add a compaction option to TWCS to ignore sstables overlapping checks 
(CASSANDRA-13418)
 + * BTree.Builder memory leak (CASSANDRA-13754)
 + * Revert CASSANDRA-10368 of supporting non-pk column filtering due to 
correctness (CASSANDRA-13798)
 + * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
 + * Better bootstrap failure message when blocked by (potential) range 
movement (CASSANDRA-13744)
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
+  * Fix sstable reader to support range-tombstone-marker for multi-slices 
(CASSANDRA-13787)
   * Fix short read protection for tables with no clustering columns 
(CASSANDRA-13880)
   * Make isBuilt volatile in PartitionUpdate (CASSANDRA-13619)
   * Prevent integer overflow of timestamps in CellTest and RowsTest 
(CASSANDRA-13866)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/66115139/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
--
diff --cc 
src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
index b6c60fe,f9e6545..c15416f
--- 
a/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
+++ 
b/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
@@@ -371,14 -329,7 +371,7 @@@ public abstract class AbstractSSTableIt
  openMarker = marker.isOpen(false) ? 
marker.openDeletionTime(false) : null;
  }
  
- protected DeletionTime getAndClearOpenMarker()
- {
- DeletionTime toReturn = openMarker;
- openMarker = null;
- return toReturn;
- }
- 
 -public boolean hasNext() 
 +public boolean hasNext()
  {
  try
  {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/66115139/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/66115139/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
--
diff --cc 
src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
index 88d415b,76d8c4d..cf8798d
--- 
a/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
+++ 
b/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
@@@ -249,8 -239,8 +249,8 @@@ public class SSTableReversedIterator ex
  // not breaking ImmutableBTreePartition, we should skip it 
when returning from the iterator, hence the
  // skipFirstIteratedItem (this is the last item of the block, 
but we're iterating in reverse order so it will
  // be the first returned by the iterator).
 -RangeTombstone.Bound markerEnd = end == null ?

[6/6] cassandra git commit: Merge commit '66115139addfb2bb6e26fa85e4225a1178d2e99c' into trunk

2017-09-20 Thread slebresne

Merge commit '66115139addfb2bb6e26fa85e4225a1178d2e99c' into trunk

* commit '66115139addfb2bb6e26fa85e4225a1178d2e99c':
  Fix sstable reader to support range-tombstone-marker for multi-slices


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9a624748
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9a624748
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9a624748

Branch: refs/heads/trunk
Commit: 9a62474822149bca358d7535e6dd7210ca17277e
Parents: eb76692 6611513
Author: Sylvain Lebresne 
Authored: Wed Sep 20 15:16:53 2017 +0200
Committer: Sylvain Lebresne 
Committed: Wed Sep 20 15:17:32 2017 +0200

--
 CHANGES.txt |   1 +
 .../columniterator/AbstractSSTableIterator.java |   7 -
 .../db/columniterator/SSTableIterator.java  |   8 +-
 .../columniterator/SSTableReversedIterator.java |   2 +-
 .../org/apache/cassandra/cql3/ViewTest.java |  49 +++
 .../db/SinglePartitionSliceCommandTest.java | 143 ++-
 6 files changed, 195 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9a624748/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9a624748/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9a624748/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9a624748/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9a624748/test/unit/org/apache/cassandra/cql3/ViewTest.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9a624748/test/unit/org/apache/cassandra/db/SinglePartitionSliceCommandTest.java
--
diff --cc test/unit/org/apache/cassandra/db/SinglePartitionSliceCommandTest.java
index f79066b,7ad6198..d03d3bc
--- a/test/unit/org/apache/cassandra/db/SinglePartitionSliceCommandTest.java
+++ b/test/unit/org/apache/cassandra/db/SinglePartitionSliceCommandTest.java
@@@ -20,9 -20,12 +20,10 @@@
   */
  package org.apache.cassandra.db;
  
+ import static org.junit.Assert.*;
+ 
  import java.io.IOException;
 -import java.nio.ByteBuffer;
 -import java.util.Collections;
  import java.util.Iterator;
- 
  import org.junit.Assert;
  import org.junit.Before;
  import org.junit.BeforeClass;
@@@ -30,14 -33,17 +31,17 @@@ import org.junit.Test
  
  import org.slf4j.Logger;
  import org.slf4j.LoggerFactory;
- 
  import org.apache.cassandra.SchemaLoader;
 -import org.apache.cassandra.Util;
 -import org.apache.cassandra.config.CFMetaData;
 -import org.apache.cassandra.config.ColumnDefinition;
 +import org.apache.cassandra.schema.ColumnMetadata;
 +import org.apache.cassandra.schema.TableMetadata;
  import org.apache.cassandra.config.DatabaseDescriptor;
 -import org.apache.cassandra.config.Schema;
 +import org.apache.cassandra.schema.Schema;
  import org.apache.cassandra.cql3.ColumnIdentifier;
++import org.apache.cassandra.Util;
  import org.apache.cassandra.cql3.QueryProcessor;
+ import org.apache.cassandra.cql3.UntypedResultSet;
+ import org.apache.cassandra.db.filter.AbstractClusteringIndexFilter;
+ import org.apache.cassandra.db.filter.ClusteringIndexNamesFilter;
  import org.apache.cassandra.db.filter.ClusteringIndexSliceFilter;
  import org.apache.cassandra.db.filter.ColumnFilter;
  import org.apache.cassandra.db.filter.DataLimits;
@@@ -64,33 -74,221 +72,162 @@@ public class SinglePartitionSliceComman
  private static final String KEYSPACE = "ks";
  private static final String TABLE = "tbl";
  
 -private static CFMetaData cfm;
 -private static ColumnDefinition v;
 -private static ColumnDefinition s;
 +private static TableMetadata metadata;
 +private static ColumnMetadata v;
 +private static ColumnMetadata s;
  
+ private static final String TABLE_SCLICES = "tbl_slices";
 -private static CFMetaData CFM_SLICES;
++private static TableMetadata CFM_SLICES;
+ 
  @BeforeClass
  public static void defineSchema() throws ConfigurationException
  {
  DatabaseDescriptor.daemonInitialization();
  
 -cfm = CFMetaData.Builder.create(KEYSPACE, TABLE)
 -.addPartitionKey("k", UTF8Type.instance)
 -

[4/6] cassandra git commit: Merge commit '975c3d81b67e9c1e1dcefdda3f90e8edf6be5efa' into cassandra-3.11

2017-09-20 Thread slebresne

Merge commit '975c3d81b67e9c1e1dcefdda3f90e8edf6be5efa' into cassandra-3.11

* commit '975c3d81b67e9c1e1dcefdda3f90e8edf6be5efa':
  Fix sstable reader to support range-tombstone-marker for multi-slices


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/66115139
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/66115139
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/66115139

Branch: refs/heads/trunk
Commit: 66115139addfb2bb6e26fa85e4225a1178d2e99c
Parents: 594f1c1 975c3d8
Author: Sylvain Lebresne 
Authored: Wed Sep 20 15:11:58 2017 +0200
Committer: Sylvain Lebresne 
Committed: Wed Sep 20 15:12:40 2017 +0200

--
 CHANGES.txt |   1 +
 .../columniterator/AbstractSSTableIterator.java |   7 -
 .../db/columniterator/SSTableIterator.java  |   8 +-
 .../columniterator/SSTableReversedIterator.java |   2 +-
 .../org/apache/cassandra/cql3/ViewTest.java |  49 +++
 .../db/SinglePartitionSliceCommandTest.java | 145 ++-
 6 files changed, 197 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/66115139/CHANGES.txt
--
diff --cc CHANGES.txt
index 01955f6,2d11a3e..39270e5
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,15 -1,5 +1,16 @@@
 -3.0.15
 +3.11.1
 + * AbstractTokenTreeBuilder#serializedSize returns wrong value when there is 
a single leaf and overflow collisions (CASSANDRA-13869)
 + * Add a compaction option to TWCS to ignore sstables overlapping checks 
(CASSANDRA-13418)
 + * BTree.Builder memory leak (CASSANDRA-13754)
 + * Revert CASSANDRA-10368 of supporting non-pk column filtering due to 
correctness (CASSANDRA-13798)
 + * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
 + * Better bootstrap failure message when blocked by (potential) range 
movement (CASSANDRA-13744)
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
+  * Fix sstable reader to support range-tombstone-marker for multi-slices 
(CASSANDRA-13787)
   * Fix short read protection for tables with no clustering columns 
(CASSANDRA-13880)
   * Make isBuilt volatile in PartitionUpdate (CASSANDRA-13619)
   * Prevent integer overflow of timestamps in CellTest and RowsTest 
(CASSANDRA-13866)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/66115139/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
--
diff --cc 
src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
index b6c60fe,f9e6545..c15416f
--- 
a/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
+++ 
b/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
@@@ -371,14 -329,7 +371,7 @@@ public abstract class AbstractSSTableIt
  openMarker = marker.isOpen(false) ? 
marker.openDeletionTime(false) : null;
  }
  
- protected DeletionTime getAndClearOpenMarker()
- {
- DeletionTime toReturn = openMarker;
- openMarker = null;
- return toReturn;
- }
- 
 -public boolean hasNext() 
 +public boolean hasNext()
  {
  try
  {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/66115139/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/66115139/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
--
diff --cc 
src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
index 88d415b,76d8c4d..cf8798d
--- 
a/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
+++ 
b/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
@@@ -249,8 -239,8 +249,8 @@@ public class SSTableReversedIterator ex
  // not breaking ImmutableBTreePartition, we should skip it 
when returning from the iterator, hence the
  // skipFirstIteratedItem (this is the last item of the block, 
but we're iterating in reverse order so it will
  // be the first returned by the iterator).
 -RangeTombstone.Bound markerEnd = end == null ?

[2/6] cassandra git commit: Fix sstable reader to support range-tombstone-marker for multi-slices

2017-09-20 Thread slebresne

Fix sstable reader to support range-tombstone-marker for multi-slices

patch by Zhao Yang; reviewed by Sylvain Lebresne for CASSANDRA-13787


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/975c3d81
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/975c3d81
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/975c3d81

Branch: refs/heads/cassandra-3.11
Commit: 975c3d81b67e9c1e1dcefdda3f90e8edf6be5efa
Parents: 35e32f2
Author: Zhao Yang 
Authored: Wed Aug 23 16:15:25 2017 +0800
Committer: Sylvain Lebresne 
Committed: Wed Sep 20 15:09:49 2017 +0200

--
 CHANGES.txt |   1 +
 .../columniterator/AbstractSSTableIterator.java |   7 -
 .../db/columniterator/SSTableIterator.java  |   8 +-
 .../columniterator/SSTableReversedIterator.java |   2 +-
 .../org/apache/cassandra/cql3/ViewTest.java |  49 +++
 .../db/SinglePartitionSliceCommandTest.java | 145 ++-
 6 files changed, 197 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/975c3d81/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 74e70e1..2d11a3e 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Fix sstable reader to support range-tombstone-marker for multi-slices 
(CASSANDRA-13787)
  * Fix short read protection for tables with no clustering columns 
(CASSANDRA-13880)
  * Make isBuilt volatile in PartitionUpdate (CASSANDRA-13619)
  * Prevent integer overflow of timestamps in CellTest and RowsTest 
(CASSANDRA-13866)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/975c3d81/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
--
diff --git 
a/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java 
b/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
index c61b6aa..f9e6545 100644
--- 
a/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
+++ 
b/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java
@@ -329,13 +329,6 @@ abstract class AbstractSSTableIterator implements 
SliceableUnfilteredRowIterator
 openMarker = marker.isOpen(false) ? marker.openDeletionTime(false) 
: null;
 }
 
-protected DeletionTime getAndClearOpenMarker()
-{
-DeletionTime toReturn = openMarker;
-openMarker = null;
-return toReturn;
-}
-
 public boolean hasNext() 
 {
 try

http://git-wip-us.apache.org/repos/asf/cassandra/blob/975c3d81/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
--
diff --git 
a/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java 
b/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
index ff91871..47f85ac 100644
--- a/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
+++ b/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
@@ -173,14 +173,14 @@ public class SSTableIterator extends 
AbstractSSTableIterator
 if (next != null)
 return true;
 
-// If we have an open marker, we should close it before finishing
+// for current slice, no data read from deserialization
+sliceDone = true;
+// If we have an open marker, we should not close it, there could 
be more slices
 if (openMarker != null)
 {
-next = new RangeTombstoneBoundMarker(end, 
getAndClearOpenMarker());
+next = new RangeTombstoneBoundMarker(end, openMarker);
 return true;
 }
-
-sliceDone = true; // not absolutely necessary but accurate and 
cheap
 return false;
 }
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/975c3d81/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
--
diff --git 
a/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java 
b/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
index b12ed67..76d8c4d 100644
--- 
a/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
+++ 
b/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
@@ -240,7 +240,7 @@ public class SSTableReversedIterator extends 
AbstractSSTableIterator
 // skipFirstIteratedItem (this is the last item of the

[2/6] cassandra git commit: Potential AssertionError during ReadRepair of range tombstone and partition deletions

2017-08-24 Thread slebresne

Potential AssertionError during ReadRepair of range tombstone and partition 
deletions

patch by Sylvain Lebresne; reviewed by Branimir Lambov for CASSANDRA-13719


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5e57dd14
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5e57dd14
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5e57dd14

Branch: refs/heads/cassandra-3.11
Commit: 5e57dd14eb37adf06d2105227e0105d871ea6f76
Parents: 6a1b1f2
Author: Sylvain Lebresne 
Authored: Fri Jul 21 16:58:53 2017 +0200
Committer: Sylvain Lebresne 
Committed: Thu Aug 24 11:28:57 2017 +0200

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/db/ReadResponse.java   | 22 +
 .../db/partitions/AbstractBTreePartition.java   | 14 +--
 .../db/partitions/PartitionUpdate.java  |  9 ++
 .../apache/cassandra/service/DataResolver.java  | 70 +--
 .../cassandra/service/DataResolverTest.java | 91 +++-
 6 files changed, 192 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5e57dd14/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 97dda05..2b49bc3 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Potential AssertionError during ReadRepair of range tombstone and partition 
deletions (CASSANDRA-13719)
  * Don't let stress write warmup data if n=0 (CASSANDRA-13773)
  * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
  * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5e57dd14/src/java/org/apache/cassandra/db/ReadResponse.java
--
diff --git a/src/java/org/apache/cassandra/db/ReadResponse.java 
b/src/java/org/apache/cassandra/db/ReadResponse.java
index 693b52b..c59d00a 100644
--- a/src/java/org/apache/cassandra/db/ReadResponse.java
+++ b/src/java/org/apache/cassandra/db/ReadResponse.java
@@ -92,6 +92,28 @@ public abstract class ReadResponse
 
 public abstract boolean isDigestResponse();
 
+/**
+ * Creates a string of the requested partition in this read response 
suitable for debugging.
+ */
+public String toDebugString(ReadCommand command, DecoratedKey key)
+{
+if (isDigestResponse())
+return "Digest:0x" + ByteBufferUtil.bytesToHex(digest(command));
+
+try (UnfilteredPartitionIterator iter = makeIterator(command))
+{
+while (iter.hasNext())
+{
+try (UnfilteredRowIterator partition = iter.next())
+{
+if (partition.partitionKey().equals(key))
+return 
ImmutableBTreePartition.create(partition).toString();
+}
+}
+}
+return "";
+}
+
 protected static ByteBuffer makeDigest(UnfilteredPartitionIterator 
iterator, ReadCommand command)
 {
 MessageDigest digest = FBUtilities.threadLocalMD5Digest();

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5e57dd14/src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java
--
diff --git 
a/src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java 
b/src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java
index 2aa622e..1f3dbd0 100644
--- a/src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java
+++ b/src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java
@@ -99,7 +99,7 @@ public abstract class AbstractBTreePartition implements 
Partition, Iterable
 
 public DeletionTime partitionLevelDeletion()
 {
-return holder().deletionInfo.getPartitionDeletion();
+return deletionInfo().getPartitionDeletion();
 }
 
 public PartitionColumns columns()
@@ -372,17 +372,21 @@ public abstract class AbstractBTreePartition implements 
Partition, Iterable
 {
 StringBuilder sb = new StringBuilder();
 
-sb.append(String.format("[%s.%s] key=%s columns=%s",
+sb.append(String.format("[%s.%s] key=%s partition_deletion=%s 
columns=%s",
 metadata.ksName,
 metadata.cfName,
 
metadata.getKeyValidator().getString(partitionKey().getKey()),
+partitionLevelDeletion(),
 columns()));
 
 if (staticRow() != Rows.EMPTY_STATIC_ROW)
-sb.append("\n").append(staticRow().toString(metadata));
+

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-08-24 Thread slebresne

Merge branch 'cassandra-3.0' into cassandra-3.11

* cassandra-3.0:
  Potential AssertionError during ReadRepair of range tombstone and partition 
deletions


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ed9b04d6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ed9b04d6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ed9b04d6

Branch: refs/heads/trunk
Commit: ed9b04d6aeffc85a2dbcd8c136df769491f79d85
Parents: 59d4c27 5e57dd1
Author: Sylvain Lebresne 
Authored: Thu Aug 24 11:36:14 2017 +0200
Committer: Sylvain Lebresne 
Committed: Thu Aug 24 11:36:14 2017 +0200

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/db/ReadResponse.java   | 22 +
 .../db/partitions/AbstractBTreePartition.java   | 14 +--
 .../db/partitions/PartitionUpdate.java  |  9 ++
 .../apache/cassandra/service/DataResolver.java  | 70 +--
 .../cassandra/service/DataResolverTest.java | 91 +++-
 6 files changed, 192 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ed9b04d6/CHANGES.txt
--
diff --cc CHANGES.txt
index 9e42ffb,2b49bc3..b22cb5e
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,11 -1,5 +1,12 @@@
 -3.0.15
 +3.11.1
 + * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
 + * Better bootstrap failure message when blocked by (potential) range 
movement (CASSANDRA-13744)
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
+  * Potential AssertionError during ReadRepair of range tombstone and 
partition deletions (CASSANDRA-13719)
   * Don't let stress write warmup data if n=0 (CASSANDRA-13773)
   * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
   * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ed9b04d6/src/java/org/apache/cassandra/db/ReadResponse.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ed9b04d6/src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ed9b04d6/src/java/org/apache/cassandra/db/partitions/PartitionUpdate.java
--
diff --cc src/java/org/apache/cassandra/db/partitions/PartitionUpdate.java
index 64d7dd1,7bd5345..435c324
--- a/src/java/org/apache/cassandra/db/partitions/PartitionUpdate.java
+++ b/src/java/org/apache/cassandra/db/partitions/PartitionUpdate.java
@@@ -314,9 -296,18 +314,18 @@@ public class PartitionUpdate extends Ab
  
  int nowInSecs = FBUtilities.nowInSeconds();
  List asIterators = Lists.transform(updates, 
AbstractBTreePartition::unfilteredIterator);
 -return fromIterator(UnfilteredRowIterators.merge(asIterators, 
nowInSecs));
 +return fromIterator(UnfilteredRowIterators.merge(asIterators, 
nowInSecs), ColumnFilter.all(updates.get(0).metadata()));
  }
  
+ // We override this, because the version in the super-class calls 
holder(), which build the update preventing
+ // further updates, but that's not necessary here and being able to check 
at least the partition deletion without
+ // "locking" the update is nice (and used in 
DataResolver.RepairMergeListener.MergeListener).
+ @Override
+ public DeletionInfo deletionInfo()
+ {
+ return deletionInfo;
+ }
+ 
  /**
   * Modify this update to set every timestamp for live data to {@code 
newTimestamp} and
   * every deletion timestamp to {@code newTimestamp - 1}.

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ed9b04d6/src/java/org/apache/cassandra/service/DataResolver.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ed9b04d6/test/unit/org/apache/cassandra/service/DataResolverTest.java
--
diff --cc test/unit/org/apache/cassandra/service/DataResolverTest.java
index 8e4f385,65e18ce..2b1e095
--- a/test/unit/org/apache/cassandra/service/DataResolverTest.java
+++ b/test/unit/org/apache/cassandra/service/DataResolverTest.java
@@@ -576,9 -574,9 +576,9 @@@ public

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2017-08-24 Thread slebresne

Merge branch 'cassandra-3.11' into trunk

* cassandra-3.11:
  Potential AssertionError during ReadRepair of range tombstone and partition 
deletions


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/652d9f64
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/652d9f64
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/652d9f64

Branch: refs/heads/trunk
Commit: 652d9f64f14d8375a8412561271a7abf27722f20
Parents: d2dcd7f ed9b04d
Author: Sylvain Lebresne 
Authored: Thu Aug 24 11:39:35 2017 +0200
Committer: Sylvain Lebresne 
Committed: Thu Aug 24 11:39:35 2017 +0200

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/db/ReadResponse.java   | 22 +
 .../db/partitions/AbstractBTreePartition.java   | 16 ++--
 .../db/partitions/PartitionUpdate.java  |  9 ++
 .../apache/cassandra/service/DataResolver.java  | 70 +--
 .../cassandra/service/DataResolverTest.java | 91 +++-
 6 files changed, 193 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/652d9f64/CHANGES.txt
--
diff --cc CHANGES.txt
index d0ec78d,b22cb5e..e19e9dd
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -129,7 -6,9 +129,8 @@@
   * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
   * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
  Merged from 3.0:
+  * Potential AssertionError during ReadRepair of range tombstone and 
partition deletions (CASSANDRA-13719)
   * Don't let stress write warmup data if n=0 (CASSANDRA-13773)
 - * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
   * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
   * Fix digest calculation for counter cells (CASSANDRA-13750)
   * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/652d9f64/src/java/org/apache/cassandra/db/ReadResponse.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/652d9f64/src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java
--
diff --cc 
src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java
index d8f2856,90db062..d913cb3
--- a/src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java
+++ b/src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java
@@@ -94,10 -99,10 +94,10 @@@ public abstract class AbstractBTreePart
  
  public DeletionTime partitionLevelDeletion()
  {
- return holder().deletionInfo.getPartitionDeletion();
+ return deletionInfo().getPartitionDeletion();
  }
  
 -public PartitionColumns columns()
 +public RegularAndStaticColumns columns()
  {
  return holder().columns;
  }
@@@ -317,16 -322,21 +317,20 @@@
  {
  StringBuilder sb = new StringBuilder();
  
- sb.append(String.format("[%s] key=%s columns=%s",
- metadata().toString(),
 -sb.append(String.format("[%s.%s] key=%s partition_deletion=%s 
columns=%s",
 -metadata.ksName,
 -metadata.cfName,
 -
metadata.getKeyValidator().getString(partitionKey().getKey()),
++sb.append(String.format("[%s] key=%s partition_deletion=%s 
columns=%s",
++metadata(),
 +
metadata().partitionKeyType.getString(partitionKey().getKey()),
+ partitionLevelDeletion(),
  columns()));
  
  if (staticRow() != Rows.EMPTY_STATIC_ROW)
- sb.append("\n").append(staticRow().toString(metadata()));
 -sb.append("\n").append(staticRow().toString(metadata, true));
++sb.append("\n").append(staticRow().toString(metadata(), 
true));
  
- for (Row row : this)
- sb.append("\n").append(row.toString(metadata()));
+ try (UnfilteredRowIterator iter = unfilteredIterator())
+ {
+ while (iter.hasNext())
 -sb.append("\n").append(iter.next().toString(metadata, 
true));
++sb.append("\n").append(iter.next().toString(metadata(), 
true));
+ }
  
  return sb.toString();
  }

[1/6] cassandra git commit: Potential AssertionError during ReadRepair of range tombstone and partition deletions

2017-08-24 Thread slebresne

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 6a1b1f26b -> 5e57dd14e
  refs/heads/cassandra-3.11 59d4c2719 -> ed9b04d6a
  refs/heads/trunk d2dcd7f88 -> 652d9f64f


Potential AssertionError during ReadRepair of range tombstone and partition 
deletions

patch by Sylvain Lebresne; reviewed by Branimir Lambov for CASSANDRA-13719


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5e57dd14
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5e57dd14
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5e57dd14

Branch: refs/heads/cassandra-3.0
Commit: 5e57dd14eb37adf06d2105227e0105d871ea6f76
Parents: 6a1b1f2
Author: Sylvain Lebresne 
Authored: Fri Jul 21 16:58:53 2017 +0200
Committer: Sylvain Lebresne 
Committed: Thu Aug 24 11:28:57 2017 +0200

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/db/ReadResponse.java   | 22 +
 .../db/partitions/AbstractBTreePartition.java   | 14 +--
 .../db/partitions/PartitionUpdate.java  |  9 ++
 .../apache/cassandra/service/DataResolver.java  | 70 +--
 .../cassandra/service/DataResolverTest.java | 91 +++-
 6 files changed, 192 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5e57dd14/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 97dda05..2b49bc3 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Potential AssertionError during ReadRepair of range tombstone and partition 
deletions (CASSANDRA-13719)
  * Don't let stress write warmup data if n=0 (CASSANDRA-13773)
  * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
  * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5e57dd14/src/java/org/apache/cassandra/db/ReadResponse.java
--
diff --git a/src/java/org/apache/cassandra/db/ReadResponse.java 
b/src/java/org/apache/cassandra/db/ReadResponse.java
index 693b52b..c59d00a 100644
--- a/src/java/org/apache/cassandra/db/ReadResponse.java
+++ b/src/java/org/apache/cassandra/db/ReadResponse.java
@@ -92,6 +92,28 @@ public abstract class ReadResponse
 
 public abstract boolean isDigestResponse();
 
+/**
+ * Creates a string of the requested partition in this read response 
suitable for debugging.
+ */
+public String toDebugString(ReadCommand command, DecoratedKey key)
+{
+if (isDigestResponse())
+return "Digest:0x" + ByteBufferUtil.bytesToHex(digest(command));
+
+try (UnfilteredPartitionIterator iter = makeIterator(command))
+{
+while (iter.hasNext())
+{
+try (UnfilteredRowIterator partition = iter.next())
+{
+if (partition.partitionKey().equals(key))
+return 
ImmutableBTreePartition.create(partition).toString();
+}
+}
+}
+return "";
+}
+
 protected static ByteBuffer makeDigest(UnfilteredPartitionIterator 
iterator, ReadCommand command)
 {
 MessageDigest digest = FBUtilities.threadLocalMD5Digest();

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5e57dd14/src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java
--
diff --git 
a/src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java 
b/src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java
index 2aa622e..1f3dbd0 100644
--- a/src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java
+++ b/src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java
@@ -99,7 +99,7 @@ public abstract class AbstractBTreePartition implements 
Partition, Iterable
 
 public DeletionTime partitionLevelDeletion()
 {
-return holder().deletionInfo.getPartitionDeletion();
+return deletionInfo().getPartitionDeletion();
 }
 
 public PartitionColumns columns()
@@ -372,17 +372,21 @@ public abstract class AbstractBTreePartition implements 
Partition, Iterable
 {
 StringBuilder sb = new StringBuilder();
 
-sb.append(String.format("[%s.%s] key=%s columns=%s",
+sb.append(String.format("[%s.%s] key=%s partition_deletion=%s 
columns=%s",
 metadata.ksName,
 metadata.cfName,
 
metadata.getKeyValidator().getString(partitionKey().getKey()),
+

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-08-24 Thread slebresne

Merge branch 'cassandra-3.0' into cassandra-3.11

* cassandra-3.0:
  Potential AssertionError during ReadRepair of range tombstone and partition 
deletions


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ed9b04d6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ed9b04d6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ed9b04d6

Branch: refs/heads/cassandra-3.11
Commit: ed9b04d6aeffc85a2dbcd8c136df769491f79d85
Parents: 59d4c27 5e57dd1
Author: Sylvain Lebresne 
Authored: Thu Aug 24 11:36:14 2017 +0200
Committer: Sylvain Lebresne 
Committed: Thu Aug 24 11:36:14 2017 +0200

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/db/ReadResponse.java   | 22 +
 .../db/partitions/AbstractBTreePartition.java   | 14 +--
 .../db/partitions/PartitionUpdate.java  |  9 ++
 .../apache/cassandra/service/DataResolver.java  | 70 +--
 .../cassandra/service/DataResolverTest.java | 91 +++-
 6 files changed, 192 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ed9b04d6/CHANGES.txt
--
diff --cc CHANGES.txt
index 9e42ffb,2b49bc3..b22cb5e
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,11 -1,5 +1,12 @@@
 -3.0.15
 +3.11.1
 + * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
 + * Better bootstrap failure message when blocked by (potential) range 
movement (CASSANDRA-13744)
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
+  * Potential AssertionError during ReadRepair of range tombstone and 
partition deletions (CASSANDRA-13719)
   * Don't let stress write warmup data if n=0 (CASSANDRA-13773)
   * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
   * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ed9b04d6/src/java/org/apache/cassandra/db/ReadResponse.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ed9b04d6/src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ed9b04d6/src/java/org/apache/cassandra/db/partitions/PartitionUpdate.java
--
diff --cc src/java/org/apache/cassandra/db/partitions/PartitionUpdate.java
index 64d7dd1,7bd5345..435c324
--- a/src/java/org/apache/cassandra/db/partitions/PartitionUpdate.java
+++ b/src/java/org/apache/cassandra/db/partitions/PartitionUpdate.java
@@@ -314,9 -296,18 +314,18 @@@ public class PartitionUpdate extends Ab
  
  int nowInSecs = FBUtilities.nowInSeconds();
  List asIterators = Lists.transform(updates, 
AbstractBTreePartition::unfilteredIterator);
 -return fromIterator(UnfilteredRowIterators.merge(asIterators, 
nowInSecs));
 +return fromIterator(UnfilteredRowIterators.merge(asIterators, 
nowInSecs), ColumnFilter.all(updates.get(0).metadata()));
  }
  
+ // We override this, because the version in the super-class calls 
holder(), which build the update preventing
+ // further updates, but that's not necessary here and being able to check 
at least the partition deletion without
+ // "locking" the update is nice (and used in 
DataResolver.RepairMergeListener.MergeListener).
+ @Override
+ public DeletionInfo deletionInfo()
+ {
+ return deletionInfo;
+ }
+ 
  /**
   * Modify this update to set every timestamp for live data to {@code 
newTimestamp} and
   * every deletion timestamp to {@code newTimestamp - 1}.

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ed9b04d6/src/java/org/apache/cassandra/service/DataResolver.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ed9b04d6/test/unit/org/apache/cassandra/service/DataResolverTest.java
--
diff --cc test/unit/org/apache/cassandra/service/DataResolverTest.java
index 8e4f385,65e18ce..2b1e095
--- a/test/unit/org/apache/cassandra/service/DataResolverTest.java
+++ b/test/unit/org/apache/cassandra/service/DataResolverTest.java
@@@ -576,9 -574,9 +576,9

[3/6] cassandra git commit: Potential AssertionError during ReadRepair of range tombstone and partition deletions

2017-08-24 Thread slebresne

Potential AssertionError during ReadRepair of range tombstone and partition 
deletions

patch by Sylvain Lebresne; reviewed by Branimir Lambov for CASSANDRA-13719


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5e57dd14
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5e57dd14
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5e57dd14

Branch: refs/heads/trunk
Commit: 5e57dd14eb37adf06d2105227e0105d871ea6f76
Parents: 6a1b1f2
Author: Sylvain Lebresne 
Authored: Fri Jul 21 16:58:53 2017 +0200
Committer: Sylvain Lebresne 
Committed: Thu Aug 24 11:28:57 2017 +0200

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/db/ReadResponse.java   | 22 +
 .../db/partitions/AbstractBTreePartition.java   | 14 +--
 .../db/partitions/PartitionUpdate.java  |  9 ++
 .../apache/cassandra/service/DataResolver.java  | 70 +--
 .../cassandra/service/DataResolverTest.java | 91 +++-
 6 files changed, 192 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5e57dd14/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 97dda05..2b49bc3 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Potential AssertionError during ReadRepair of range tombstone and partition 
deletions (CASSANDRA-13719)
  * Don't let stress write warmup data if n=0 (CASSANDRA-13773)
  * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
  * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5e57dd14/src/java/org/apache/cassandra/db/ReadResponse.java
--
diff --git a/src/java/org/apache/cassandra/db/ReadResponse.java 
b/src/java/org/apache/cassandra/db/ReadResponse.java
index 693b52b..c59d00a 100644
--- a/src/java/org/apache/cassandra/db/ReadResponse.java
+++ b/src/java/org/apache/cassandra/db/ReadResponse.java
@@ -92,6 +92,28 @@ public abstract class ReadResponse
 
 public abstract boolean isDigestResponse();
 
+/**
+ * Creates a string of the requested partition in this read response 
suitable for debugging.
+ */
+public String toDebugString(ReadCommand command, DecoratedKey key)
+{
+if (isDigestResponse())
+return "Digest:0x" + ByteBufferUtil.bytesToHex(digest(command));
+
+try (UnfilteredPartitionIterator iter = makeIterator(command))
+{
+while (iter.hasNext())
+{
+try (UnfilteredRowIterator partition = iter.next())
+{
+if (partition.partitionKey().equals(key))
+return 
ImmutableBTreePartition.create(partition).toString();
+}
+}
+}
+return "";
+}
+
 protected static ByteBuffer makeDigest(UnfilteredPartitionIterator 
iterator, ReadCommand command)
 {
 MessageDigest digest = FBUtilities.threadLocalMD5Digest();

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5e57dd14/src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java
--
diff --git 
a/src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java 
b/src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java
index 2aa622e..1f3dbd0 100644
--- a/src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java
+++ b/src/java/org/apache/cassandra/db/partitions/AbstractBTreePartition.java
@@ -99,7 +99,7 @@ public abstract class AbstractBTreePartition implements 
Partition, Iterable
 
 public DeletionTime partitionLevelDeletion()
 {
-return holder().deletionInfo.getPartitionDeletion();
+return deletionInfo().getPartitionDeletion();
 }
 
 public PartitionColumns columns()
@@ -372,17 +372,21 @@ public abstract class AbstractBTreePartition implements 
Partition, Iterable
 {
 StringBuilder sb = new StringBuilder();
 
-sb.append(String.format("[%s.%s] key=%s columns=%s",
+sb.append(String.format("[%s.%s] key=%s partition_deletion=%s 
columns=%s",
 metadata.ksName,
 metadata.cfName,
 
metadata.getKeyValidator().getString(partitionKey().getKey()),
+partitionLevelDeletion(),
 columns()));
 
 if (staticRow() != Rows.EMPTY_STATIC_ROW)
-sb.append("\n").append(staticRow().toString(metadata));
+

[3/6] cassandra git commit: Legacy deserializer can create empty range tombstones

2017-03-29 Thread slebresne

Legacy deserializer can create empty range tombstones

patch by Sylvain Lebresne; reviewed by Branimir Lambov for CASSANDRA-13341


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/451fe9d8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/451fe9d8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/451fe9d8

Branch: refs/heads/trunk
Commit: 451fe9d8ac567942f62852f542d28d7d1116f1a1
Parents: 6edc268
Author: Sylvain Lebresne 
Authored: Thu Mar 16 17:25:39 2017 +0100
Committer: Sylvain Lebresne 
Committed: Wed Mar 29 13:17:58 2017 +0200

--
 CHANGES.txt |   1 +
 .../cassandra/db/UnfilteredDeserializer.java| 124 ++-
 .../db/rows/RangeTombstoneBoundMarker.java  |   8 ++
 .../db/rows/RangeTombstoneBoundaryMarker.java   |   5 +
 .../cassandra/db/rows/RangeTombstoneMarker.java |   2 +
 .../cassandra/db/OldFormatDeserializerTest.java |  54 
 6 files changed, 162 insertions(+), 32 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/451fe9d8/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index b46eb50..c4293de 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.13
+ * Legacy deserializer can create empty range tombstones (CASSANDRA-13341)
  * Use the Kernel32 library to retrieve the PID on Windows and fix startup 
checks (CASSANDRA-1)
  * Fix code to not exchange schema across major versions (CASSANDRA-13274)
  * Dropping column results in "corrupt" SSTable (CASSANDRA-13337)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/451fe9d8/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
--
diff --git a/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java 
b/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
index 7bbbfdb..92690e1 100644
--- a/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
+++ b/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
@@ -26,8 +26,6 @@ import com.google.common.annotations.VisibleForTesting;
 import com.google.common.collect.Iterables;
 import com.google.common.collect.PeekingIterator;
 
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
 import org.apache.cassandra.config.CFMetaData;
 import org.apache.cassandra.db.rows.*;
 import org.apache.cassandra.io.util.DataInputPlus;
@@ -43,8 +41,6 @@ import org.apache.cassandra.net.MessagingService;
  */
 public abstract class UnfilteredDeserializer
 {
-private static final Logger logger = 
LoggerFactory.getLogger(UnfilteredDeserializer.class);
-
 protected final CFMetaData metadata;
 protected final DataInputPlus in;
 protected final SerializationHelper helper;
@@ -433,21 +429,31 @@ public abstract class UnfilteredDeserializer
 {
 if (atoms.hasNext())
 {
+// If there is a range tombstone to open strictly 
before the next row/RT, we need to return that open (or boundary) marker first.
+if 
(tombstoneTracker.hasOpeningMarkerBefore(atoms.peek()))
+{
+next = tombstoneTracker.popOpeningMarker();
+}
 // If a range tombstone closes strictly before the 
next row/RT, we need to return that close (or boundary) marker first.
-if 
(tombstoneTracker.hasClosingMarkerBefore(atoms.peek()))
+else if 
(tombstoneTracker.hasClosingMarkerBefore(atoms.peek()))
 {
 next = tombstoneTracker.popClosingMarker();
 }
 else
 {
 LegacyLayout.LegacyAtom atom = atoms.next();
-if (!tombstoneTracker.isShadowed(atom))
-next = isRow(atom) ? readRow(atom) : 
tombstoneTracker.openNew(atom.asRangeTombstone());
+if (tombstoneTracker.isShadowed(atom))
+continue;
+
+if (isRow(atom))
+next = readRow(atom);
+else
+
tombstoneTracker.openNew(atom.asRangeTombstone());
 }
 }
 else if (tombstoneTracker.hasOpenTombstones())
 {
-next = tombstoneTracker.popClosingMarker();
+next = tombstoneTracker.popMarker();
 }

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2017-03-29 Thread slebresne

Merge branch 'cassandra-3.11' into trunk

* cassandra-3.11:
  Legacy deserializer can create empty range tombstones


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/07795f10
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/07795f10
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/07795f10

Branch: refs/heads/trunk
Commit: 07795f1016970ac519503293883c1f7013f3f48a
Parents: ea708c4 580963a
Author: Sylvain Lebresne 
Authored: Wed Mar 29 13:24:39 2017 +0200
Committer: Sylvain Lebresne 
Committed: Wed Mar 29 13:24:39 2017 +0200

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/db/UnfilteredDeserializer.java | 5 -
 .../apache/cassandra/db/rows/RangeTombstoneBoundMarker.java  | 8 
 .../cassandra/db/rows/RangeTombstoneBoundaryMarker.java  | 5 +
 .../org/apache/cassandra/db/rows/RangeTombstoneMarker.java   | 2 ++
 5 files changed, 16 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/07795f10/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/07795f10/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
--
diff --cc src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
index d361bac,05e8148..84ff691
--- a/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
+++ b/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
@@@ -18,13 -18,19 +18,10 @@@
  package org.apache.cassandra.db;
  
  import java.io.IOException;
 -import java.io.IOError;
 -import java.util.*;
 -import java.util.function.Supplier;
  
- import org.slf4j.Logger;
- import org.slf4j.LoggerFactory;
 -import com.google.common.annotations.VisibleForTesting;
 -import com.google.common.collect.Iterables;
 -import com.google.common.collect.PeekingIterator;
--
 -import org.apache.cassandra.config.CFMetaData;
 +import org.apache.cassandra.schema.TableMetadata;
  import org.apache.cassandra.db.rows.*;
  import org.apache.cassandra.io.util.DataInputPlus;
 -import org.apache.cassandra.io.util.FileDataInput;
 -import org.apache.cassandra.net.MessagingService;
  
  /**
   * Helper class to deserialize Unfiltered object from disk efficiently.
@@@ -33,11 -39,9 +30,9 @@@
   * we don't do more work than necessary (i.e. we don't allocate/deserialize
   * objects for things we don't care about).
   */
 -public abstract class UnfilteredDeserializer
 +public class UnfilteredDeserializer
  {
- private static final Logger logger = 
LoggerFactory.getLogger(UnfilteredDeserializer.class);
- 
 -protected final CFMetaData metadata;
 +protected final TableMetadata metadata;
  protected final DataInputPlus in;
  protected final SerializationHelper helper;
  

http://git-wip-us.apache.org/repos/asf/cassandra/blob/07795f10/src/java/org/apache/cassandra/db/rows/RangeTombstoneBoundMarker.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/07795f10/src/java/org/apache/cassandra/db/rows/RangeTombstoneBoundaryMarker.java
--

[2/6] cassandra git commit: Legacy deserializer can create empty range tombstones

2017-03-29 Thread slebresne

Legacy deserializer can create empty range tombstones

patch by Sylvain Lebresne; reviewed by Branimir Lambov for CASSANDRA-13341


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/451fe9d8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/451fe9d8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/451fe9d8

Branch: refs/heads/cassandra-3.11
Commit: 451fe9d8ac567942f62852f542d28d7d1116f1a1
Parents: 6edc268
Author: Sylvain Lebresne 
Authored: Thu Mar 16 17:25:39 2017 +0100
Committer: Sylvain Lebresne 
Committed: Wed Mar 29 13:17:58 2017 +0200

--
 CHANGES.txt |   1 +
 .../cassandra/db/UnfilteredDeserializer.java| 124 ++-
 .../db/rows/RangeTombstoneBoundMarker.java  |   8 ++
 .../db/rows/RangeTombstoneBoundaryMarker.java   |   5 +
 .../cassandra/db/rows/RangeTombstoneMarker.java |   2 +
 .../cassandra/db/OldFormatDeserializerTest.java |  54 
 6 files changed, 162 insertions(+), 32 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/451fe9d8/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index b46eb50..c4293de 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.13
+ * Legacy deserializer can create empty range tombstones (CASSANDRA-13341)
  * Use the Kernel32 library to retrieve the PID on Windows and fix startup 
checks (CASSANDRA-1)
  * Fix code to not exchange schema across major versions (CASSANDRA-13274)
  * Dropping column results in "corrupt" SSTable (CASSANDRA-13337)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/451fe9d8/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
--
diff --git a/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java 
b/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
index 7bbbfdb..92690e1 100644
--- a/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
+++ b/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
@@ -26,8 +26,6 @@ import com.google.common.annotations.VisibleForTesting;
 import com.google.common.collect.Iterables;
 import com.google.common.collect.PeekingIterator;
 
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
 import org.apache.cassandra.config.CFMetaData;
 import org.apache.cassandra.db.rows.*;
 import org.apache.cassandra.io.util.DataInputPlus;
@@ -43,8 +41,6 @@ import org.apache.cassandra.net.MessagingService;
  */
 public abstract class UnfilteredDeserializer
 {
-private static final Logger logger = 
LoggerFactory.getLogger(UnfilteredDeserializer.class);
-
 protected final CFMetaData metadata;
 protected final DataInputPlus in;
 protected final SerializationHelper helper;
@@ -433,21 +429,31 @@ public abstract class UnfilteredDeserializer
 {
 if (atoms.hasNext())
 {
+// If there is a range tombstone to open strictly 
before the next row/RT, we need to return that open (or boundary) marker first.
+if 
(tombstoneTracker.hasOpeningMarkerBefore(atoms.peek()))
+{
+next = tombstoneTracker.popOpeningMarker();
+}
 // If a range tombstone closes strictly before the 
next row/RT, we need to return that close (or boundary) marker first.
-if 
(tombstoneTracker.hasClosingMarkerBefore(atoms.peek()))
+else if 
(tombstoneTracker.hasClosingMarkerBefore(atoms.peek()))
 {
 next = tombstoneTracker.popClosingMarker();
 }
 else
 {
 LegacyLayout.LegacyAtom atom = atoms.next();
-if (!tombstoneTracker.isShadowed(atom))
-next = isRow(atom) ? readRow(atom) : 
tombstoneTracker.openNew(atom.asRangeTombstone());
+if (tombstoneTracker.isShadowed(atom))
+continue;
+
+if (isRow(atom))
+next = readRow(atom);
+else
+
tombstoneTracker.openNew(atom.asRangeTombstone());
 }
 }
 else if (tombstoneTracker.hasOpenTombstones())
 {
-next = tombstoneTracker.popClosingMarker();
+next = tombstoneTracker.popMarker();
 }

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-03-29 Thread slebresne

Merge branch 'cassandra-3.0' into cassandra-3.11

* cassandra-3.0:
  Legacy deserializer can create empty range tombstones


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/580963a2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/580963a2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/580963a2

Branch: refs/heads/trunk
Commit: 580963a2184b8f70f319ba59353fd258de239e36
Parents: bd77081 451fe9d
Author: Sylvain Lebresne 
Authored: Wed Mar 29 13:22:29 2017 +0200
Committer: Sylvain Lebresne 
Committed: Wed Mar 29 13:22:29 2017 +0200

--
 CHANGES.txt |   1 +
 .../cassandra/db/UnfilteredDeserializer.java| 119 ++-
 .../db/rows/RangeTombstoneBoundMarker.java  |   8 ++
 .../db/rows/RangeTombstoneBoundaryMarker.java   |   5 +
 .../cassandra/db/rows/RangeTombstoneMarker.java |   2 +
 .../cassandra/db/OldFormatDeserializerTest.java |  54 +
 6 files changed, 162 insertions(+), 27 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/580963a2/CHANGES.txt
--
diff --cc CHANGES.txt
index 9cd1ca7,c4293de..81f4688
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,24 -1,5 +1,25 @@@
 -3.0.13
 +3.11.0
 + * cdc column addition strikes again (CASSANDRA-13382)
 + * Fix static column indexes (CASSANDRA-13277) 
 + * DataOutputBuffer.asNewBuffer broken (CASSANDRA-13298)
 + * unittest CipherFactoryTest failed on MacOS (CASSANDRA-13370)
 + * Forbid SELECT restrictions and CREATE INDEX over non-frozen UDT columns 
(CASSANDRA-13247)
 + * Default logging we ship will incorrectly print "?:?" for "%F:%L" pattern 
(CASSANDRA-13317)
 + * Possible AssertionError in UnfilteredRowIteratorWithLowerBound 
(CASSANDRA-13366)
 + * Support unaligned memory access for AArch64 (CASSANDRA-13326)
 + * Improve SASI range iterator efficiency on intersection with an empty range 
(CASSANDRA-12915).
 + * Fix equality comparisons of columns using the duration type 
(CASSANDRA-13174)
 + * Obfuscate password in stress-graphs (CASSANDRA-12233)
 + * Move to FastThreadLocalThread and FastThreadLocal (CASSANDRA-13034)
 + * nodetool stopdaemon errors out (CASSANDRA-13030)
 + * Tables in system_distributed should not use gcgs of 0 (CASSANDRA-12954)
 + * Fix primary index calculation for SASI (CASSANDRA-12910)
 + * More fixes to the TokenAllocator (CASSANDRA-12990)
 + * NoReplicationTokenAllocator should work with zero replication factor 
(CASSANDRA-12983)
 + * Address message coalescing regression (CASSANDRA-12676)
 +Merged from 3.0:
+  * Legacy deserializer can create empty range tombstones (CASSANDRA-13341)
 + * Legacy caching options can prevent 3.0 upgrade (CASSANDRA-13384)
   * Use the Kernel32 library to retrieve the PID on Windows and fix startup 
checks (CASSANDRA-1)
   * Fix code to not exchange schema across major versions (CASSANDRA-13274)
   * Dropping column results in "corrupt" SSTable (CASSANDRA-13337)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/580963a2/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/580963a2/src/java/org/apache/cassandra/db/rows/RangeTombstoneBoundMarker.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/580963a2/src/java/org/apache/cassandra/db/rows/RangeTombstoneBoundaryMarker.java
--
diff --cc 
src/java/org/apache/cassandra/db/rows/RangeTombstoneBoundaryMarker.java
index fd41bea,0683d76..70d6a9d
--- a/src/java/org/apache/cassandra/db/rows/RangeTombstoneBoundaryMarker.java
+++ b/src/java/org/apache/cassandra/db/rows/RangeTombstoneBoundaryMarker.java
@@@ -120,9 -120,14 +120,14 @@@ public class RangeTombstoneBoundaryMark
  return new RangeTombstoneBoundaryMarker(clustering().copy(allocator), 
endDeletion, startDeletion);
  }
  
+ public RangeTombstoneBoundaryMarker withNewOpeningDeletionTime(boolean 
reversed, DeletionTime newDeletionTime)
+ {
+ return new RangeTombstoneBoundaryMarker(clustering(), reversed ? 
newDeletionTime : endDeletion, reversed ? startDeletion : newDeletionTime);
+ }
+ 
 -public static RangeTombstoneBoundaryMarker makeBoundary(boolean reversed, 
Slice.Bound close, Slice.Bound open, DeletionTime closeDeletion, DeletionTime 
openDeletion)
 +public static RangeTombstoneBoundaryMarker makeBoundary(boolean reversed, 
ClusteringBound close, ClusteringBound open, DeletionTime closeDeletion, 
DeletionTime openDeletion)
  {
 -assert

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-03-29 Thread slebresne

Merge branch 'cassandra-3.0' into cassandra-3.11

* cassandra-3.0:
  Legacy deserializer can create empty range tombstones


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/580963a2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/580963a2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/580963a2

Branch: refs/heads/cassandra-3.11
Commit: 580963a2184b8f70f319ba59353fd258de239e36
Parents: bd77081 451fe9d
Author: Sylvain Lebresne 
Authored: Wed Mar 29 13:22:29 2017 +0200
Committer: Sylvain Lebresne 
Committed: Wed Mar 29 13:22:29 2017 +0200

--
 CHANGES.txt |   1 +
 .../cassandra/db/UnfilteredDeserializer.java| 119 ++-
 .../db/rows/RangeTombstoneBoundMarker.java  |   8 ++
 .../db/rows/RangeTombstoneBoundaryMarker.java   |   5 +
 .../cassandra/db/rows/RangeTombstoneMarker.java |   2 +
 .../cassandra/db/OldFormatDeserializerTest.java |  54 +
 6 files changed, 162 insertions(+), 27 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/580963a2/CHANGES.txt
--
diff --cc CHANGES.txt
index 9cd1ca7,c4293de..81f4688
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,24 -1,5 +1,25 @@@
 -3.0.13
 +3.11.0
 + * cdc column addition strikes again (CASSANDRA-13382)
 + * Fix static column indexes (CASSANDRA-13277) 
 + * DataOutputBuffer.asNewBuffer broken (CASSANDRA-13298)
 + * unittest CipherFactoryTest failed on MacOS (CASSANDRA-13370)
 + * Forbid SELECT restrictions and CREATE INDEX over non-frozen UDT columns 
(CASSANDRA-13247)
 + * Default logging we ship will incorrectly print "?:?" for "%F:%L" pattern 
(CASSANDRA-13317)
 + * Possible AssertionError in UnfilteredRowIteratorWithLowerBound 
(CASSANDRA-13366)
 + * Support unaligned memory access for AArch64 (CASSANDRA-13326)
 + * Improve SASI range iterator efficiency on intersection with an empty range 
(CASSANDRA-12915).
 + * Fix equality comparisons of columns using the duration type 
(CASSANDRA-13174)
 + * Obfuscate password in stress-graphs (CASSANDRA-12233)
 + * Move to FastThreadLocalThread and FastThreadLocal (CASSANDRA-13034)
 + * nodetool stopdaemon errors out (CASSANDRA-13030)
 + * Tables in system_distributed should not use gcgs of 0 (CASSANDRA-12954)
 + * Fix primary index calculation for SASI (CASSANDRA-12910)
 + * More fixes to the TokenAllocator (CASSANDRA-12990)
 + * NoReplicationTokenAllocator should work with zero replication factor 
(CASSANDRA-12983)
 + * Address message coalescing regression (CASSANDRA-12676)
 +Merged from 3.0:
+  * Legacy deserializer can create empty range tombstones (CASSANDRA-13341)
 + * Legacy caching options can prevent 3.0 upgrade (CASSANDRA-13384)
   * Use the Kernel32 library to retrieve the PID on Windows and fix startup 
checks (CASSANDRA-1)
   * Fix code to not exchange schema across major versions (CASSANDRA-13274)
   * Dropping column results in "corrupt" SSTable (CASSANDRA-13337)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/580963a2/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/580963a2/src/java/org/apache/cassandra/db/rows/RangeTombstoneBoundMarker.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/580963a2/src/java/org/apache/cassandra/db/rows/RangeTombstoneBoundaryMarker.java
--
diff --cc 
src/java/org/apache/cassandra/db/rows/RangeTombstoneBoundaryMarker.java
index fd41bea,0683d76..70d6a9d
--- a/src/java/org/apache/cassandra/db/rows/RangeTombstoneBoundaryMarker.java
+++ b/src/java/org/apache/cassandra/db/rows/RangeTombstoneBoundaryMarker.java
@@@ -120,9 -120,14 +120,14 @@@ public class RangeTombstoneBoundaryMark
  return new RangeTombstoneBoundaryMarker(clustering().copy(allocator), 
endDeletion, startDeletion);
  }
  
+ public RangeTombstoneBoundaryMarker withNewOpeningDeletionTime(boolean 
reversed, DeletionTime newDeletionTime)
+ {
+ return new RangeTombstoneBoundaryMarker(clustering(), reversed ? 
newDeletionTime : endDeletion, reversed ? startDeletion : newDeletionTime);
+ }
+ 
 -public static RangeTombstoneBoundaryMarker makeBoundary(boolean reversed, 
Slice.Bound close, Slice.Bound open, DeletionTime closeDeletion, DeletionTime 
openDeletion)
 +public static RangeTombstoneBoundaryMarker makeBoundary(boolean reversed, 
ClusteringBound close, ClusteringBound open, DeletionTime closeDeletion, 
DeletionTime openDeletion)
  {
 -assert

[1/6] cassandra git commit: Legacy deserializer can create empty range tombstones

2017-03-29 Thread slebresne

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 6edc26824 -> 451fe9d8a
  refs/heads/cassandra-3.11 bd7708192 -> 580963a21
  refs/heads/trunk ea708c47e -> 07795f101


Legacy deserializer can create empty range tombstones

patch by Sylvain Lebresne; reviewed by Branimir Lambov for CASSANDRA-13341


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/451fe9d8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/451fe9d8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/451fe9d8

Branch: refs/heads/cassandra-3.0
Commit: 451fe9d8ac567942f62852f542d28d7d1116f1a1
Parents: 6edc268
Author: Sylvain Lebresne 
Authored: Thu Mar 16 17:25:39 2017 +0100
Committer: Sylvain Lebresne 
Committed: Wed Mar 29 13:17:58 2017 +0200

--
 CHANGES.txt |   1 +
 .../cassandra/db/UnfilteredDeserializer.java| 124 ++-
 .../db/rows/RangeTombstoneBoundMarker.java  |   8 ++
 .../db/rows/RangeTombstoneBoundaryMarker.java   |   5 +
 .../cassandra/db/rows/RangeTombstoneMarker.java |   2 +
 .../cassandra/db/OldFormatDeserializerTest.java |  54 
 6 files changed, 162 insertions(+), 32 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/451fe9d8/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index b46eb50..c4293de 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.13
+ * Legacy deserializer can create empty range tombstones (CASSANDRA-13341)
  * Use the Kernel32 library to retrieve the PID on Windows and fix startup 
checks (CASSANDRA-1)
  * Fix code to not exchange schema across major versions (CASSANDRA-13274)
  * Dropping column results in "corrupt" SSTable (CASSANDRA-13337)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/451fe9d8/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
--
diff --git a/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java 
b/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
index 7bbbfdb..92690e1 100644
--- a/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
+++ b/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
@@ -26,8 +26,6 @@ import com.google.common.annotations.VisibleForTesting;
 import com.google.common.collect.Iterables;
 import com.google.common.collect.PeekingIterator;
 
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
 import org.apache.cassandra.config.CFMetaData;
 import org.apache.cassandra.db.rows.*;
 import org.apache.cassandra.io.util.DataInputPlus;
@@ -43,8 +41,6 @@ import org.apache.cassandra.net.MessagingService;
  */
 public abstract class UnfilteredDeserializer
 {
-private static final Logger logger = 
LoggerFactory.getLogger(UnfilteredDeserializer.class);
-
 protected final CFMetaData metadata;
 protected final DataInputPlus in;
 protected final SerializationHelper helper;
@@ -433,21 +429,31 @@ public abstract class UnfilteredDeserializer
 {
 if (atoms.hasNext())
 {
+// If there is a range tombstone to open strictly 
before the next row/RT, we need to return that open (or boundary) marker first.
+if 
(tombstoneTracker.hasOpeningMarkerBefore(atoms.peek()))
+{
+next = tombstoneTracker.popOpeningMarker();
+}
 // If a range tombstone closes strictly before the 
next row/RT, we need to return that close (or boundary) marker first.
-if 
(tombstoneTracker.hasClosingMarkerBefore(atoms.peek()))
+else if 
(tombstoneTracker.hasClosingMarkerBefore(atoms.peek()))
 {
 next = tombstoneTracker.popClosingMarker();
 }
 else
 {
 LegacyLayout.LegacyAtom atom = atoms.next();
-if (!tombstoneTracker.isShadowed(atom))
-next = isRow(atom) ? readRow(atom) : 
tombstoneTracker.openNew(atom.asRangeTombstone());
+if (tombstoneTracker.isShadowed(atom))
+continue;
+
+if (isRow(atom))
+next = readRow(atom);
+else
+
tombstoneTracker.openNew(atom.asRangeTombstone());
 }
 }
 else if (tombstoneTracker.hasOpenTombstones())

[1/3] cassandra git commit: cdc column addition strikes again

2017-03-28 Thread slebresne

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.11 41befde22 -> 8ea0afa3d
  refs/heads/trunk 380a614f1 -> 3bd152e6c


cdc column addition strikes again

patch by Sylvain Lebresne; reviewed by Aleksey Yeschenko for CASSANDRA-13382


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8ea0afa3
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8ea0afa3
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8ea0afa3

Branch: refs/heads/cassandra-3.11
Commit: 8ea0afa3dbced228dbe9332bae6b5b2a1760bd8a
Parents: 41befde
Author: Sylvain Lebresne 
Authored: Mon Mar 27 11:27:28 2017 +0200
Committer: Sylvain Lebresne 
Committed: Tue Mar 28 14:35:00 2017 +0200

--
 CHANGES.txt  |  1 +
 src/java/org/apache/cassandra/schema/SchemaKeyspace.java | 11 +--
 2 files changed, 10 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/8ea0afa3/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 217..c7cc26e 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.11.0
+ * cdc column addition strikes again (CASSANDRA-13382)
  * Fix static column indexes (CASSANDRA-13277) 
  * DataOutputBuffer.asNewBuffer broken (CASSANDRA-13298)
  * unittest CipherFactoryTest failed on MacOS (CASSANDRA-13370)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/8ea0afa3/src/java/org/apache/cassandra/schema/SchemaKeyspace.java
--
diff --git a/src/java/org/apache/cassandra/schema/SchemaKeyspace.java 
b/src/java/org/apache/cassandra/schema/SchemaKeyspace.java
index 58580b9..ff7cf04 100644
--- a/src/java/org/apache/cassandra/schema/SchemaKeyspace.java
+++ b/src/java/org/apache/cassandra/schema/SchemaKeyspace.java
@@ -26,6 +26,7 @@ import java.util.concurrent.TimeUnit;
 import java.util.stream.Collectors;
 
 import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.MapDifference;
 import com.google.common.collect.Maps;
 import org.slf4j.Logger;
@@ -84,6 +85,12 @@ public final class SchemaKeyspace
 public static final List ALL =
 ImmutableList.of(KEYSPACES, TABLES, COLUMNS, DROPPED_COLUMNS, 
TRIGGERS, VIEWS, TYPES, FUNCTIONS, AGGREGATES, INDEXES);
 
+/**
+ * The tables to which we added the cdc column. This is used in {@link 
#makeUpdateForSchema} below to make sure we skip that
+ * column is cdc is disabled as the columns breaks pre-cdc to post-cdc 
upgrades (typically, 3.0 -> 3.X).
+ */
+private static final Set TABLES_WITH_CDC_ADDED = 
ImmutableSet.of(TABLES, VIEWS);
+
 private static final CFMetaData Keyspaces =
 compile(KEYSPACES,
 "keyspace definitions",
@@ -385,9 +392,9 @@ public final class SchemaKeyspace
 private static PartitionUpdate makeUpdateForSchema(UnfilteredRowIterator 
partition, ColumnFilter filter)
 {
 // This method is used during schema migration tasks, and if cdc is 
disabled, we want to force excluding the
-// 'cdc' column from the TABLES schema table because it is problematic 
if received by older nodes (see #12236
+// 'cdc' column from the TABLES/VIEWS schema table because it is 
problematic if received by older nodes (see #12236
 // and #12697). Otherwise though, we just simply "buffer" the content 
of the partition into a PartitionUpdate.
-if (DatabaseDescriptor.isCDCEnabled() || 
!partition.metadata().cfName.equals(TABLES))
+if (DatabaseDescriptor.isCDCEnabled() || 
!TABLES_WITH_CDC_ADDED.contains(partition.metadata().cfName))
 return PartitionUpdate.fromIterator(partition, filter);
 
 // We want to skip the 'cdc' column. A simple solution for that is 
based on the fact that

[3/3] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2017-03-28 Thread slebresne

Merge branch 'cassandra-3.11' into trunk

* cassandra-3.11:
  cdc column addition strikes again


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3bd152e6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3bd152e6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3bd152e6

Branch: refs/heads/trunk
Commit: 3bd152e6cc82f5314ec14ea854ebbf6ad602609f
Parents: 380a614 8ea0afa
Author: Sylvain Lebresne 
Authored: Tue Mar 28 14:38:17 2017 +0200
Committer: Sylvain Lebresne 
Committed: Tue Mar 28 14:38:17 2017 +0200

--
 CHANGES.txt  |  1 +
 src/java/org/apache/cassandra/schema/SchemaKeyspace.java | 11 +--
 2 files changed, 10 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3bd152e6/CHANGES.txt
--
diff --cc CHANGES.txt
index a9f0c06,c7cc26e..7ec782c
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,52 -1,5 +1,53 @@@
 +4.0
 + * Outbound TCP connections ignore internode authenticator (CASSANDRA-13324)
 + * Upgrade junit from 4.6 to 4.12 (CASSANDRA-13360)
 + * Cleanup ParentRepairSession after repairs (CASSANDRA-13359)
 + * Incremental repair not streaming correct sstables (CASSANDRA-13328)
 + * Upgrade the jna version to 4.3.0 (CASSANDRA-13300)
 + * Add the currentTimestamp, currentDate, currentTime and currentTimeUUID 
functions (CASSANDRA-13132)
 + * Remove config option index_interval (CASSANDRA-10671)
 + * Reduce lock contention for collection types and serializers 
(CASSANDRA-13271)
 + * Make it possible to override MessagingService.Verb ids (CASSANDRA-13283)
 + * Avoid synchronized on prepareForRepair in ActiveRepairService 
(CASSANDRA-9292)
 + * Adds the ability to use uncompressed chunks in compressed files 
(CASSANDRA-10520)
 + * Don't flush sstables when streaming for incremental repair 
(CASSANDRA-13226)
 + * Remove unused method (CASSANDRA-13227)
 + * Fix minor bugs related to #9143 (CASSANDRA-13217)
 + * Output warning if user increases RF (CASSANDRA-13079)
 + * Remove pre-3.0 streaming compatibility code for 4.0 (CASSANDRA-13081)
 + * Add support for + and - operations on dates (CASSANDRA-11936)
 + * Fix consistency of incrementally repaired data (CASSANDRA-9143)
 + * Increase commitlog version (CASSANDRA-13161)
 + * Make TableMetadata immutable, optimize Schema (CASSANDRA-9425)
 + * Refactor ColumnCondition (CASSANDRA-12981)
 + * Parallelize streaming of different keyspaces (CASSANDRA-4663)
 + * Improved compactions metrics (CASSANDRA-13015)
 + * Speed-up start-up sequence by avoiding un-needed flushes (CASSANDRA-13031)
 + * Use Caffeine (W-TinyLFU) for on-heap caches (CASSANDRA-10855)
 + * Thrift removal (CASSANDRA-5)
 + * Remove pre-3.0 compatibility code for 4.0 (CASSANDRA-12716)
 + * Add column definition kind to dropped columns in schema (CASSANDRA-12705)
 + * Add (automate) Nodetool Documentation (CASSANDRA-12672)
 + * Update bundled cqlsh python driver to 3.7.0 (CASSANDRA-12736)
 + * Reject invalid replication settings when creating or altering a keyspace 
(CASSANDRA-12681)
 + * Clean up the SSTableReader#getScanner API wrt removal of RateLimiter 
(CASSANDRA-12422)
 + * Use new token allocation for non bootstrap case as well (CASSANDRA-13080)
 + * Avoid byte-array copy when key cache is disabled (CASSANDRA-13084)
 + * Require forceful decommission if number of nodes is less than replication 
factor (CASSANDRA-12510)
 + * Allow IN restrictions on column families with collections (CASSANDRA-12654)
 + * Log message size in trace message in OutboundTcpConnection 
(CASSANDRA-13028)
 + * Add timeUnit Days for cassandra-stress (CASSANDRA-13029)
 + * Add mutation size and batch metrics (CASSANDRA-12649)
 + * Add method to get size of endpoints to TokenMetadata (CASSANDRA-12999)
 + * Expose time spent waiting in thread pool queue (CASSANDRA-8398)
 + * Conditionally update index built status to avoid unnecessary flushes 
(CASSANDRA-12969)
 + * cqlsh auto completion: refactor definition of compaction strategy options 
(CASSANDRA-12946)
 + * Add support for arithmetic operators (CASSANDRA-11935)
 + * Add histogram for delay to deliver hints (CASSANDRA-13234)
 +
 +
  3.11.0
+  * cdc column addition strikes again (CASSANDRA-13382)
   * Fix static column indexes (CASSANDRA-13277) 
   * DataOutputBuffer.asNewBuffer broken (CASSANDRA-13298)
   * unittest CipherFactoryTest failed on MacOS (CASSANDRA-13370)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3bd152e6/src/java/org/apache/cassandra/schema/SchemaKeyspace.java
--
diff --cc src/java/org/apache/cassandra/schema/SchemaKeyspace.java
index

[2/3] cassandra git commit: cdc column addition strikes again

2017-03-28 Thread slebresne

cdc column addition strikes again

patch by Sylvain Lebresne; reviewed by Aleksey Yeschenko for CASSANDRA-13382


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8ea0afa3
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8ea0afa3
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8ea0afa3

Branch: refs/heads/trunk
Commit: 8ea0afa3dbced228dbe9332bae6b5b2a1760bd8a
Parents: 41befde
Author: Sylvain Lebresne 
Authored: Mon Mar 27 11:27:28 2017 +0200
Committer: Sylvain Lebresne 
Committed: Tue Mar 28 14:35:00 2017 +0200

--
 CHANGES.txt  |  1 +
 src/java/org/apache/cassandra/schema/SchemaKeyspace.java | 11 +--
 2 files changed, 10 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/8ea0afa3/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 217..c7cc26e 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.11.0
+ * cdc column addition strikes again (CASSANDRA-13382)
  * Fix static column indexes (CASSANDRA-13277) 
  * DataOutputBuffer.asNewBuffer broken (CASSANDRA-13298)
  * unittest CipherFactoryTest failed on MacOS (CASSANDRA-13370)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/8ea0afa3/src/java/org/apache/cassandra/schema/SchemaKeyspace.java
--
diff --git a/src/java/org/apache/cassandra/schema/SchemaKeyspace.java 
b/src/java/org/apache/cassandra/schema/SchemaKeyspace.java
index 58580b9..ff7cf04 100644
--- a/src/java/org/apache/cassandra/schema/SchemaKeyspace.java
+++ b/src/java/org/apache/cassandra/schema/SchemaKeyspace.java
@@ -26,6 +26,7 @@ import java.util.concurrent.TimeUnit;
 import java.util.stream.Collectors;
 
 import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.MapDifference;
 import com.google.common.collect.Maps;
 import org.slf4j.Logger;
@@ -84,6 +85,12 @@ public final class SchemaKeyspace
 public static final List ALL =
 ImmutableList.of(KEYSPACES, TABLES, COLUMNS, DROPPED_COLUMNS, 
TRIGGERS, VIEWS, TYPES, FUNCTIONS, AGGREGATES, INDEXES);
 
+/**
+ * The tables to which we added the cdc column. This is used in {@link 
#makeUpdateForSchema} below to make sure we skip that
+ * column is cdc is disabled as the columns breaks pre-cdc to post-cdc 
upgrades (typically, 3.0 -> 3.X).
+ */
+private static final Set TABLES_WITH_CDC_ADDED = 
ImmutableSet.of(TABLES, VIEWS);
+
 private static final CFMetaData Keyspaces =
 compile(KEYSPACES,
 "keyspace definitions",
@@ -385,9 +392,9 @@ public final class SchemaKeyspace
 private static PartitionUpdate makeUpdateForSchema(UnfilteredRowIterator 
partition, ColumnFilter filter)
 {
 // This method is used during schema migration tasks, and if cdc is 
disabled, we want to force excluding the
-// 'cdc' column from the TABLES schema table because it is problematic 
if received by older nodes (see #12236
+// 'cdc' column from the TABLES/VIEWS schema table because it is 
problematic if received by older nodes (see #12236
 // and #12697). Otherwise though, we just simply "buffer" the content 
of the partition into a PartitionUpdate.
-if (DatabaseDescriptor.isCDCEnabled() || 
!partition.metadata().cfName.equals(TABLES))
+if (DatabaseDescriptor.isCDCEnabled() || 
!TABLES_WITH_CDC_ADDED.contains(partition.metadata().cfName))
 return PartitionUpdate.fromIterator(partition, filter);
 
 // We want to skip the 'cdc' column. A simple solution for that is 
based on the fact that

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2017-03-27 Thread slebresne

Merge branch 'cassandra-3.11' into trunk

* cassandra-3.11:
  Dropping column results in "corrupt" SSTable


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3dabeeaa
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3dabeeaa
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3dabeeaa

Branch: refs/heads/trunk
Commit: 3dabeeaa29bfb6757e79f17031cb467b7b052fcd
Parents: 23cd27f c0ac928
Author: Sylvain Lebresne 
Authored: Mon Mar 27 12:04:20 2017 +0200
Committer: Sylvain Lebresne 
Committed: Mon Mar 27 12:04:20 2017 +0200

--
 CHANGES.txt |  1 +
 .../db/columniterator/SSTableIterator.java  | 78 
 .../columniterator/SSTableReversedIterator.java |  4 +-
 .../cassandra/db/rows/RangeTombstoneMarker.java |  6 ++
 .../apache/cassandra/db/rows/Unfiltered.java|  2 +
 .../cassandra/db/rows/UnfilteredSerializer.java | 43 +--
 .../cql3/validation/operations/AlterTest.java   | 51 +
 7 files changed, 145 insertions(+), 40 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3dabeeaa/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3dabeeaa/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3dabeeaa/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
--
diff --cc 
src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
index e67d164,88d415b..837562a
--- 
a/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
+++ 
b/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
@@@ -223,8 -228,12 +223,10 @@@ public class SSTableReversedIterator ex
 && !stopReadingDisk())
  {
  Unfiltered unfiltered = deserializer.readNext();
- buffer.add(unfiltered);
+ // We may get empty row for the same reason expressed on 
UnfilteredSerializer.deserializeOne.
 -if (!unfiltered.isEmpty() && (!isFirst || includeFirst))
++if (!unfiltered.isEmpty())
+ buffer.add(unfiltered);
  
 -isFirst = false;
 -
  if (unfiltered.isRangeTombstoneMarker())
  updateOpenMarker((RangeTombstoneMarker)unfiltered);
  }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3dabeeaa/src/java/org/apache/cassandra/db/rows/Unfiltered.java
--
diff --cc src/java/org/apache/cassandra/db/rows/Unfiltered.java
index 39d9e75,e75c632..e1158e2
--- a/src/java/org/apache/cassandra/db/rows/Unfiltered.java
+++ b/src/java/org/apache/cassandra/db/rows/Unfiltered.java
@@@ -53,11 -53,13 +53,13 @@@ public interface Unfiltered extends Clu
   * invalid (some value is invalid for its column type, or some field
   * is nonsensical).
   */
 -public void validateData(CFMetaData metadata);
 +public void validateData(TableMetadata metadata);
  
+ public boolean isEmpty();
+ 
 -public String toString(CFMetaData metadata);
 -public String toString(CFMetaData metadata, boolean fullDetails);
 -public String toString(CFMetaData metadata, boolean includeClusterKeys, 
boolean fullDetails);
 +public String toString(TableMetadata metadata);
 +public String toString(TableMetadata metadata, boolean fullDetails);
 +public String toString(TableMetadata metadata, boolean 
includeClusterKeys, boolean fullDetails);
  
  default boolean isRow()
  {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3dabeeaa/src/java/org/apache/cassandra/db/rows/UnfilteredSerializer.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3dabeeaa/test/unit/org/apache/cassandra/cql3/validation/operations/AlterTest.java
--

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-03-27 Thread slebresne

Merge branch 'cassandra-3.0' into cassandra-3.11

* cassandra-3.0:
  Dropping column results in "corrupt" SSTable


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c0ac928d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c0ac928d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c0ac928d

Branch: refs/heads/trunk
Commit: c0ac928d985a59d00310fed124c5a2b3655db8e0
Parents: ee7023e 5262bb1
Author: Sylvain Lebresne 
Authored: Mon Mar 27 11:59:06 2017 +0200
Committer: Sylvain Lebresne 
Committed: Mon Mar 27 11:59:06 2017 +0200

--
 CHANGES.txt |  1 +
 .../db/columniterator/SSTableIterator.java  | 78 
 .../columniterator/SSTableReversedIterator.java |  3 +-
 .../cassandra/db/rows/RangeTombstoneMarker.java |  6 ++
 .../apache/cassandra/db/rows/Unfiltered.java|  2 +
 .../cassandra/db/rows/UnfilteredSerializer.java | 43 +--
 .../cql3/validation/operations/AlterTest.java   | 51 +
 7 files changed, 144 insertions(+), 40 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0ac928d/CHANGES.txt
--
diff --cc CHANGES.txt
index 071dd1a,0b1bb01..98ed992
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,20 -1,5 +1,21 @@@
 -3.0.13
 +3.11.0
 + * unittest CipherFactoryTest failed on MacOS (CASSANDRA-13370)
 + * Forbid SELECT restrictions and CREATE INDEX over non-frozen UDT columns 
(CASSANDRA-13247)
 + * Default logging we ship will incorrectly print "?:?" for "%F:%L" pattern 
(CASSANDRA-13317)
 + * Possible AssertionError in UnfilteredRowIteratorWithLowerBound 
(CASSANDRA-13366)
 + * Support unaligned memory access for AArch64 (CASSANDRA-13326)
 + * Improve SASI range iterator efficiency on intersection with an empty range 
(CASSANDRA-12915).
 + * Fix equality comparisons of columns using the duration type 
(CASSANDRA-13174)
 + * Obfuscate password in stress-graphs (CASSANDRA-12233)
 + * Move to FastThreadLocalThread and FastThreadLocal (CASSANDRA-13034)
 + * nodetool stopdaemon errors out (CASSANDRA-13030)
 + * Tables in system_distributed should not use gcgs of 0 (CASSANDRA-12954)
 + * Fix primary index calculation for SASI (CASSANDRA-12910)
 + * More fixes to the TokenAllocator (CASSANDRA-12990)
 + * NoReplicationTokenAllocator should work with zero replication factor 
(CASSANDRA-12983)
 + * Address message coalescing regression (CASSANDRA-12676)
 +Merged from 3.0:
+  * Dropping column results in "corrupt" SSTable (CASSANDRA-13337)
   * Bugs handling range tombstones in the sstable iterators (CASSANDRA-13340)
   * Fix CONTAINS filtering for null collections (CASSANDRA-13246)
   * Applying: Use a unique metric reservoir per test run when using 
Cassandra-wide metrics residing in MBeans (CASSANDRA-13216)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0ac928d/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0ac928d/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0ac928d/src/java/org/apache/cassandra/db/rows/RangeTombstoneMarker.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0ac928d/src/java/org/apache/cassandra/db/rows/Unfiltered.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0ac928d/src/java/org/apache/cassandra/db/rows/UnfilteredSerializer.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0ac928d/test/unit/org/apache/cassandra/cql3/validation/operations/AlterTest.java
--

[2/6] cassandra git commit: Dropping column results in "corrupt" SSTable

2017-03-27 Thread slebresne

Dropping column results in "corrupt" SSTable

patch by Sylvain Lebresne; reviewed by Alex Petrov for CASSANDRA-13337


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5262bb17
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5262bb17
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5262bb17

Branch: refs/heads/cassandra-3.11
Commit: 5262bb17b46fc8c02f9f836ddf9317d0de2698cd
Parents: 6311622
Author: Sylvain Lebresne 
Authored: Mon Mar 20 15:49:27 2017 +0100
Committer: Sylvain Lebresne 
Committed: Mon Mar 27 11:58:32 2017 +0200

--
 CHANGES.txt |  1 +
 .../db/columniterator/SSTableIterator.java  | 78 
 .../columniterator/SSTableReversedIterator.java |  3 +-
 .../cassandra/db/rows/RangeTombstoneMarker.java |  6 ++
 .../apache/cassandra/db/rows/Unfiltered.java|  2 +
 .../cassandra/db/rows/UnfilteredSerializer.java | 43 +--
 .../cql3/validation/operations/AlterTest.java   | 51 +
 7 files changed, 144 insertions(+), 40 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5262bb17/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 2c5573a..0b1bb01 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.13
+ * Dropping column results in "corrupt" SSTable (CASSANDRA-13337)
  * Bugs handling range tombstones in the sstable iterators (CASSANDRA-13340)
  * Fix CONTAINS filtering for null collections (CASSANDRA-13246)
  * Applying: Use a unique metric reservoir per test run when using 
Cassandra-wide metrics residing in MBeans (CASSANDRA-13216)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5262bb17/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
--
diff --git 
a/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java 
b/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
index 9bcca48..fa337c0 100644
--- a/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
+++ b/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
@@ -123,20 +123,27 @@ public class SSTableIterator extends 
AbstractSSTableIterator
 {
 assert deserializer != null;
 
-// We use a same reasoning as in handlePreSliceData regarding the 
strictness of the inequality below.
-// We want to exclude deserialized unfiltered equal to end, 
because 1) we won't miss any rows since those
-// woudn't be equal to a slice bound and 2) a end bound can be 
equal to a start bound
-// (EXCL_END(x) == INCL_START(x) for instance) and in that case we 
don't want to return start bound because
-// it's fundamentally excluded. And if the bound is a  end (for a 
range tombstone), it means it's exactly
-// our slice end, but in that  case we will properly close the 
range tombstone anyway as part of our "close
-// an open marker" code in hasNextInterna
-if (!deserializer.hasNext() || deserializer.compareNextTo(end) >= 
0)
-return null;
-
-Unfiltered next = deserializer.readNext();
-if (next.kind() == Unfiltered.Kind.RANGE_TOMBSTONE_MARKER)
-updateOpenMarker((RangeTombstoneMarker)next);
-return next;
+while (true)
+{
+// We use a same reasoning as in handlePreSliceData regarding 
the strictness of the inequality below.
+// We want to exclude deserialized unfiltered equal to end, 
because 1) we won't miss any rows since those
+// woudn't be equal to a slice bound and 2) a end bound can be 
equal to a start bound
+// (EXCL_END(x) == INCL_START(x) for instance) and in that 
case we don't want to return start bound because
+// it's fundamentally excluded. And if the bound is a  end 
(for a range tombstone), it means it's exactly
+// our slice end, but in that  case we will properly close the 
range tombstone anyway as part of our "close
+// an open marker" code in hasNextInterna
+if (!deserializer.hasNext() || deserializer.compareNextTo(end) 
>= 0)
+return null;
+
+Unfiltered next = deserializer.readNext();
+// We may get empty row for the same reason expressed on 
UnfilteredSerializer.deserializeOne.
+if (next.isEmpty())
+continue;
+
+if (next.kind() == Unfiltered.Kind.RANGE_TOMBSTONE_MARKER)
+

[3/6] cassandra git commit: Dropping column results in "corrupt" SSTable

2017-03-27 Thread slebresne

Dropping column results in "corrupt" SSTable

patch by Sylvain Lebresne; reviewed by Alex Petrov for CASSANDRA-13337


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5262bb17
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5262bb17
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5262bb17

Branch: refs/heads/trunk
Commit: 5262bb17b46fc8c02f9f836ddf9317d0de2698cd
Parents: 6311622
Author: Sylvain Lebresne 
Authored: Mon Mar 20 15:49:27 2017 +0100
Committer: Sylvain Lebresne 
Committed: Mon Mar 27 11:58:32 2017 +0200

--
 CHANGES.txt |  1 +
 .../db/columniterator/SSTableIterator.java  | 78 
 .../columniterator/SSTableReversedIterator.java |  3 +-
 .../cassandra/db/rows/RangeTombstoneMarker.java |  6 ++
 .../apache/cassandra/db/rows/Unfiltered.java|  2 +
 .../cassandra/db/rows/UnfilteredSerializer.java | 43 +--
 .../cql3/validation/operations/AlterTest.java   | 51 +
 7 files changed, 144 insertions(+), 40 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5262bb17/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 2c5573a..0b1bb01 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.13
+ * Dropping column results in "corrupt" SSTable (CASSANDRA-13337)
  * Bugs handling range tombstones in the sstable iterators (CASSANDRA-13340)
  * Fix CONTAINS filtering for null collections (CASSANDRA-13246)
  * Applying: Use a unique metric reservoir per test run when using 
Cassandra-wide metrics residing in MBeans (CASSANDRA-13216)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5262bb17/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
--
diff --git 
a/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java 
b/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
index 9bcca48..fa337c0 100644
--- a/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
+++ b/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
@@ -123,20 +123,27 @@ public class SSTableIterator extends 
AbstractSSTableIterator
 {
 assert deserializer != null;
 
-// We use a same reasoning as in handlePreSliceData regarding the 
strictness of the inequality below.
-// We want to exclude deserialized unfiltered equal to end, 
because 1) we won't miss any rows since those
-// woudn't be equal to a slice bound and 2) a end bound can be 
equal to a start bound
-// (EXCL_END(x) == INCL_START(x) for instance) and in that case we 
don't want to return start bound because
-// it's fundamentally excluded. And if the bound is a  end (for a 
range tombstone), it means it's exactly
-// our slice end, but in that  case we will properly close the 
range tombstone anyway as part of our "close
-// an open marker" code in hasNextInterna
-if (!deserializer.hasNext() || deserializer.compareNextTo(end) >= 
0)
-return null;
-
-Unfiltered next = deserializer.readNext();
-if (next.kind() == Unfiltered.Kind.RANGE_TOMBSTONE_MARKER)
-updateOpenMarker((RangeTombstoneMarker)next);
-return next;
+while (true)
+{
+// We use a same reasoning as in handlePreSliceData regarding 
the strictness of the inequality below.
+// We want to exclude deserialized unfiltered equal to end, 
because 1) we won't miss any rows since those
+// woudn't be equal to a slice bound and 2) a end bound can be 
equal to a start bound
+// (EXCL_END(x) == INCL_START(x) for instance) and in that 
case we don't want to return start bound because
+// it's fundamentally excluded. And if the bound is a  end 
(for a range tombstone), it means it's exactly
+// our slice end, but in that  case we will properly close the 
range tombstone anyway as part of our "close
+// an open marker" code in hasNextInterna
+if (!deserializer.hasNext() || deserializer.compareNextTo(end) 
>= 0)
+return null;
+
+Unfiltered next = deserializer.readNext();
+// We may get empty row for the same reason expressed on 
UnfilteredSerializer.deserializeOne.
+if (next.isEmpty())
+continue;
+
+if (next.kind() == Unfiltered.Kind.RANGE_TOMBSTONE_MARKER)
+

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-03-27 Thread slebresne

Merge branch 'cassandra-3.0' into cassandra-3.11

* cassandra-3.0:
  Dropping column results in "corrupt" SSTable


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c0ac928d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c0ac928d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c0ac928d

Branch: refs/heads/cassandra-3.11
Commit: c0ac928d985a59d00310fed124c5a2b3655db8e0
Parents: ee7023e 5262bb1
Author: Sylvain Lebresne 
Authored: Mon Mar 27 11:59:06 2017 +0200
Committer: Sylvain Lebresne 
Committed: Mon Mar 27 11:59:06 2017 +0200

--
 CHANGES.txt |  1 +
 .../db/columniterator/SSTableIterator.java  | 78 
 .../columniterator/SSTableReversedIterator.java |  3 +-
 .../cassandra/db/rows/RangeTombstoneMarker.java |  6 ++
 .../apache/cassandra/db/rows/Unfiltered.java|  2 +
 .../cassandra/db/rows/UnfilteredSerializer.java | 43 +--
 .../cql3/validation/operations/AlterTest.java   | 51 +
 7 files changed, 144 insertions(+), 40 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0ac928d/CHANGES.txt
--
diff --cc CHANGES.txt
index 071dd1a,0b1bb01..98ed992
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,20 -1,5 +1,21 @@@
 -3.0.13
 +3.11.0
 + * unittest CipherFactoryTest failed on MacOS (CASSANDRA-13370)
 + * Forbid SELECT restrictions and CREATE INDEX over non-frozen UDT columns 
(CASSANDRA-13247)
 + * Default logging we ship will incorrectly print "?:?" for "%F:%L" pattern 
(CASSANDRA-13317)
 + * Possible AssertionError in UnfilteredRowIteratorWithLowerBound 
(CASSANDRA-13366)
 + * Support unaligned memory access for AArch64 (CASSANDRA-13326)
 + * Improve SASI range iterator efficiency on intersection with an empty range 
(CASSANDRA-12915).
 + * Fix equality comparisons of columns using the duration type 
(CASSANDRA-13174)
 + * Obfuscate password in stress-graphs (CASSANDRA-12233)
 + * Move to FastThreadLocalThread and FastThreadLocal (CASSANDRA-13034)
 + * nodetool stopdaemon errors out (CASSANDRA-13030)
 + * Tables in system_distributed should not use gcgs of 0 (CASSANDRA-12954)
 + * Fix primary index calculation for SASI (CASSANDRA-12910)
 + * More fixes to the TokenAllocator (CASSANDRA-12990)
 + * NoReplicationTokenAllocator should work with zero replication factor 
(CASSANDRA-12983)
 + * Address message coalescing regression (CASSANDRA-12676)
 +Merged from 3.0:
+  * Dropping column results in "corrupt" SSTable (CASSANDRA-13337)
   * Bugs handling range tombstones in the sstable iterators (CASSANDRA-13340)
   * Fix CONTAINS filtering for null collections (CASSANDRA-13246)
   * Applying: Use a unique metric reservoir per test run when using 
Cassandra-wide metrics residing in MBeans (CASSANDRA-13216)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0ac928d/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0ac928d/src/java/org/apache/cassandra/db/columniterator/SSTableReversedIterator.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0ac928d/src/java/org/apache/cassandra/db/rows/RangeTombstoneMarker.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0ac928d/src/java/org/apache/cassandra/db/rows/Unfiltered.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0ac928d/src/java/org/apache/cassandra/db/rows/UnfilteredSerializer.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0ac928d/test/unit/org/apache/cassandra/cql3/validation/operations/AlterTest.java
--

[1/6] cassandra git commit: Dropping column results in "corrupt" SSTable

2017-03-27 Thread slebresne

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 631162271 -> 5262bb17b
  refs/heads/cassandra-3.11 ee7023e32 -> c0ac928d9
  refs/heads/trunk 23cd27fcf -> 3dabeeaa2


Dropping column results in "corrupt" SSTable

patch by Sylvain Lebresne; reviewed by Alex Petrov for CASSANDRA-13337


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5262bb17
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5262bb17
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5262bb17

Branch: refs/heads/cassandra-3.0
Commit: 5262bb17b46fc8c02f9f836ddf9317d0de2698cd
Parents: 6311622
Author: Sylvain Lebresne 
Authored: Mon Mar 20 15:49:27 2017 +0100
Committer: Sylvain Lebresne 
Committed: Mon Mar 27 11:58:32 2017 +0200

--
 CHANGES.txt |  1 +
 .../db/columniterator/SSTableIterator.java  | 78 
 .../columniterator/SSTableReversedIterator.java |  3 +-
 .../cassandra/db/rows/RangeTombstoneMarker.java |  6 ++
 .../apache/cassandra/db/rows/Unfiltered.java|  2 +
 .../cassandra/db/rows/UnfilteredSerializer.java | 43 +--
 .../cql3/validation/operations/AlterTest.java   | 51 +
 7 files changed, 144 insertions(+), 40 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5262bb17/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 2c5573a..0b1bb01 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.13
+ * Dropping column results in "corrupt" SSTable (CASSANDRA-13337)
  * Bugs handling range tombstones in the sstable iterators (CASSANDRA-13340)
  * Fix CONTAINS filtering for null collections (CASSANDRA-13246)
  * Applying: Use a unique metric reservoir per test run when using 
Cassandra-wide metrics residing in MBeans (CASSANDRA-13216)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5262bb17/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
--
diff --git 
a/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java 
b/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
index 9bcca48..fa337c0 100644
--- a/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
+++ b/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
@@ -123,20 +123,27 @@ public class SSTableIterator extends 
AbstractSSTableIterator
 {
 assert deserializer != null;
 
-// We use a same reasoning as in handlePreSliceData regarding the 
strictness of the inequality below.
-// We want to exclude deserialized unfiltered equal to end, 
because 1) we won't miss any rows since those
-// woudn't be equal to a slice bound and 2) a end bound can be 
equal to a start bound
-// (EXCL_END(x) == INCL_START(x) for instance) and in that case we 
don't want to return start bound because
-// it's fundamentally excluded. And if the bound is a  end (for a 
range tombstone), it means it's exactly
-// our slice end, but in that  case we will properly close the 
range tombstone anyway as part of our "close
-// an open marker" code in hasNextInterna
-if (!deserializer.hasNext() || deserializer.compareNextTo(end) >= 
0)
-return null;
-
-Unfiltered next = deserializer.readNext();
-if (next.kind() == Unfiltered.Kind.RANGE_TOMBSTONE_MARKER)
-updateOpenMarker((RangeTombstoneMarker)next);
-return next;
+while (true)
+{
+// We use a same reasoning as in handlePreSliceData regarding 
the strictness of the inequality below.
+// We want to exclude deserialized unfiltered equal to end, 
because 1) we won't miss any rows since those
+// woudn't be equal to a slice bound and 2) a end bound can be 
equal to a start bound
+// (EXCL_END(x) == INCL_START(x) for instance) and in that 
case we don't want to return start bound because
+// it's fundamentally excluded. And if the bound is a  end 
(for a range tombstone), it means it's exactly
+// our slice end, but in that  case we will properly close the 
range tombstone anyway as part of our "close
+// an open marker" code in hasNextInterna
+if (!deserializer.hasNext() || deserializer.compareNextTo(end) 
>= 0)
+return null;
+
+Unfiltered next = deserializer.readNext();
+// We may get empty row for the same reason expressed on 
UnfilteredSerializer.deserializeOne.
+

[3/3] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2017-03-23 Thread slebresne

Merge branch 'cassandra-3.11' into trunk

* cassandra-3.11:
  Bugs handling range tombstones in the sstable iterators


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9330409a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9330409a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9330409a

Branch: refs/heads/trunk
Commit: 9330409accf0506526d25e17e70f89e5cb6a341e
Parents: ea662ce a85eeef
Author: Sylvain Lebresne 
Authored: Thu Mar 23 17:18:36 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Mar 23 17:18:36 2017 +0100

--

--

[2/3] cassandra git commit: Bugs handling range tombstones in the sstable iterators

2017-03-23 Thread slebresne

Bugs handling range tombstones in the sstable iterators

patch by Sylvain Lebresne; reviewed by Branimir Lambov for CASSANDRA-13340


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a85eeefe
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a85eeefe
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a85eeefe

Branch: refs/heads/trunk
Commit: a85eeefe88eb036a9cd9fa85a1c8c31c2bfad78a
Parents: 3e95c5b
Author: Sylvain Lebresne 
Authored: Thu Mar 16 17:05:15 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Mar 23 17:17:16 2017 +0100

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ClusteringPrefix.java   |   2 +-
 .../cassandra/db/UnfilteredDeserializer.java|   1 -
 .../db/columniterator/SSTableIterator.java  |  11 +-
 .../columniterator/SSTableReversedIterator.java | 126 +++
 .../cql3/validation/operations/DeleteTest.java  |  70 +++
 6 files changed, 181 insertions(+), 30 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a85eeefe/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index c58fad8..728e3e7 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -13,6 +13,7 @@
  * NoReplicationTokenAllocator should work with zero replication factor 
(CASSANDRA-12983)
  * Address message coalescing regression (CASSANDRA-12676)
 Merged from 3.0:
+ * Bugs handling range tombstones in the sstable iterators (CASSANDRA-13340)
  * Fix CONTAINS filtering for null collections (CASSANDRA-13246)
  * Applying: Use a unique metric reservoir per test run when using 
Cassandra-wide metrics residing in MBeans (CASSANDRA-13216)
  * Propagate row deletions in 2i tables on upgrade (CASSANDRA-13320)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a85eeefe/src/java/org/apache/cassandra/db/ClusteringPrefix.java
--
diff --git a/src/java/org/apache/cassandra/db/ClusteringPrefix.java 
b/src/java/org/apache/cassandra/db/ClusteringPrefix.java
index 340e237..1ecc92d 100644
--- a/src/java/org/apache/cassandra/db/ClusteringPrefix.java
+++ b/src/java/org/apache/cassandra/db/ClusteringPrefix.java
@@ -482,7 +482,7 @@ public interface ClusteringPrefix extends 
IMeasurableMemory, Clusterable
 }
 
 if (bound.size() == nextSize)
-return nextKind.compareTo(bound.kind());
+return Kind.compare(nextKind, bound.kind());
 
 // We know that we'll have exited already if nextSize < bound.size
 return -bound.kind().comparedToClustering;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a85eeefe/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
--
diff --git a/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java 
b/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
index 79b8636..b977907 100644
--- a/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
+++ b/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
@@ -690,6 +690,5 @@ public abstract class UnfilteredDeserializer
 }
 }
 }
-
 }
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a85eeefe/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
--
diff --git 
a/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java 
b/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
index b3c2e94..e21bd72 100644
--- a/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
+++ b/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
@@ -138,7 +138,14 @@ public class SSTableIterator extends 
AbstractSSTableIterator
 {
 assert deserializer != null;
 
-if (!deserializer.hasNext() || deserializer.compareNextTo(end) > 0)
+// We use a same reasoning as in handlePreSliceData regarding the 
strictness of the inequality below.
+// We want to exclude deserialized unfiltered equal to end, 
because 1) we won't miss any rows since those
+// woudn't be equal to a slice bound and 2) a end bound can be 
equal to a start bound
+// (EXCL_END(x) == INCL_START(x) for instance) and in that case we 
don't want to return start bound because
+// it's fundamentally excluded. And if the bound is a  end (for a 
range tombstone), it means it's exactly
+// our slice end, but in that  case we will properly close the 
range tombstone anyway as part of our "close
+

[1/3] cassandra git commit: Bugs handling range tombstones in the sstable iterators

2017-03-23 Thread slebresne

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.11 3e95c5b0c -> a85eeefe8
  refs/heads/trunk ea662ce21 -> 9330409ac


Bugs handling range tombstones in the sstable iterators

patch by Sylvain Lebresne; reviewed by Branimir Lambov for CASSANDRA-13340


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a85eeefe
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a85eeefe
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a85eeefe

Branch: refs/heads/cassandra-3.11
Commit: a85eeefe88eb036a9cd9fa85a1c8c31c2bfad78a
Parents: 3e95c5b
Author: Sylvain Lebresne 
Authored: Thu Mar 16 17:05:15 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Mar 23 17:17:16 2017 +0100

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ClusteringPrefix.java   |   2 +-
 .../cassandra/db/UnfilteredDeserializer.java|   1 -
 .../db/columniterator/SSTableIterator.java  |  11 +-
 .../columniterator/SSTableReversedIterator.java | 126 +++
 .../cql3/validation/operations/DeleteTest.java  |  70 +++
 6 files changed, 181 insertions(+), 30 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a85eeefe/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index c58fad8..728e3e7 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -13,6 +13,7 @@
  * NoReplicationTokenAllocator should work with zero replication factor 
(CASSANDRA-12983)
  * Address message coalescing regression (CASSANDRA-12676)
 Merged from 3.0:
+ * Bugs handling range tombstones in the sstable iterators (CASSANDRA-13340)
  * Fix CONTAINS filtering for null collections (CASSANDRA-13246)
  * Applying: Use a unique metric reservoir per test run when using 
Cassandra-wide metrics residing in MBeans (CASSANDRA-13216)
  * Propagate row deletions in 2i tables on upgrade (CASSANDRA-13320)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a85eeefe/src/java/org/apache/cassandra/db/ClusteringPrefix.java
--
diff --git a/src/java/org/apache/cassandra/db/ClusteringPrefix.java 
b/src/java/org/apache/cassandra/db/ClusteringPrefix.java
index 340e237..1ecc92d 100644
--- a/src/java/org/apache/cassandra/db/ClusteringPrefix.java
+++ b/src/java/org/apache/cassandra/db/ClusteringPrefix.java
@@ -482,7 +482,7 @@ public interface ClusteringPrefix extends 
IMeasurableMemory, Clusterable
 }
 
 if (bound.size() == nextSize)
-return nextKind.compareTo(bound.kind());
+return Kind.compare(nextKind, bound.kind());
 
 // We know that we'll have exited already if nextSize < bound.size
 return -bound.kind().comparedToClustering;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a85eeefe/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
--
diff --git a/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java 
b/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
index 79b8636..b977907 100644
--- a/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
+++ b/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
@@ -690,6 +690,5 @@ public abstract class UnfilteredDeserializer
 }
 }
 }
-
 }
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a85eeefe/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
--
diff --git 
a/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java 
b/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
index b3c2e94..e21bd72 100644
--- a/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
+++ b/src/java/org/apache/cassandra/db/columniterator/SSTableIterator.java
@@ -138,7 +138,14 @@ public class SSTableIterator extends 
AbstractSSTableIterator
 {
 assert deserializer != null;
 
-if (!deserializer.hasNext() || deserializer.compareNextTo(end) > 0)
+// We use a same reasoning as in handlePreSliceData regarding the 
strictness of the inequality below.
+// We want to exclude deserialized unfiltered equal to end, 
because 1) we won't miss any rows since those
+// woudn't be equal to a slice bound and 2) a end bound can be 
equal to a start bound
+// (EXCL_END(x) == INCL_START(x) for instance) and in that case we 
don't want to return start bound because
+// it's fundamentally excluded. And if the bound is a  end (for a 
range tombstone), it

[1/3] cassandra git commit: Possible AssertionError in UnfilteredRowIteratorWithLowerBound

2017-03-23 Thread slebresne

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.11 ec9ce3dfb -> f55cb88ab
  refs/heads/trunk 8b74ae4b6 -> a87b15d1d


Possible AssertionError in UnfilteredRowIteratorWithLowerBound

patch by Sylvain Lebresne; reviewed by Stefania for CASSANDRA-13366


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f55cb88a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f55cb88a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f55cb88a

Branch: refs/heads/cassandra-3.11
Commit: f55cb88ab595ccb941ebb4a088ab90f860f463d5
Parents: ec9ce3d
Author: Sylvain Lebresne 
Authored: Wed Mar 22 15:41:49 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Mar 23 10:26:37 2017 +0100

--
 CHANGES.txt |  1 +
 .../db/SinglePartitionReadCommand.java  |  4 +--
 .../db/compaction/CompactionController.java |  2 +-
 .../UnfilteredRowIteratorWithLowerBound.java| 31 ++---
 .../io/sstable/format/SSTableReader.java| 35 
 5 files changed, 44 insertions(+), 29 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f55cb88a/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 8386c20..f4e48ff 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.11.0
+ * Possible AssertionError in UnfilteredRowIteratorWithLowerBound 
(CASSANDRA-13366)
  * Support unaligned memory access for AArch64 (CASSANDRA-13326)
  * Improve SASI range iterator efficiency on intersection with an empty range 
(CASSANDRA-12915).
  * Fix equality comparisons of columns using the duration type 
(CASSANDRA-13174)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f55cb88a/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
--
diff --git a/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java 
b/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
index f6d10f5..724f59e 100644
--- a/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
+++ b/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
@@ -584,7 +584,7 @@ public class SinglePartitionReadCommand extends ReadCommand
 if (!shouldInclude(sstable))
 {
 nonIntersectingSSTables++;
-if (sstable.hasTombstones())
+if (sstable.mayHaveTombstones())
 { // if sstable has tombstones we need to check after one 
pass if it can be safely skipped
 if (skippedSSTablesWithTombstones == null)
 skippedSSTablesWithTombstones = new ArrayList<>();
@@ -773,7 +773,7 @@ public class SinglePartitionReadCommand extends ReadCommand
 // however: if it is set, it impacts everything and must be 
included. Getting that top-level partition deletion costs us
 // some seek in general however (unless the partition is 
indexed and is in the key cache), so we first check if the sstable
 // has any tombstone at all as a shortcut.
-if (!sstable.hasTombstones())
+if (!sstable.mayHaveTombstones())
 continue; // no tombstone at all, we can skip that sstable
 
 // We need to get the partition deletion and include it if 
it's live. In any case though, we're done with that sstable.

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f55cb88a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
--
diff --git 
a/src/java/org/apache/cassandra/db/compaction/CompactionController.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
index 64c35d9..bf3647a 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
@@ -297,7 +297,7 @@ public class CompactionController implements AutoCloseable
 {
 if (reader.isMarkedSuspect() ||
 reader.getMaxTimestamp() <= minTimestamp ||
-tombstoneOnly && !reader.hasTombstones())
+tombstoneOnly && !reader.mayHaveTombstones())
 return null;
 RowIndexEntry position = reader.getPosition(key, 
SSTableReader.Operator.EQ);
 if (position == null)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f55cb88a/src/java/org/apache/cassandra/db/rows/UnfilteredRowIteratorWithLowerBound.java
--
diff --git

[3/3] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2017-03-23 Thread slebresne

Merge branch 'cassandra-3.11' into trunk

* cassandra-3.11:
  Possible AssertionError in UnfilteredRowIteratorWithLowerBound


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a87b15d1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a87b15d1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a87b15d1

Branch: refs/heads/trunk
Commit: a87b15d1d6c42f4247c84b460ed39899d8813a6f
Parents: 8b74ae4 f55cb88
Author: Sylvain Lebresne 
Authored: Thu Mar 23 10:29:59 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Mar 23 10:29:59 2017 +0100

--
 CHANGES.txt |  1 +
 .../db/SinglePartitionReadCommand.java  |  4 +--
 .../db/compaction/CompactionController.java |  2 +-
 .../UnfilteredRowIteratorWithLowerBound.java| 30 +++---
 .../io/sstable/format/SSTableReader.java| 33 +++-
 5 files changed, 42 insertions(+), 28 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a87b15d1/CHANGES.txt
--
diff --cc CHANGES.txt
index b68e51c,f4e48ff..c1d5e94
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,50 -1,5 +1,51 @@@
 +4.0
 + * Cleanup ParentRepairSession after repairs (CASSANDRA-13359)
 + * Incremental repair not streaming correct sstables (CASSANDRA-13328)
 + * Upgrade the jna version to 4.3.0 (CASSANDRA-13300)
 + * Add the currentTimestamp, currentDate, currentTime and currentTimeUUID 
functions (CASSANDRA-13132)
 + * Remove config option index_interval (CASSANDRA-10671)
 + * Reduce lock contention for collection types and serializers 
(CASSANDRA-13271)
 + * Make it possible to override MessagingService.Verb ids (CASSANDRA-13283)
 + * Avoid synchronized on prepareForRepair in ActiveRepairService 
(CASSANDRA-9292)
 + * Adds the ability to use uncompressed chunks in compressed files 
(CASSANDRA-10520)
 + * Don't flush sstables when streaming for incremental repair 
(CASSANDRA-13226)
 + * Remove unused method (CASSANDRA-13227)
 + * Fix minor bugs related to #9143 (CASSANDRA-13217)
 + * Output warning if user increases RF (CASSANDRA-13079)
 + * Remove pre-3.0 streaming compatibility code for 4.0 (CASSANDRA-13081)
 + * Add support for + and - operations on dates (CASSANDRA-11936)
 + * Fix consistency of incrementally repaired data (CASSANDRA-9143)
 + * Increase commitlog version (CASSANDRA-13161)
 + * Make TableMetadata immutable, optimize Schema (CASSANDRA-9425)
 + * Refactor ColumnCondition (CASSANDRA-12981)
 + * Parallelize streaming of different keyspaces (CASSANDRA-4663)
 + * Improved compactions metrics (CASSANDRA-13015)
 + * Speed-up start-up sequence by avoiding un-needed flushes (CASSANDRA-13031)
 + * Use Caffeine (W-TinyLFU) for on-heap caches (CASSANDRA-10855)
 + * Thrift removal (CASSANDRA-5)
 + * Remove pre-3.0 compatibility code for 4.0 (CASSANDRA-12716)
 + * Add column definition kind to dropped columns in schema (CASSANDRA-12705)
 + * Add (automate) Nodetool Documentation (CASSANDRA-12672)
 + * Update bundled cqlsh python driver to 3.7.0 (CASSANDRA-12736)
 + * Reject invalid replication settings when creating or altering a keyspace 
(CASSANDRA-12681)
 + * Clean up the SSTableReader#getScanner API wrt removal of RateLimiter 
(CASSANDRA-12422)
 + * Use new token allocation for non bootstrap case as well (CASSANDRA-13080)
 + * Avoid byte-array copy when key cache is disabled (CASSANDRA-13084)
 + * Require forceful decommission if number of nodes is less than replication 
factor (CASSANDRA-12510)
 + * Allow IN restrictions on column families with collections (CASSANDRA-12654)
 + * Log message size in trace message in OutboundTcpConnection 
(CASSANDRA-13028)
 + * Add timeUnit Days for cassandra-stress (CASSANDRA-13029)
 + * Add mutation size and batch metrics (CASSANDRA-12649)
 + * Add method to get size of endpoints to TokenMetadata (CASSANDRA-12999)
 + * Expose time spent waiting in thread pool queue (CASSANDRA-8398)
 + * Conditionally update index built status to avoid unnecessary flushes 
(CASSANDRA-12969)
 + * cqlsh auto completion: refactor definition of compaction strategy options 
(CASSANDRA-12946)
 + * Add support for arithmetic operators (CASSANDRA-11935)
 + * Add histogram for delay to deliver hints (CASSANDRA-13234)
 +
 +
  3.11.0
+  * Possible AssertionError in UnfilteredRowIteratorWithLowerBound 
(CASSANDRA-13366)
   * Support unaligned memory access for AArch64 (CASSANDRA-13326)
   * Improve SASI range iterator efficiency on intersection with an empty range 
(CASSANDRA-12915).
   * Fix equality comparisons of columns using the duration type 
(CASSANDRA-13174)

[2/3] cassandra git commit: Possible AssertionError in UnfilteredRowIteratorWithLowerBound

2017-03-23 Thread slebresne

Possible AssertionError in UnfilteredRowIteratorWithLowerBound

patch by Sylvain Lebresne; reviewed by Stefania for CASSANDRA-13366


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f55cb88a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f55cb88a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f55cb88a

Branch: refs/heads/trunk
Commit: f55cb88ab595ccb941ebb4a088ab90f860f463d5
Parents: ec9ce3d
Author: Sylvain Lebresne 
Authored: Wed Mar 22 15:41:49 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Mar 23 10:26:37 2017 +0100

--
 CHANGES.txt |  1 +
 .../db/SinglePartitionReadCommand.java  |  4 +--
 .../db/compaction/CompactionController.java |  2 +-
 .../UnfilteredRowIteratorWithLowerBound.java| 31 ++---
 .../io/sstable/format/SSTableReader.java| 35 
 5 files changed, 44 insertions(+), 29 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f55cb88a/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 8386c20..f4e48ff 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.11.0
+ * Possible AssertionError in UnfilteredRowIteratorWithLowerBound 
(CASSANDRA-13366)
  * Support unaligned memory access for AArch64 (CASSANDRA-13326)
  * Improve SASI range iterator efficiency on intersection with an empty range 
(CASSANDRA-12915).
  * Fix equality comparisons of columns using the duration type 
(CASSANDRA-13174)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f55cb88a/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
--
diff --git a/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java 
b/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
index f6d10f5..724f59e 100644
--- a/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
+++ b/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
@@ -584,7 +584,7 @@ public class SinglePartitionReadCommand extends ReadCommand
 if (!shouldInclude(sstable))
 {
 nonIntersectingSSTables++;
-if (sstable.hasTombstones())
+if (sstable.mayHaveTombstones())
 { // if sstable has tombstones we need to check after one 
pass if it can be safely skipped
 if (skippedSSTablesWithTombstones == null)
 skippedSSTablesWithTombstones = new ArrayList<>();
@@ -773,7 +773,7 @@ public class SinglePartitionReadCommand extends ReadCommand
 // however: if it is set, it impacts everything and must be 
included. Getting that top-level partition deletion costs us
 // some seek in general however (unless the partition is 
indexed and is in the key cache), so we first check if the sstable
 // has any tombstone at all as a shortcut.
-if (!sstable.hasTombstones())
+if (!sstable.mayHaveTombstones())
 continue; // no tombstone at all, we can skip that sstable
 
 // We need to get the partition deletion and include it if 
it's live. In any case though, we're done with that sstable.

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f55cb88a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
--
diff --git 
a/src/java/org/apache/cassandra/db/compaction/CompactionController.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
index 64c35d9..bf3647a 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
@@ -297,7 +297,7 @@ public class CompactionController implements AutoCloseable
 {
 if (reader.isMarkedSuspect() ||
 reader.getMaxTimestamp() <= minTimestamp ||
-tombstoneOnly && !reader.hasTombstones())
+tombstoneOnly && !reader.mayHaveTombstones())
 return null;
 RowIndexEntry position = reader.getPosition(key, 
SSTableReader.Operator.EQ);
 if (position == null)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f55cb88a/src/java/org/apache/cassandra/db/rows/UnfilteredRowIteratorWithLowerBound.java
--
diff --git 
a/src/java/org/apache/cassandra/db/rows/UnfilteredRowIteratorWithLowerBound.java
 
b/src/java/org/apache/cassandra/db/rows/UnfilteredRowIteratorWithLowerBound.java

[2/6] cassandra git commit: Slice.isEmpty() returns false for some empty slices

2017-03-09 Thread slebresne

Slice.isEmpty() returns false for some empty slices

patch by Sylvain Lebresne; reviewed by Branimir Lambov for CASSANDRA-13305


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/31dec3d5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/31dec3d5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/31dec3d5

Branch: refs/heads/cassandra-3.11
Commit: 31dec3d548ae2c76d7c8bf4bffa9d506f670f756
Parents: 60d3292
Author: Sylvain Lebresne 
Authored: Thu Mar 9 11:58:40 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Mar 9 11:58:40 2017 +0100

--
 CHANGES.txt   |  1 +
 src/java/org/apache/cassandra/db/Slice.java   |  2 +-
 .../cql3/validation/operations/DeleteTest.java| 18 ++
 3 files changed, 20 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/31dec3d5/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 0979852..1876922 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.13
+ * Slice.isEmpty() returns false for some empty slices (CASSANDRA-13305)
  * Add formatted row output to assertEmpty in CQL Tester (CASSANDRA-13238)
 Merged from 2.2:
  * Fix GRANT/REVOKE when keyspace isn't specified (CASSANDRA-13053)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/31dec3d5/src/java/org/apache/cassandra/db/Slice.java
--
diff --git a/src/java/org/apache/cassandra/db/Slice.java 
b/src/java/org/apache/cassandra/db/Slice.java
index 7fde45e..3c645dc 100644
--- a/src/java/org/apache/cassandra/db/Slice.java
+++ b/src/java/org/apache/cassandra/db/Slice.java
@@ -160,7 +160,7 @@ public class Slice
 public static boolean isEmpty(ClusteringComparator comparator, Slice.Bound 
start, Slice.Bound end)
 {
 assert start.isStart() && end.isEnd();
-return comparator.compare(end, start) < 0;
+return comparator.compare(end, start) <= 0;
 }
 
 /**

http://git-wip-us.apache.org/repos/asf/cassandra/blob/31dec3d5/test/unit/org/apache/cassandra/cql3/validation/operations/DeleteTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/cql3/validation/operations/DeleteTest.java 
b/test/unit/org/apache/cassandra/cql3/validation/operations/DeleteTest.java
index 09098ac..9d7d4a3 100644
--- a/test/unit/org/apache/cassandra/cql3/validation/operations/DeleteTest.java
+++ b/test/unit/org/apache/cassandra/cql3/validation/operations/DeleteTest.java
@@ -1292,6 +1292,24 @@ public class DeleteTest extends CQLTester
 }
 
 /**
+ * Test for CASSANDRA-13305
+ */
+@Test
+public void testWithEmptyRange() throws Throwable
+{
+createTable("CREATE TABLE %s (k text, a int, b int, PRIMARY KEY (k, a, 
b))");
+
+// Both of the following should be doing nothing, but before #13305 
this inserted broken ranges. We do it twice
+// and the follow-up delete mainly as a way to show the bug as the 
combination of this will trigger an assertion
+// in RangeTombstoneList pre-#13305 showing that something wrong 
happened.
+execute("DELETE FROM %s WHERE k = ? AND a >= ? AND a < ?", "a", 1, 1);
+execute("DELETE FROM %s WHERE k = ? AND a >= ? AND a < ?", "a", 1, 1);
+
+execute("DELETE FROM %s WHERE k = ? AND a >= ? AND a < ?", "a", 0, 2);
+}
+
+
+/**
  * Checks if the memtable is empty or not
  * @return {@code true} if the memtable is empty, {@code false} otherwise.
  */

[3/6] cassandra git commit: Slice.isEmpty() returns false for some empty slices

2017-03-09 Thread slebresne

Slice.isEmpty() returns false for some empty slices

patch by Sylvain Lebresne; reviewed by Branimir Lambov for CASSANDRA-13305


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/31dec3d5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/31dec3d5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/31dec3d5

Branch: refs/heads/trunk
Commit: 31dec3d548ae2c76d7c8bf4bffa9d506f670f756
Parents: 60d3292
Author: Sylvain Lebresne 
Authored: Thu Mar 9 11:58:40 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Mar 9 11:58:40 2017 +0100

--
 CHANGES.txt   |  1 +
 src/java/org/apache/cassandra/db/Slice.java   |  2 +-
 .../cql3/validation/operations/DeleteTest.java| 18 ++
 3 files changed, 20 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/31dec3d5/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 0979852..1876922 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.13
+ * Slice.isEmpty() returns false for some empty slices (CASSANDRA-13305)
  * Add formatted row output to assertEmpty in CQL Tester (CASSANDRA-13238)
 Merged from 2.2:
  * Fix GRANT/REVOKE when keyspace isn't specified (CASSANDRA-13053)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/31dec3d5/src/java/org/apache/cassandra/db/Slice.java
--
diff --git a/src/java/org/apache/cassandra/db/Slice.java 
b/src/java/org/apache/cassandra/db/Slice.java
index 7fde45e..3c645dc 100644
--- a/src/java/org/apache/cassandra/db/Slice.java
+++ b/src/java/org/apache/cassandra/db/Slice.java
@@ -160,7 +160,7 @@ public class Slice
 public static boolean isEmpty(ClusteringComparator comparator, Slice.Bound 
start, Slice.Bound end)
 {
 assert start.isStart() && end.isEnd();
-return comparator.compare(end, start) < 0;
+return comparator.compare(end, start) <= 0;
 }
 
 /**

http://git-wip-us.apache.org/repos/asf/cassandra/blob/31dec3d5/test/unit/org/apache/cassandra/cql3/validation/operations/DeleteTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/cql3/validation/operations/DeleteTest.java 
b/test/unit/org/apache/cassandra/cql3/validation/operations/DeleteTest.java
index 09098ac..9d7d4a3 100644
--- a/test/unit/org/apache/cassandra/cql3/validation/operations/DeleteTest.java
+++ b/test/unit/org/apache/cassandra/cql3/validation/operations/DeleteTest.java
@@ -1292,6 +1292,24 @@ public class DeleteTest extends CQLTester
 }
 
 /**
+ * Test for CASSANDRA-13305
+ */
+@Test
+public void testWithEmptyRange() throws Throwable
+{
+createTable("CREATE TABLE %s (k text, a int, b int, PRIMARY KEY (k, a, 
b))");
+
+// Both of the following should be doing nothing, but before #13305 
this inserted broken ranges. We do it twice
+// and the follow-up delete mainly as a way to show the bug as the 
combination of this will trigger an assertion
+// in RangeTombstoneList pre-#13305 showing that something wrong 
happened.
+execute("DELETE FROM %s WHERE k = ? AND a >= ? AND a < ?", "a", 1, 1);
+execute("DELETE FROM %s WHERE k = ? AND a >= ? AND a < ?", "a", 1, 1);
+
+execute("DELETE FROM %s WHERE k = ? AND a >= ? AND a < ?", "a", 0, 2);
+}
+
+
+/**
  * Checks if the memtable is empty or not
  * @return {@code true} if the memtable is empty, {@code false} otherwise.
  */

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2017-03-09 Thread slebresne

Merge branch 'cassandra-3.11' into trunk

* cassandra-3.11:
  Slice.isEmpty() returns false for some empty slices


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9e8e8914
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9e8e8914
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9e8e8914

Branch: refs/heads/trunk
Commit: 9e8e8914d8a4e55a2d647c4c462e1cb7b622a930
Parents: b5a5fbe dc65a57
Author: Sylvain Lebresne 
Authored: Thu Mar 9 11:59:42 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Mar 9 11:59:42 2017 +0100

--
 CHANGES.txt   |  1 +
 src/java/org/apache/cassandra/db/Slice.java   |  2 +-
 .../cql3/validation/operations/DeleteTest.java| 18 ++
 3 files changed, 20 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9e8e8914/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9e8e8914/src/java/org/apache/cassandra/db/Slice.java
--

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-03-09 Thread slebresne

Merge branch 'cassandra-3.0' into cassandra-3.11

* cassandra-3.0:
  Slice.isEmpty() returns false for some empty slices


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/dc65a576
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/dc65a576
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/dc65a576

Branch: refs/heads/trunk
Commit: dc65a576553f8766076920c8b639d80763d6e1f5
Parents: 7707a0e 31dec3d
Author: Sylvain Lebresne 
Authored: Thu Mar 9 11:59:32 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Mar 9 11:59:32 2017 +0100

--
 CHANGES.txt   |  1 +
 src/java/org/apache/cassandra/db/Slice.java   |  2 +-
 .../cql3/validation/operations/DeleteTest.java| 18 ++
 3 files changed, 20 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/dc65a576/CHANGES.txt
--
diff --cc CHANGES.txt
index f73dc12,1876922..2772fc2
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,15 -1,11 +1,16 @@@
 -3.0.13
 +3.11.0
 + * Fix equality comparisons of columns using the duration type 
(CASSANDRA-13174)
 + * Obfuscate password in stress-graphs (CASSANDRA-12233)
 + * Move to FastThreadLocalThread and FastThreadLocal (CASSANDRA-13034)
 + * nodetool stopdaemon errors out (CASSANDRA-13030)
 + * Tables in system_distributed should not use gcgs of 0 (CASSANDRA-12954)
 + * Fix primary index calculation for SASI (CASSANDRA-12910)
 + * More fixes to the TokenAllocator (CASSANDRA-12990)
 + * NoReplicationTokenAllocator should work with zero replication factor 
(CASSANDRA-12983)
 + * Address message coalescing regression (CASSANDRA-12676)
 +Merged from 3.0:
+  * Slice.isEmpty() returns false for some empty slices (CASSANDRA-13305)
   * Add formatted row output to assertEmpty in CQL Tester (CASSANDRA-13238)
 -Merged from 2.2:
 - * Fix GRANT/REVOKE when keyspace isn't specified (CASSANDRA-13053)
 -
 -
 -3.0.12
   * Prevent data loss on upgrade 2.1 - 3.0 by adding component separator to 
LogRecord absolute path (CASSANDRA-13294)
   * Improve testing on macOS by eliminating sigar logging (CASSANDRA-13233)
   * Cqlsh copy-from should error out when csv contains invalid data for 
collections (CASSANDRA-13071)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dc65a576/src/java/org/apache/cassandra/db/Slice.java
--
diff --cc src/java/org/apache/cassandra/db/Slice.java
index c3da222,3c645dc..4b36677
--- a/src/java/org/apache/cassandra/db/Slice.java
+++ b/src/java/org/apache/cassandra/db/Slice.java
@@@ -157,10 -157,10 +157,10 @@@ public class Slic
   * @return whether the slice formed by {@code start} and {@code end} is
   * empty or not.
   */
 -public static boolean isEmpty(ClusteringComparator comparator, 
Slice.Bound start, Slice.Bound end)
 +public static boolean isEmpty(ClusteringComparator comparator, 
ClusteringBound start, ClusteringBound end)
  {
  assert start.isStart() && end.isEnd();
- return comparator.compare(end, start) < 0;
+ return comparator.compare(end, start) <= 0;
  }
  
  /**

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dc65a576/test/unit/org/apache/cassandra/cql3/validation/operations/DeleteTest.java
--

[1/6] cassandra git commit: Slice.isEmpty() returns false for some empty slices

2017-03-09 Thread slebresne

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 60d3292b0 -> 31dec3d54
  refs/heads/cassandra-3.11 7707a0ed5 -> dc65a5765
  refs/heads/trunk b5a5fbe1f -> 9e8e8914d


Slice.isEmpty() returns false for some empty slices

patch by Sylvain Lebresne; reviewed by Branimir Lambov for CASSANDRA-13305


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/31dec3d5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/31dec3d5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/31dec3d5

Branch: refs/heads/cassandra-3.0
Commit: 31dec3d548ae2c76d7c8bf4bffa9d506f670f756
Parents: 60d3292
Author: Sylvain Lebresne 
Authored: Thu Mar 9 11:58:40 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Mar 9 11:58:40 2017 +0100

--
 CHANGES.txt   |  1 +
 src/java/org/apache/cassandra/db/Slice.java   |  2 +-
 .../cql3/validation/operations/DeleteTest.java| 18 ++
 3 files changed, 20 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/31dec3d5/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 0979852..1876922 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.13
+ * Slice.isEmpty() returns false for some empty slices (CASSANDRA-13305)
  * Add formatted row output to assertEmpty in CQL Tester (CASSANDRA-13238)
 Merged from 2.2:
  * Fix GRANT/REVOKE when keyspace isn't specified (CASSANDRA-13053)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/31dec3d5/src/java/org/apache/cassandra/db/Slice.java
--
diff --git a/src/java/org/apache/cassandra/db/Slice.java 
b/src/java/org/apache/cassandra/db/Slice.java
index 7fde45e..3c645dc 100644
--- a/src/java/org/apache/cassandra/db/Slice.java
+++ b/src/java/org/apache/cassandra/db/Slice.java
@@ -160,7 +160,7 @@ public class Slice
 public static boolean isEmpty(ClusteringComparator comparator, Slice.Bound 
start, Slice.Bound end)
 {
 assert start.isStart() && end.isEnd();
-return comparator.compare(end, start) < 0;
+return comparator.compare(end, start) <= 0;
 }
 
 /**

http://git-wip-us.apache.org/repos/asf/cassandra/blob/31dec3d5/test/unit/org/apache/cassandra/cql3/validation/operations/DeleteTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/cql3/validation/operations/DeleteTest.java 
b/test/unit/org/apache/cassandra/cql3/validation/operations/DeleteTest.java
index 09098ac..9d7d4a3 100644
--- a/test/unit/org/apache/cassandra/cql3/validation/operations/DeleteTest.java
+++ b/test/unit/org/apache/cassandra/cql3/validation/operations/DeleteTest.java
@@ -1292,6 +1292,24 @@ public class DeleteTest extends CQLTester
 }
 
 /**
+ * Test for CASSANDRA-13305
+ */
+@Test
+public void testWithEmptyRange() throws Throwable
+{
+createTable("CREATE TABLE %s (k text, a int, b int, PRIMARY KEY (k, a, 
b))");
+
+// Both of the following should be doing nothing, but before #13305 
this inserted broken ranges. We do it twice
+// and the follow-up delete mainly as a way to show the bug as the 
combination of this will trigger an assertion
+// in RangeTombstoneList pre-#13305 showing that something wrong 
happened.
+execute("DELETE FROM %s WHERE k = ? AND a >= ? AND a < ?", "a", 1, 1);
+execute("DELETE FROM %s WHERE k = ? AND a >= ? AND a < ?", "a", 1, 1);
+
+execute("DELETE FROM %s WHERE k = ? AND a >= ? AND a < ?", "a", 0, 2);
+}
+
+
+/**
  * Checks if the memtable is empty or not
  * @return {@code true} if the memtable is empty, {@code false} otherwise.
  */

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-03-09 Thread slebresne

Merge branch 'cassandra-3.0' into cassandra-3.11

* cassandra-3.0:
  Slice.isEmpty() returns false for some empty slices


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/dc65a576
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/dc65a576
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/dc65a576

Branch: refs/heads/cassandra-3.11
Commit: dc65a576553f8766076920c8b639d80763d6e1f5
Parents: 7707a0e 31dec3d
Author: Sylvain Lebresne 
Authored: Thu Mar 9 11:59:32 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Mar 9 11:59:32 2017 +0100

--
 CHANGES.txt   |  1 +
 src/java/org/apache/cassandra/db/Slice.java   |  2 +-
 .../cql3/validation/operations/DeleteTest.java| 18 ++
 3 files changed, 20 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/dc65a576/CHANGES.txt
--
diff --cc CHANGES.txt
index f73dc12,1876922..2772fc2
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,15 -1,11 +1,16 @@@
 -3.0.13
 +3.11.0
 + * Fix equality comparisons of columns using the duration type 
(CASSANDRA-13174)
 + * Obfuscate password in stress-graphs (CASSANDRA-12233)
 + * Move to FastThreadLocalThread and FastThreadLocal (CASSANDRA-13034)
 + * nodetool stopdaemon errors out (CASSANDRA-13030)
 + * Tables in system_distributed should not use gcgs of 0 (CASSANDRA-12954)
 + * Fix primary index calculation for SASI (CASSANDRA-12910)
 + * More fixes to the TokenAllocator (CASSANDRA-12990)
 + * NoReplicationTokenAllocator should work with zero replication factor 
(CASSANDRA-12983)
 + * Address message coalescing regression (CASSANDRA-12676)
 +Merged from 3.0:
+  * Slice.isEmpty() returns false for some empty slices (CASSANDRA-13305)
   * Add formatted row output to assertEmpty in CQL Tester (CASSANDRA-13238)
 -Merged from 2.2:
 - * Fix GRANT/REVOKE when keyspace isn't specified (CASSANDRA-13053)
 -
 -
 -3.0.12
   * Prevent data loss on upgrade 2.1 - 3.0 by adding component separator to 
LogRecord absolute path (CASSANDRA-13294)
   * Improve testing on macOS by eliminating sigar logging (CASSANDRA-13233)
   * Cqlsh copy-from should error out when csv contains invalid data for 
collections (CASSANDRA-13071)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dc65a576/src/java/org/apache/cassandra/db/Slice.java
--
diff --cc src/java/org/apache/cassandra/db/Slice.java
index c3da222,3c645dc..4b36677
--- a/src/java/org/apache/cassandra/db/Slice.java
+++ b/src/java/org/apache/cassandra/db/Slice.java
@@@ -157,10 -157,10 +157,10 @@@ public class Slic
   * @return whether the slice formed by {@code start} and {@code end} is
   * empty or not.
   */
 -public static boolean isEmpty(ClusteringComparator comparator, 
Slice.Bound start, Slice.Bound end)
 +public static boolean isEmpty(ClusteringComparator comparator, 
ClusteringBound start, ClusteringBound end)
  {
  assert start.isStart() && end.isEnd();
- return comparator.compare(end, start) < 0;
+ return comparator.compare(end, start) <= 0;
  }
  
  /**

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dc65a576/test/unit/org/apache/cassandra/cql3/validation/operations/DeleteTest.java
--

[1/2] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-02-23 Thread slebresne

Repository: cassandra
Updated Branches:
  refs/heads/trunk 831c05b1c -> 2d8be34a2


Merge branch 'cassandra-3.0' into cassandra-3.11

* cassandra-3.0:
  Legacy deserializer can create unexpected boundary range tombstones


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1dc1aa19
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1dc1aa19
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1dc1aa19

Branch: refs/heads/trunk
Commit: 1dc1aa1982a7ab84034c95fa6ce6b3e4e2346fd2
Parents: 6487876 ab71748
Author: Sylvain Lebresne 
Authored: Thu Feb 23 14:37:35 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Feb 23 14:37:35 2017 +0100

--
 CHANGES.txt |   1 +
 .../cassandra/db/UnfilteredDeserializer.java| 334 ++-
 .../cassandra/db/rows/RangeTombstoneMarker.java |   1 -
 .../apache/cassandra/service/DataResolver.java  |  31 +-
 .../cassandra/db/OldFormatDeserializerTest.java | 110 ++
 .../cassandra/service/DataResolverTest.java | 125 ++-
 6 files changed, 432 insertions(+), 170 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1dc1aa19/CHANGES.txt
--
diff --cc CHANGES.txt
index 233898f,386029e..f5b9d28
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,18 -1,19 +1,19 @@@
 -3.0.12
 +3.11.0
 + * Fix equality comparisons of columns using the duration type 
(CASSANDRA-13174)
 + * Obfuscate password in stress-graphs (CASSANDRA-12233)
 + * Move to FastThreadLocalThread and FastThreadLocal (CASSANDRA-13034)
 + * nodetool stopdaemon errors out (CASSANDRA-13030)
 + * Tables in system_distributed should not use gcgs of 0 (CASSANDRA-12954)
 + * Fix primary index calculation for SASI (CASSANDRA-12910)
 + * More fixes to the TokenAllocator (CASSANDRA-12990)
 + * NoReplicationTokenAllocator should work with zero replication factor 
(CASSANDRA-12983)
 +Merged from 3.0:
+  * Legacy deserializer can create unexpected boundary range tombstones 
(CASSANDRA-13237)
   * Remove unnecessary assertion from AntiCompactionTest (CASSANDRA-13070)
   * Fix cqlsh COPY for dates before 1900 (CASSANDRA-13185)
 -Merged from 2.2
 - * Fix flaky LongLeveledCompactionStrategyTest (CASSANDRA-12202)
 - * Fix failing COPY TO STDOUT (CASSANDRA-12497)
 - * Fix ColumnCounter::countAll behaviour for reverse queries (CASSANDRA-13222)
 - * Exceptions encountered calling getSeeds() breaks OTC thread 
(CASSANDRA-13018)
 -Merged from 2.1:
 - * Log stacktrace of uncaught exceptions (CASSANDRA-13108)
 -
 -3.0.11
   * Use keyspace replication settings on system.size_estimates table 
(CASSANDRA-9639)
   * Add vm.max_map_count StartupCheck (CASSANDRA-13008)
 - * Hint related logging should include the IP address of the destination in 
addition to 
 + * Hint related logging should include the IP address of the destination in 
addition to
 host ID (CASSANDRA-13205)
   * Reloading logback.xml does not work (CASSANDRA-13173)
   * Lightweight transactions temporarily fail after upgrade from 2.1 to 3.0 
(CASSANDRA-13109)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1dc1aa19/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
--
diff --cc src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
index 2c3bc1b,42a806a..79b8636
--- a/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
+++ b/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
@@@ -317,16 -331,7 +331,7 @@@ public abstract class UnfilteredDeseria
  }
  }
  
- private boolean isRow(LegacyLayout.LegacyAtom atom)
- {
- if (atom.isCell())
- return true;
- 
- LegacyLayout.LegacyRangeTombstone tombstone = 
atom.asRangeTombstone();
- return tombstone.isCollectionTombstone() || 
tombstone.isRowDeletion(metadata);
- }
- 
 -public int compareNextTo(Slice.Bound bound) throws IOException
 +public int compareNextTo(ClusteringBound bound) throws IOException
  {
  if (!hasNext())
  throw new IllegalStateException();
@@@ -397,13 -405,27 +405,28 @@@
  
  private Unfiltered next;
  
- private UnfilteredIterator(DeletionTime partitionDeletion)
+ UnfilteredIterator(CFMetaData metadata,
+DeletionTime partitionDeletion,
+SerializationHelper helper,
+Supplier atomReader)
  {
+ this.metadata = metadata;
+ this.helper = helper;
  this.grouper = new

[2/2] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2017-02-23 Thread slebresne

Merge branch 'cassandra-3.11' into trunk

* cassandra-3.11:


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2d8be34a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2d8be34a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2d8be34a

Branch: refs/heads/trunk
Commit: 2d8be34a240db84d4368a4d30a2258ca6e67043f
Parents: 831c05b 1dc1aa1
Author: Sylvain Lebresne 
Authored: Thu Feb 23 15:48:25 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Feb 23 15:48:25 2017 +0100

--

--

[2/5] cassandra git commit: Legacy deserializer can create unexpected boundary range tombstones

2017-02-23 Thread slebresne

Legacy deserializer can create unexpected boundary range tombstones

patch by Sylvain Lebresne; reviewed by Branimir Lambov for CASSANDRA-13237


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ab717484
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ab717484
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ab717484

Branch: refs/heads/cassandra-3.11
Commit: ab7174849599c62f4bef3cb719c644bae13e9321
Parents: 42977db
Author: Sylvain Lebresne 
Authored: Thu Feb 23 14:32:03 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Feb 23 14:32:34 2017 +0100

--
 CHANGES.txt |   1 +
 .../cassandra/db/UnfilteredDeserializer.java| 343 ++-
 .../cassandra/db/rows/RangeTombstoneMarker.java |   2 +-
 .../apache/cassandra/service/DataResolver.java  |  31 +-
 .../cassandra/db/OldFormatDeserializerTest.java | 110 ++
 .../cassandra/service/DataResolverTest.java | 129 ++-
 6 files changed, 436 insertions(+), 180 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ab717484/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e978a5c..386029e 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.12
+ * Legacy deserializer can create unexpected boundary range tombstones 
(CASSANDRA-13237)
  * Remove unnecessary assertion from AntiCompactionTest (CASSANDRA-13070)
  * Fix cqlsh COPY for dates before 1900 (CASSANDRA-13185)
 Merged from 2.2

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ab717484/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
--
diff --git a/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java 
b/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
index a2d51e13..42a806a 100644
--- a/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
+++ b/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java
@@ -20,7 +20,9 @@ package org.apache.cassandra.db;
 import java.io.IOException;
 import java.io.IOError;
 import java.util.*;
+import java.util.function.Supplier;
 
+import com.google.common.annotations.VisibleForTesting;
 import com.google.common.collect.Iterables;
 import com.google.common.collect.PeekingIterator;
 
@@ -265,11 +267,23 @@ public abstract class UnfilteredDeserializer
   boolean readAllAsDynamic)
 {
 super(metadata, in, helper);
-this.iterator = new UnfilteredIterator(partitionDeletion);
+this.iterator = new UnfilteredIterator(metadata, 
partitionDeletion, helper, this::readAtom);
 this.readAllAsDynamic = readAllAsDynamic;
 this.lastConsumedPosition = currentPosition();
 }
 
+private LegacyLayout.LegacyAtom readAtom()
+{
+try
+{
+return LegacyLayout.readLegacyAtom(metadata, in, 
readAllAsDynamic);
+}
+catch (IOException e)
+{
+throw new IOError(e);
+}
+}
+
 public void setSkipStatic()
 {
 this.skipStatic = true;
@@ -317,15 +331,6 @@ public abstract class UnfilteredDeserializer
 }
 }
 
-private boolean isRow(LegacyLayout.LegacyAtom atom)
-{
-if (atom.isCell())
-return true;
-
-LegacyLayout.LegacyRangeTombstone tombstone = 
atom.asRangeTombstone();
-return tombstone.isCollectionTombstone() || 
tombstone.isRowDeletion(metadata);
-}
-
 public int compareNextTo(Slice.Bound bound) throws IOException
 {
 if (!hasNext())
@@ -389,19 +394,36 @@ public abstract class UnfilteredDeserializer
 // Groups atoms from the input into proper Unfiltered.
 // Note: this could use guava AbstractIterator except that we want to 
be able to clear
 // the internal state of the iterator so it's cleaner to do it 
ourselves.
-private class UnfilteredIterator implements PeekingIterator
+@VisibleForTesting
+static class UnfilteredIterator implements PeekingIterator
 {
 private final AtomIterator atoms;
 private final LegacyLayout.CellGrouper grouper;
 private final TombstoneTracker tombstoneTracker;
+private final CFMetaData metadata;
+private final SerializationHelper helper;
 
 private Unfiltered next;
 
-private UnfilteredIterator(DeletionTime partitionDeletion)
+UnfilteredIterator(CFMetaData metadata,
+

1 2 3 4 5 6 7 8 9 10 >

1 - 100 of 5419 matches

Mail list logo