[jira] [Commented] (CASSANDRA-13935) Indexes creation should have IF EXISTS on its String representation

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401439#comment-16401439
 ] 

Kurt Greaves commented on CASSANDRA-13935:
--

Thanks Javier, change seems straightforward enough to me. I've kicked off unit 
tests.

|[3.0|https://circleci.com/gh/kgreav/cassandra/142]|[3.11|https://circleci.com/gh/kgreav/cassandra/143]|[trunk|https://circleci.com/gh/kgreav/cassandra/144]|

> Indexes creation should have IF EXISTS on its String representation
> ---
>
> Key: CASSANDRA-13935
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13935
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL, Secondary Indexes
> Environment: Ubuntu 16.04.2 LTS
> java version "1.8.0_144"
> Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
> Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
>Reporter: Javier Canillas
>Assignee: Javier Canillas
>Priority: Trivial
> Fix For: 3.11.x
>
> Attachments: 13935-3.0.txt, 13935-3.11.txt, 13935-trunk.txt
>
>
> I came across something that bothers me a lot. I'm using snapshots to backup 
> data from my Cassandra cluster in case something really bad happens (like 
> dropping a table or a keyspace).
> Exercising the recovery actions from those backups, I discover that the 
> schema put on the file "schema.cql" as a result of the snapshot has the 
> "CREATE IF NOT EXISTS" for the table, but not for the indexes.
> When restoring from snapshots, and relying on the execution of these schemas 
> to build up the table structure, everything seems fine for tables without 
> secondary indexes, but for the ones that make use of them, the execution of 
> these statements fail miserably.
> Here I paste a generated schema.cql content for a table with indexes:
> CREATE TABLE IF NOT EXISTS keyspace1.table1 (
>   id text PRIMARY KEY,
>   content text,
>   last_update_date date,
>   last_update_date_time timestamp)
>   WITH ID = f1045fc0-2f59-11e7-95ec-295c3c064920
>   AND bloom_filter_fp_chance = 0.01
>   AND dclocal_read_repair_chance = 0.1
>   AND crc_check_chance = 1.0
>   AND default_time_to_live = 864
>   AND gc_grace_seconds = 864000
>   AND min_index_interval = 128
>   AND max_index_interval = 2048
>   AND memtable_flush_period_in_ms = 0
>   AND read_repair_chance = 0.0
>   AND speculative_retry = '99PERCENTILE'
>   AND caching = { 'keys': 'NONE', 'rows_per_partition': 'NONE' }
>   AND compaction = { 'max_threshold': '32', 'min_threshold': '4', 
> 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy' }
>   AND compression = { 'chunk_length_in_kb': '64', 'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor' }
>   AND cdc = false
>   AND extensions = {  };
> CREATE INDEX table1_last_update_date_idx ON keyspace1.table1 
> (last_update_date);
> I think the last part should be:
> CREATE INDEX IF NOT EXISTS table1_last_update_date_idx ON keyspace1.table1 
> (last_update_date);



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14317) Auditing Plug-in for Cassandra

[Laxmikant Upadhyay (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401436#comment-16401436
 ] 

Laxmikant Upadhyay commented on CASSANDRA-14317:


Please assign this to me.

> Auditing Plug-in for Cassandra
> --
>
> Key: CASSANDRA-14317
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14317
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Tools
> Environment: Cassandra 3.11.x
>Reporter: Anuj Wadehra
>Priority: Major
>  Labels: security
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Cassandra lacks database auditing feature. Till the new feature is 
> implemented as part of CASSANDRA-12151, a database auditing plug-in can be 
> built. The plug-in can be implemented and plugged into Cassandra by 
> customizing components such as Query Handler , Authenticator and Role 
> Manager. The Auditing plug-in shall log all CQL queries and user logins. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14317) Auditing Plug-in for Cassandra

[Anuj Wadehra (JIRA)]

Anuj Wadehra created CASSANDRA-14317:


 Summary: Auditing Plug-in for Cassandra
 Key: CASSANDRA-14317
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14317
 Project: Cassandra
  Issue Type: New Feature
  Components: Tools
 Environment: Cassandra 3.11.x
Reporter: Anuj Wadehra


Cassandra lacks database auditing feature. Till the new feature is implemented 
as part of CASSANDRA-12151, a database auditing plug-in can be built. The 
plug-in can be implemented and plugged into Cassandra by customizing components 
such as Query Handler , Authenticator and Role Manager. The Auditing plug-in 
shall log all CQL queries and user logins. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-5836) Seed nodes should be able to bootstrap without manual intervention

[Jeremiah Jordan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-5836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401409#comment-16401409
 ] 

Jeremiah Jordan commented on CASSANDRA-5836:


People used to recommend setting auto bootstrap false to make sure that you 
didn’t end up with a node owning 100% of the ring. Since we have changed NTS 
such that you can’t set the new DC name in until after there are nodes in that 
DC this is no longer something someone could easily do by going in the “wrong” 
order and altering keyspaces first.

> Seed nodes should be able to bootstrap without manual intervention
> --
>
> Key: CASSANDRA-5836
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5836
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Bill Hathaway
>Priority: Minor
>
> The current logic doesn't allow a seed node to be bootstrapped.  If a user 
> wants to bootstrap a node configured as a seed (for example to replace a seed 
> node via replace_token), they first need to remove the node's own IP from the 
> seed list, and then start the bootstrap process.  This seems like an 
> unnecessary step since a node never uses itself as a seed.
> I think it would be a better experience if the logic was changed to allow a 
> seed node to bootstrap without manual intervention when there are other seed 
> nodes up in a ring.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14118) Refactor write path

[Dikang Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dikang Gu updated CASSANDRA-14118:
--
Reviewer: Blake Eggleston
  Status: Patch Available  (was: Open)

> Refactor write path
> ---
>
> Key: CASSANDRA-14118
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14118
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Dikang Gu
>Assignee: Dikang Gu
>Priority: Major
>
> As part of the pluggable storage engine effort, we'd like to modularize the 
> write path related code, make it to be independent from existing storage 
> engine implementation details.
> For now, refer to 
> https://docs.google.com/document/d/1suZlvhzgB6NIyBNpM9nxoHxz_Ri7qAm-UEO8v8AIFsc
>  for high level designs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-5836) Seed nodes should be able to bootstrap without manual intervention

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-5836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401328#comment-16401328
 ] 

Kurt Greaves commented on CASSANDRA-5836:
-

bq. But then I don't see at all where does the recommendation to set 
auto_bootstrap=false for new DC come from?
That was always a datastax recommendation so I don't know where it came from. 
As I'm sure you're aware, the Cassandra docs are quite sparse in the operations 
area, but all of this should be documented properly.

bq. Unfortunately, it already is
I'm well aware of that. I'm saying it shouldn't be. And we certainly shouldn't 
be making a bad situation worse.

bq. That's already a contradiction, don't you think? And more precisely it 
should be spelled as "if a node believes it is the very first one". A big 
question to me still: can this be done in the code reliably?

No, I don't think so. Caveat was probably the wrong word here, because it's not 
a caveat. It's literally that it's *irrelevant* what the first node does. If 
auto_bootstrap is true for the first node, it's a no-op, if it's false, it's a 
defined no-op. The first node still respects auto_bootstrap, but the result is 
the same for either true or false. This is always going to be the case.
If it can be done by hand it can be done by code. The first node would be 
defined as a node that only has itself as a seed, and no existing knowledge of 
any other node in the cluster. This is quite straightforward (but not part of 
my patch yet).

bq. This attitude is exactly what makes Cassandra hard to use in my experience. 
 I cannot even count the number of times when I had to dive deeply into the 
source code trying to figure some detail which was not properly documented, 
because the devs thought the same: users don't need to know about it...
I think you misunderstand. It's not that attitude that makes Cassandra hard to 
use, it's the fact that things get implemented without documentation and 
problems get missed when considering the effects on all aspects of Cassandra, 
because it's so large and complex. The only reason you need to dive into the 
code/jira tickets is because there's no accurate documentation for what you're 
looking for, or reasoning for why something behaves as it does. 

My point is precisely that users don't need to know about it, therefore they 
shouldn't have to dig into the code. If they have to dig into the code the 
implementation is likely wrong. We should be aiming to make it so that no user 
ever has to even consider looking in the code to figure out how seeds work, let 
alone even the documentation. It literally should not even cross their mind 
short of the fact that they have to set some seeds.




> Seed nodes should be able to bootstrap without manual intervention
> --
>
> Key: CASSANDRA-5836
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5836
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Bill Hathaway
>Priority: Minor
>
> The current logic doesn't allow a seed node to be bootstrapped.  If a user 
> wants to bootstrap a node configured as a seed (for example to replace a seed 
> node via replace_token), they first need to remove the node's own IP from the 
> seed list, and then start the bootstrap process.  This seems like an 
> unnecessary step since a node never uses itself as a seed.
> I think it would be a better experience if the logic was changed to allow a 
> seed node to bootstrap without manual intervention when there are other seed 
> nodes up in a ring.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[4/5] cassandra git commit: Abstract streaming for pluggable storage

[bdeggleston]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9714a7c8/src/java/org/apache/cassandra/db/streaming/CompressedInputStream.java
--
diff --git 
a/src/java/org/apache/cassandra/db/streaming/CompressedInputStream.java 
b/src/java/org/apache/cassandra/db/streaming/CompressedInputStream.java
new file mode 100644
index 000..2f56786
--- /dev/null
+++ b/src/java/org/apache/cassandra/db/streaming/CompressedInputStream.java
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.db.streaming;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.channels.ReadableByteChannel;
+import java.util.Iterator;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.function.DoubleSupplier;
+
+import com.google.common.collect.Iterators;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import io.netty.util.concurrent.FastThreadLocalThread;
+import org.apache.cassandra.io.compress.CompressionMetadata;
+import org.apache.cassandra.io.util.DataInputPlus;
+import org.apache.cassandra.io.util.FileUtils;
+import org.apache.cassandra.io.util.RebufferingInputStream;
+import 
org.apache.cassandra.db.streaming.CassandraStreamReader.StreamDeserializer;
+import org.apache.cassandra.utils.ChecksumType;
+import org.apache.cassandra.utils.WrappedRunnable;
+
+/**
+ * InputStream which reads data from underlining source with given {@link 
CompressionInfo}. Uses {@link #buffer} as a buffer
+ * for uncompressed data (which is read by stream consumers - {@link 
StreamDeserializer} in this case).
+ */
+public class CompressedInputStream extends RebufferingInputStream implements 
AutoCloseable
+{
+
+private static final Logger logger = 
LoggerFactory.getLogger(CompressedInputStream.class);
+
+private final CompressionInfo info;
+// chunk buffer
+private final BlockingQueue dataBuffer;
+private final DoubleSupplier crcCheckChanceSupplier;
+
+/**
+ * The base offset of the current {@link #buffer} from the beginning of 
the stream.
+ */
+private long bufferOffset = 0;
+
+/**
+ * The current {@link CompressedCassandraStreamReader#sections} offset in 
the stream.
+ */
+private long current = 0;
+
+private final ChecksumType checksumType;
+
+private static final int CHECKSUM_LENGTH = 4;
+
+/**
+ * Indicates there was a problem when reading from source stream.
+ * When this is added to the dataBuffer by the stream Reader,
+ * it is expected that the readException variable is populated
+ * with the cause of the error when reading from source stream, so it is
+ * thrown to the consumer on subsequent read operation.
+ */
+private static final ByteBuffer POISON_PILL = ByteBuffer.wrap(new byte[0]);
+
+private volatile IOException readException = null;
+
+private long totalCompressedBytesRead;
+
+/**
+ * @param source Input source to read compressed data from
+ * @param info Compression info
+ */
+public CompressedInputStream(DataInputPlus source, CompressionInfo info, 
ChecksumType checksumType, DoubleSupplier crcCheckChanceSupplier)
+{
+super(ByteBuffer.allocateDirect(info.parameters.chunkLength()));
+buffer.limit(buffer.position()); // force the buffer to appear 
"consumed" so that it triggers reBuffer on the first read
+this.info = info;
+this.dataBuffer = new 
ArrayBlockingQueue<>(Math.min(info.chunks.length, 1024));
+this.crcCheckChanceSupplier = crcCheckChanceSupplier;
+this.checksumType = checksumType;
+
+new FastThreadLocalThread(new Reader(source, info, 
dataBuffer)).start();
+}
+
+/**
+ * Invoked when crossing into the next stream boundary in {@link 
CompressedCassandraStreamReader#sections}.
+ */
+public void position(long position) throws IOException
+{
+if (readException != null)
+throw readException;
+
+assert position >= current : "stream can only read forward.";
+

[3/5] cassandra git commit: Abstract streaming for pluggable storage

[bdeggleston]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9714a7c8/src/java/org/apache/cassandra/streaming/StreamSession.java
--
diff --git a/src/java/org/apache/cassandra/streaming/StreamSession.java 
b/src/java/org/apache/cassandra/streaming/StreamSession.java
index 4085c43..adf5d76 100644
--- a/src/java/org/apache/cassandra/streaming/StreamSession.java
+++ b/src/java/org/apache/cassandra/streaming/StreamSession.java
@@ -17,27 +17,17 @@
  */
 package org.apache.cassandra.streaming;
 
-import java.net.InetAddress;
-import java.net.InetSocketAddress;
 import java.net.SocketTimeoutException;
 import java.util.*;
 import java.util.concurrent.*;
 import java.util.concurrent.atomic.AtomicBoolean;
 
 import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Predicate;
-import com.google.common.base.Predicates;
 import com.google.common.collect.*;
 import com.google.common.util.concurrent.Futures;
 
 import org.apache.cassandra.concurrent.ScheduledExecutors;
 import org.apache.cassandra.config.DatabaseDescriptor;
-import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
-import org.apache.cassandra.db.lifecycle.SSTableIntervalTree;
-import org.apache.cassandra.db.lifecycle.SSTableSet;
-import org.apache.cassandra.db.lifecycle.View;
-import org.apache.cassandra.io.sstable.SSTableMultiWriter;
-import org.apache.cassandra.io.sstable.format.SSTableReader;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -45,7 +35,6 @@ import io.netty.channel.Channel;
 import io.netty.channel.ChannelId;
 import org.apache.cassandra.db.ColumnFamilyStore;
 import org.apache.cassandra.db.Keyspace;
-import org.apache.cassandra.db.PartitionPosition;
 import org.apache.cassandra.dht.Range;
 import org.apache.cassandra.dht.Token;
 import org.apache.cassandra.gms.*;
@@ -54,19 +43,13 @@ import org.apache.cassandra.metrics.StreamingMetrics;
 import org.apache.cassandra.net.MessagingService;
 import org.apache.cassandra.net.async.OutboundConnectionIdentifier;
 import org.apache.cassandra.schema.TableId;
-import org.apache.cassandra.service.ActiveRepairService;
 import org.apache.cassandra.streaming.async.NettyStreamingMessageSender;
 import org.apache.cassandra.streaming.messages.*;
 import org.apache.cassandra.utils.FBUtilities;
 import org.apache.cassandra.utils.JVMStabilityInspector;
-import org.apache.cassandra.utils.Pair;
-import org.apache.cassandra.utils.concurrent.Ref;
-import org.apache.cassandra.utils.concurrent.Refs;
 
 /**
- * Handles the streaming a one or more section of one of more sstables to and 
from a specific
- * remote node. The sending side performs a block-level transfer of the source 
sstable, while the receiver
- * must deserilaize that data stream into an partitions and rows, and then 
write that out as an sstable.
+ * Handles the streaming a one or more streams to and from a specific remote 
node.
  *
  * Both this node and the remote one will create a similar symmetrical {@link 
StreamSession}. A streaming
  * session has the following life-cycle:
@@ -98,15 +81,15 @@ import org.apache.cassandra.utils.concurrent.Refs;
  *
  *   (a) The streaming phase is started at each node by calling {@link 
StreamSession#startStreamingFiles(boolean)}.
  *   This will send, sequentially on each outbound streaming connection 
(see {@link NettyStreamingMessageSender}),
- *   an {@link OutgoingFileMessage} for each file in each of the {@link 
StreamTransferTask}.
- *   Each {@link OutgoingFileMessage} consists of a {@link 
FileMessageHeader} that contains metadata about the file
- *   being streamed, followed by the file content itself. Once all the 
files for a {@link StreamTransferTask} are sent,
+ *   an {@link OutgoingStreamMessage} for each stream in each of the 
{@link StreamTransferTask}.
+ *   Each {@link OutgoingStreamMessage} consists of a {@link 
StreamMessageHeader} that contains metadata about
+ *   the stream, followed by the stream content itself. Once all the files 
for a {@link StreamTransferTask} are sent,
  *   the task is marked complete {@link StreamTransferTask#complete(int)}.
- *   (b) On the receiving side, a SSTable will be written for the incoming 
file, and once the file is fully received,
- *   the file will be marked as complete ({@link 
StreamReceiveTask#received(SSTableMultiWriter)}). When all files
- *   for the {@link StreamReceiveTask} have been received, the sstables 
are added to the CFS (and 2ndary indexes/MV are built),
+ *   (b) On the receiving side, the incoming data is written to disk, and once 
the stream is fully received,
+ *   it will be marked as complete ({@link 
StreamReceiveTask#received(IncomingStream)}). When all streams
+ *   for the {@link StreamReceiveTask} have been received, the data is 
added to the CFS (and 2ndary indexes/MV are built),
  *and the task is marked complete ({@link 

[jira] [Updated] (CASSANDRA-14115) Refactor streaming

[Blake Eggleston (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Blake Eggleston updated CASSANDRA-14115:

Resolution: Fixed
Status: Resolved  (was: Ready to Commit)

Clean test run 
[here|https://circleci.com/workflow-run/87f9f1bd-7d40-4f2c-af9f-8e5dbd4985a6]

Committed as {{9714a7c817b64a3358f69e536535c756c5c6df48}}

> Refactor streaming
> --
>
> Key: CASSANDRA-14115
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14115
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Streaming and Messaging
>Reporter: Dikang Gu
>Assignee: Blake Eggleston
>Priority: Major
> Fix For: 4.0
>
>
> As part of the pluggable storage engine effort, we'd like to modularize the 
> streaming related code, make it to be independent from existing storage 
> engine implementation details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[5/5] cassandra git commit: Abstract streaming for pluggable storage

[bdeggleston]

Abstract streaming for pluggable storage

Patch by Blake Eggleston; Reviewed by Jason Brown for CASSANDRA-14115


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9714a7c8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9714a7c8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9714a7c8

Branch: refs/heads/trunk
Commit: 9714a7c817b64a3358f69e536535c756c5c6df48
Parents: 253c003
Author: Blake Eggleston 
Authored: Sun Mar 4 14:44:30 2018 -0800
Committer: Blake Eggleston 
Committed: Thu Mar 15 16:40:05 2018 -0700

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |  10 +
 .../db/streaming/CassandraIncomingFile.java | 117 +++
 .../db/streaming/CassandraOutgoingFile.java | 151 +
 .../db/streaming/CassandraStreamHeader.java | 212 +
 .../db/streaming/CassandraStreamManager.java| 166 ++
 .../db/streaming/CassandraStreamReader.java | 285 +
 .../db/streaming/CassandraStreamReceiver.java   | 248 +++
 .../db/streaming/CassandraStreamWriter.java | 176 +++
 .../CompressedCassandraStreamReader.java| 131 
 .../CompressedCassandraStreamWriter.java| 153 ++
 .../db/streaming/CompressedInputStream.java | 285 +
 .../cassandra/db/streaming/CompressionInfo.java | 110 +++
 .../cassandra/db/streaming/package-info.java|  53 
 .../org/apache/cassandra/dht/RangeStreamer.java |   4 +-
 .../cassandra/io/sstable/SSTableLoader.java |  18 +-
 .../repair/AsymmetricLocalSyncTask.java |   1 -
 .../apache/cassandra/repair/LocalSyncTask.java  |   2 +-
 .../cassandra/repair/StreamingRepairTask.java   |   2 +-
 .../cassandra/streaming/IncomingStream.java |  45 +++
 .../cassandra/streaming/OutgoingStream.java |  52 
 .../apache/cassandra/streaming/PreviewKind.java |  21 +-
 .../cassandra/streaming/ProgressInfo.java   |   4 +-
 .../apache/cassandra/streaming/SessionInfo.java |   8 +-
 .../cassandra/streaming/StreamCoordinator.java  |  40 ++-
 .../apache/cassandra/streaming/StreamHook.java  |  14 +-
 .../apache/cassandra/streaming/StreamPlan.java  |  20 +-
 .../cassandra/streaming/StreamReader.java   | 278 -
 .../cassandra/streaming/StreamReceiveTask.java  | 211 ++---
 .../cassandra/streaming/StreamReceiver.java |  58 
 .../cassandra/streaming/StreamResultFuture.java |   7 +-
 .../cassandra/streaming/StreamSession.java  | 199 
 .../cassandra/streaming/StreamTransferTask.java |  58 ++--
 .../cassandra/streaming/StreamWriter.java   | 173 ---
 .../cassandra/streaming/TableStreamManager.java |  57 
 .../async/NettyStreamingMessageSender.java  |  41 ++-
 .../async/StreamingInboundHandler.java  |  23 +-
 .../cassandra/streaming/async/package-info.java |  36 +--
 .../compress/CompressedInputStream.java | 285 -
 .../compress/CompressedStreamReader.java| 132 
 .../compress/CompressedStreamWriter.java| 154 --
 .../streaming/compress/CompressionInfo.java |  95 --
 .../management/ProgressInfoCompositeData.java   |   2 +-
 .../streaming/messages/FileMessageHeader.java   | 304 ---
 .../streaming/messages/IncomingFileMessage.java |  95 --
 .../messages/IncomingStreamMessage.java | 107 +++
 .../streaming/messages/OutgoingFileMessage.java | 158 --
 .../messages/OutgoingStreamMessage.java | 138 +
 .../streaming/messages/StreamInitMessage.java   |   9 +-
 .../streaming/messages/StreamMessage.java   |   2 +-
 .../streaming/messages/StreamMessageHeader.java | 144 +
 .../db/streaming/CassandraStreamHeaderTest.java |  50 +++
 .../streaming/CassandraStreamManagerTest.java   | 237 +++
 .../cassandra/dht/StreamStateStoreTest.java |   5 +-
 .../cassandra/io/sstable/LegacySSTableTest.java |  14 +-
 .../io/sstable/SSTableRewriterTest.java |  59 
 .../serializers/SerializationUtils.java |  68 +
 .../cassandra/streaming/StreamSessionTest.java  | 132 
 .../streaming/StreamTransferTaskTest.java   |  19 +-
 .../streaming/StreamingTransferTest.java|  27 +-
 .../async/NettyStreamingMessageSenderTest.java  |   5 +-
 .../async/StreamingInboundHandlerTest.java  |  23 +-
 .../compression/CompressedInputStreamTest.java  |   4 +-
 63 files changed, 3314 insertions(+), 2424 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9714a7c8/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 42add69..9458f19 100644

[2/5] cassandra git commit: Abstract streaming for pluggable storage

[bdeggleston]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9714a7c8/src/java/org/apache/cassandra/streaming/messages/FileMessageHeader.java
--
diff --git 
a/src/java/org/apache/cassandra/streaming/messages/FileMessageHeader.java 
b/src/java/org/apache/cassandra/streaming/messages/FileMessageHeader.java
deleted file mode 100644
index 13a3358..000
--- a/src/java/org/apache/cassandra/streaming/messages/FileMessageHeader.java
+++ /dev/null
@@ -1,304 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.cassandra.streaming.messages;
-
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.UUID;
-
-import org.apache.cassandra.db.SerializationHeader;
-import org.apache.cassandra.db.TypeSizes;
-import org.apache.cassandra.io.compress.CompressionMetadata;
-import org.apache.cassandra.io.sstable.format.SSTableFormat;
-import org.apache.cassandra.io.util.DataInputPlus;
-import org.apache.cassandra.io.util.DataOutputPlus;
-import org.apache.cassandra.io.sstable.format.Version;
-import org.apache.cassandra.locator.InetAddressAndPort;
-import org.apache.cassandra.net.CompactEndpointSerializationHelper;
-import org.apache.cassandra.net.MessagingService;
-import org.apache.cassandra.schema.TableId;
-import org.apache.cassandra.streaming.StreamSession;
-import org.apache.cassandra.streaming.compress.CompressionInfo;
-import org.apache.cassandra.utils.Pair;
-import org.apache.cassandra.utils.UUIDSerializer;
-
-/**
- * StreamingFileHeader is appended before sending actual data to describe what 
it's sending.
- */
-public class FileMessageHeader
-{
-public static FileMessageHeaderSerializer serializer = new 
FileMessageHeaderSerializer();
-
-public final TableId tableId;
-public UUID planId;
-public int sessionIndex;
-public final int sequenceNumber;
-/** SSTable version */
-public final Version version;
-
-/** SSTable format **/
-public final SSTableFormat.Type format;
-public final long estimatedKeys;
-public final List> sections;
-/**
- * Compression info for SSTable to send. Can be null if SSTable is not 
compressed.
- * On sender, this field is always null to avoid holding large number of 
Chunks.
- * Use compressionMetadata instead.
- */
-public final CompressionInfo compressionInfo;
-private final CompressionMetadata compressionMetadata;
-public final long repairedAt;
-public final UUID pendingRepair;
-public final int sstableLevel;
-public final SerializationHeader.Component header;
-public final InetAddressAndPort sender;
-
-/* cached size value */
-private transient final long size;
-
-private FileMessageHeader(TableId tableId,
- InetAddressAndPort sender,
- UUID planId,
- int sessionIndex,
- int sequenceNumber,
- Version version,
- SSTableFormat.Type format,
- long estimatedKeys,
- List> sections,
- CompressionInfo compressionInfo,
- long repairedAt,
- UUID pendingRepair,
- int sstableLevel,
- SerializationHeader.Component header)
-{
-this.tableId = tableId;
-this.sender = sender;
-this.planId = planId;
-this.sessionIndex = sessionIndex;
-this.sequenceNumber = sequenceNumber;
-this.version = version;
-this.format = format;
-this.estimatedKeys = estimatedKeys;
-this.sections = sections;
-this.compressionInfo = compressionInfo;
-this.compressionMetadata = null;
-this.repairedAt = repairedAt;
-this.pendingRepair = pendingRepair;
-this.sstableLevel = sstableLevel;
-this.header = header;
-this.size = calculateSize();
-}
-
-public FileMessageHeader(TableId tableId,
- 

[1/5] cassandra git commit: Abstract streaming for pluggable storage

[bdeggleston]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 253c003df -> 9714a7c81


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9714a7c8/test/unit/org/apache/cassandra/streaming/async/StreamingInboundHandlerTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/streaming/async/StreamingInboundHandlerTest.java
 
b/test/unit/org/apache/cassandra/streaming/async/StreamingInboundHandlerTest.java
index b4a736e..78a7879 100644
--- 
a/test/unit/org/apache/cassandra/streaming/async/StreamingInboundHandlerTest.java
+++ 
b/test/unit/org/apache/cassandra/streaming/async/StreamingInboundHandlerTest.java
@@ -20,7 +20,6 @@ package org.apache.cassandra.streaming.async;
 
 import java.io.EOFException;
 import java.io.IOException;
-import java.net.InetSocketAddress;
 import java.util.ArrayList;
 import java.util.UUID;
 
@@ -46,8 +45,8 @@ import org.apache.cassandra.streaming.StreamResultFuture;
 import org.apache.cassandra.streaming.StreamSession;
 import 
org.apache.cassandra.streaming.async.StreamingInboundHandler.SessionIdentifier;
 import org.apache.cassandra.streaming.messages.CompleteMessage;
-import org.apache.cassandra.streaming.messages.FileMessageHeader;
-import org.apache.cassandra.streaming.messages.IncomingFileMessage;
+import org.apache.cassandra.streaming.messages.StreamMessageHeader;
+import org.apache.cassandra.streaming.messages.IncomingStreamMessage;
 import org.apache.cassandra.streaming.messages.StreamInitMessage;
 import org.apache.cassandra.streaming.messages.StreamMessage;
 
@@ -125,7 +124,7 @@ public class StreamingInboundHandlerTest
 @Test
 public void StreamDeserializingTask_deriveSession_StreamInitMessage() 
throws InterruptedException, IOException
 {
-StreamInitMessage msg = new StreamInitMessage(REMOTE_ADDR, 0, 
UUID.randomUUID(), StreamOperation.REPAIR, true, UUID.randomUUID(), 
PreviewKind.ALL);
+StreamInitMessage msg = new StreamInitMessage(REMOTE_ADDR, 0, 
UUID.randomUUID(), StreamOperation.REPAIR, UUID.randomUUID(), PreviewKind.ALL);
 StreamingInboundHandler.StreamDeserializingTask task = handler.new 
StreamDeserializingTask(sid -> createSession(sid), null, channel);
 StreamSession session = task.deriveSession(msg);
 Assert.assertNotNull(session);
@@ -133,7 +132,7 @@ public class StreamingInboundHandlerTest
 
 private StreamSession createSession(SessionIdentifier sid)
 {
-return new StreamSession(sid.from, sid.from, (connectionId, 
protocolVersion) -> null, sid.sessionIndex, true, UUID.randomUUID(), 
PreviewKind.ALL);
+return new StreamSession(StreamOperation.BOOTSTRAP, sid.from, 
sid.from, (connectionId, protocolVersion) -> null, sid.sessionIndex, 
UUID.randomUUID(), PreviewKind.ALL);
 }
 
 @Test (expected = IllegalStateException.class)
@@ -147,9 +146,9 @@ public class StreamingInboundHandlerTest
 @Test (expected = IllegalStateException.class)
 public void StreamDeserializingTask_deriveSession_IFM_NoSession() throws 
InterruptedException, IOException
 {
-FileMessageHeader header = new FileMessageHeader(TableId.generate(), 
REMOTE_ADDR, UUID.randomUUID(), 0, 0,
- 
BigFormat.latestVersion, SSTableFormat.Type.BIG, 0, new ArrayList<>(), null, 0, 
UUID.randomUUID(), 0 , null);
-IncomingFileMessage msg = new IncomingFileMessage(null, header);
+StreamMessageHeader header = new 
StreamMessageHeader(TableId.generate(), REMOTE_ADDR, UUID.randomUUID(),
+ 0, 0, 0, 
UUID.randomUUID());
+IncomingStreamMessage msg = new IncomingStreamMessage(null, header);
 StreamingInboundHandler.StreamDeserializingTask task = handler.new 
StreamDeserializingTask(sid -> StreamManager.instance.findSession(sid.from, 
sid.planId, sid.sessionIndex), null, channel);
 task.deriveSession(msg);
 }
@@ -158,11 +157,11 @@ public class StreamingInboundHandlerTest
 public void StreamDeserializingTask_deriveSession_IFM_HasSession() throws 
InterruptedException, IOException
 {
 UUID planId = UUID.randomUUID();
-StreamResultFuture future = StreamResultFuture.initReceivingSide(0, 
planId, StreamOperation.REPAIR, REMOTE_ADDR, channel, true, UUID.randomUUID(), 
PreviewKind.ALL);
+StreamResultFuture future = StreamResultFuture.initReceivingSide(0, 
planId, StreamOperation.REPAIR, REMOTE_ADDR, channel, UUID.randomUUID(), 
PreviewKind.ALL);
 StreamManager.instance.register(future);
-FileMessageHeader header = new FileMessageHeader(TableId.generate(), 
REMOTE_ADDR, planId, 0, 0,
- 
BigFormat.latestVersion, SSTableFormat.Type.BIG, 0, new ArrayList<>(), null, 0, 
UUID.randomUUID(), 0 , null);
-IncomingFileMessage msg = new IncomingFileMessage(null, header);
+StreamMessageHeader header 

[jira] [Commented] (CASSANDRA-14118) Refactor write path

[Dikang Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401258#comment-16401258
 ] 

Dikang Gu commented on CASSANDRA-14118:
---

[~bdeggleston], wait for your feedback on  
[https://github.com/DikangGu/cassandra/commits/CASSANDRA-14118-v2], thanks!

> Refactor write path
> ---
>
> Key: CASSANDRA-14118
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14118
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Dikang Gu
>Assignee: Dikang Gu
>Priority: Major
>
> As part of the pluggable storage engine effort, we'd like to modularize the 
> write path related code, make it to be independent from existing storage 
> engine implementation details.
> For now, refer to 
> https://docs.google.com/document/d/1suZlvhzgB6NIyBNpM9nxoHxz_Ri7qAm-UEO8v8AIFsc
>  for high level designs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14118) Refactor write path

[Dikang Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401218#comment-16401218
 ] 

Dikang Gu edited comment on CASSANDRA-14118 at 3/15/18 11:29 PM:
-

[~jasobrown]-, I agree with you. That's what I'm trying to do in the original 
approach, [https://github.com/DikangGu/cassandra/commits/CASSANDRA-14118-v1.] 
Do you want me to continue that route?- 


was (Author: dikanggu):
[~jasobrown], I agree with you. That's what I'm trying to do in the original 
approach, [https://github.com/DikangGu/cassandra/commits/CASSANDRA-14118-v1.] 
Do you want me to continue that route? 

> Refactor write path
> ---
>
> Key: CASSANDRA-14118
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14118
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Dikang Gu
>Assignee: Dikang Gu
>Priority: Major
>
> As part of the pluggable storage engine effort, we'd like to modularize the 
> write path related code, make it to be independent from existing storage 
> engine implementation details.
> For now, refer to 
> https://docs.google.com/document/d/1suZlvhzgB6NIyBNpM9nxoHxz_Ri7qAm-UEO8v8AIFsc
>  for high level designs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14118) Refactor write path

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401220#comment-16401220
 ] 

Jason Brown commented on CASSANDRA-14118:
-

[~dikanggu] lol - we posted at the same time. Read my update, and then listen 
to [~bdeggleston] :)

> Refactor write path
> ---
>
> Key: CASSANDRA-14118
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14118
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Dikang Gu
>Assignee: Dikang Gu
>Priority: Major
>
> As part of the pluggable storage engine effort, we'd like to modularize the 
> write path related code, make it to be independent from existing storage 
> engine implementation details.
> For now, refer to 
> https://docs.google.com/document/d/1suZlvhzgB6NIyBNpM9nxoHxz_Ri7qAm-UEO8v8AIFsc
>  for high level designs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14118) Refactor write path

[Dikang Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401218#comment-16401218
 ] 

Dikang Gu commented on CASSANDRA-14118:
---

[~jasobrown], I agree with you. That's what I'm trying to do in the original 
approach, [https://github.com/DikangGu/cassandra/commits/CASSANDRA-14118-v1.] 
Do you want me to continue that route? 

> Refactor write path
> ---
>
> Key: CASSANDRA-14118
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14118
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Dikang Gu
>Assignee: Dikang Gu
>Priority: Major
>
> As part of the pluggable storage engine effort, we'd like to modularize the 
> write path related code, make it to be independent from existing storage 
> engine implementation details.
> For now, refer to 
> https://docs.google.com/document/d/1suZlvhzgB6NIyBNpM9nxoHxz_Ri7qAm-UEO8v8AIFsc
>  for high level designs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14118) Refactor write path

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401198#comment-16401198
 ] 

Jason Brown edited comment on CASSANDRA-14118 at 3/15/18 10:53 PM:
---

-[~dikanggu] I think the entire tracker needs to move out of the CFS, as only 
the original/existing storage engine will use it (as it's specific to the 
existing memtable, commitlog, and so on)-

Ignore this comment. What I stated is where we want to get to, but not quite 
yet.


was (Author: jasobrown):
[~dikanggu] I think the entire tracker needs to move out of the CFS, as only 
the original/existing storage engine will use it (as it's specific to the 
existing memtable, commitlog, and so on)

> Refactor write path
> ---
>
> Key: CASSANDRA-14118
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14118
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Dikang Gu
>Assignee: Dikang Gu
>Priority: Major
>
> As part of the pluggable storage engine effort, we'd like to modularize the 
> write path related code, make it to be independent from existing storage 
> engine implementation details.
> For now, refer to 
> https://docs.google.com/document/d/1suZlvhzgB6NIyBNpM9nxoHxz_Ri7qAm-UEO8v8AIFsc
>  for high level designs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14118) Refactor write path

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401198#comment-16401198
 ] 

Jason Brown commented on CASSANDRA-14118:
-

[~dikanggu] I think the entire tracker needs to move out of the CFS, as only 
the original/existing storage engine will use it (as it's specific to the 
existing memtable, commitlog, and so on)

> Refactor write path
> ---
>
> Key: CASSANDRA-14118
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14118
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Dikang Gu
>Assignee: Dikang Gu
>Priority: Major
>
> As part of the pluggable storage engine effort, we'd like to modularize the 
> write path related code, make it to be independent from existing storage 
> engine implementation details.
> For now, refer to 
> https://docs.google.com/document/d/1suZlvhzgB6NIyBNpM9nxoHxz_Ri7qAm-UEO8v8AIFsc
>  for high level designs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14118) Refactor write path

[Dikang Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401141#comment-16401141
 ] 

Dikang Gu commented on CASSANDRA-14118:
---

[~bdeggleston], Write will be very straight forward, if we keep the 
memtable/sstable in CFS, is this what you want? 
https://github.com/DikangGu/cassandra/commits/CASSANDRA-14118-v2

> Refactor write path
> ---
>
> Key: CASSANDRA-14118
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14118
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Dikang Gu
>Assignee: Dikang Gu
>Priority: Major
>
> As part of the pluggable storage engine effort, we'd like to modularize the 
> write path related code, make it to be independent from existing storage 
> engine implementation details.
> For now, refer to 
> https://docs.google.com/document/d/1suZlvhzgB6NIyBNpM9nxoHxz_Ri7qAm-UEO8v8AIFsc
>  for high level designs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14316) Read repair mutations should be sent to pending nodes

[Blake Eggleston (JIRA)]

Blake Eggleston created CASSANDRA-14316:
---

 Summary: Read repair mutations should be sent to pending nodes
 Key: CASSANDRA-14316
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14316
 Project: Cassandra
  Issue Type: Improvement
Reporter: Blake Eggleston


Since read repair doesn't mirror mutations to pending endpoints, it seems 
likely that there's an edge case that can break the monotonic quorum read 
guarantee blocking read repair is supposed to provide.

Assuming there are 3 nodes (A, B, & C) which replicate a token range. A new 
node D is added, which will take over some of A's token range. During the 
bootstrap of D, if there's a failed write that only makes it to a single node 
(A) after bootstrap has started, then there's a quorum read including A & B, 
which replicates that value to B. If A is removed when D finishes 
bootstrapping, a quorum read including node C & D will not see the value 
returned in the last quorum read which queried A & B. 

Table to illustrate:

|state | A | B | C | D|
|1 begin |  | | | pending|
|2 write |1 | | | pending|
|3 repair|1|1| | pending|
|4 joined| n/a |1| | |



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Issue Comment Deleted] (CASSANDRA-14302) Log when sstables are deleted

[Chris Lohfink (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Lohfink updated CASSANDRA-14302:
--
Comment: was deleted

(was: Id rather see it changed by having compaction logs as info personally.)

> Log when sstables are deleted
> -
>
> Key: CASSANDRA-14302
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14302
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
> Fix For: 4.0, 3.0.x, 3.11.x
>
>
> This was removed in 3.0 and is super helpful for debugging issues in prod



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14115) Refactor streaming

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401104#comment-16401104
 ] 

Jason Brown commented on CASSANDRA-14115:
-

I only had one minor nit, which I commented on the PR and Blake fixed.

+1. nice work, [~bdeggleston]

> Refactor streaming
> --
>
> Key: CASSANDRA-14115
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14115
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Streaming and Messaging
>Reporter: Dikang Gu
>Assignee: Blake Eggleston
>Priority: Major
> Fix For: 4.0
>
>
> As part of the pluggable storage engine effort, we'd like to modularize the 
> streaming related code, make it to be independent from existing storage 
> engine implementation details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14115) Refactor streaming

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown updated CASSANDRA-14115:

Status: Ready to Commit  (was: Patch Available)

> Refactor streaming
> --
>
> Key: CASSANDRA-14115
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14115
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Streaming and Messaging
>Reporter: Dikang Gu
>Assignee: Blake Eggleston
>Priority: Major
> Fix For: 4.0
>
>
> As part of the pluggable storage engine effort, we'd like to modularize the 
> streaming related code, make it to be independent from existing storage 
> engine implementation details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14302) Log when sstables are deleted

[Chris Lohfink (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401092#comment-16401092
 ] 

Chris Lohfink edited comment on CASSANDRA-14302 at 3/15/18 8:44 PM:


Id rather see it changed by having compaction logs as info personally.


was (Author: cnlwsu):
Id rather see it changed by having compaction logs as info personally. The 
process of when things are being added to disk with memtable flushes are info, 
why not when things are deleted or added through compactions.

> Log when sstables are deleted
> -
>
> Key: CASSANDRA-14302
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14302
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
> Fix For: 4.0, 3.0.x, 3.11.x
>
>
> This was removed in 3.0 and is super helpful for debugging issues in prod



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14302) Log when sstables are deleted

[Chris Lohfink (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401092#comment-16401092
 ] 

Chris Lohfink commented on CASSANDRA-14302:
---

Id rather see it changed by having compaction logs as info personally. The 
process of when things are being added to disk with memtable flushes are info, 
why not when things are deleted or added through compactions.

> Log when sstables are deleted
> -
>
> Key: CASSANDRA-14302
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14302
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
> Fix For: 4.0, 3.0.x, 3.11.x
>
>
> This was removed in 3.0 and is super helpful for debugging issues in prod



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13740) Orphan hint file gets created while node is being removed from cluster

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400994#comment-16400994
 ] 

Aleksey Yeschenko commented on CASSANDRA-13740:
---

And the changes to {{deleteAllHints()}} I don't fully understand. I don't think 
it's the responsibility of the method to close any writers. The contract is (as 
I understand it) - please remove all written hints files at this point. One 
problem is that for catalogues that aren't loaded, if some files are remaining, 
they won't be deleted - but this change doesn't address it. More importantly, I 
think that with excise fixed, that would be less of a problem and probably not 
needed to fix..

So let's leave {{deleteAllHints()}} alone. And also leave excise more or less 
alone, but call it instead at the end of {{StorageProxy.excise()}}, with a 
delay. Not sure why you picked {{RING_DELAY}} for the delay though.. can you 
explain? 

> Orphan hint file gets created while node is being removed from cluster
> --
>
> Key: CASSANDRA-13740
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13740
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, Hints
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 3.0.x, 3.11.x
>
> Attachments: 13740-3.0.15.txt, gossip_hang_test.py
>
>
> I have found this new issue during my test, whenever node is being removed 
> then hint file for that node gets written and stays inside the hint directory 
> forever. I debugged the code and found that it is due to the race condition 
> between [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
>  and [HintsWriteExecutor.java::closeWriter | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L106]
> . 
>  
> *Time t1* Node is down, as a result Hints are being written by 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
> *Time t2* Node is removed from cluster as a result it calls 
> [HintsService.java-exciseStore | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L327]
>  which removes hint files for the node being removed
> *Time t3* Mutation stage keeps pumping Hints through [HintService.java::write 
> | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L145]
>  which again calls [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215]
>  and new orphan file gets created
> I was writing a new dtest for {CASSANDRA-13562, CASSANDRA-13308} and that 
> helped me reproduce this new bug. I will submit patch for this new dtest 
> later.
> I also tried following to check how this orphan hint file responds:
> 1. I tried {{nodetool truncatehints }} but it fails as node is no 
> longer part of the ring
> 2. I then tried {{nodetool truncatehints}}, that still doesn’t remove hint 
> file because it is not yet included in the [dispatchDequeue | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsStore.java#L53]
> Reproducible steps:
> Please find dTest python file {{gossip_hang_test.py}} attached which 
> reproduces this bug.
> Solution:
> This is due to race condition as mentioned above. Since 
> {{HintsWriteExecutor.java}} creates thread pool with only 1 worker, so 
> solution becomes little simple. Whenever we [HintService.java::excise | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L303]
>  a host, just store it in-memory, and check for already evicted host inside 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215].
>  If already evicted host is found then ignore hints.
> Jaydeep



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13740) Orphan hint file gets created while node is being removed from cluster

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400985#comment-16400985
 ] 

Aleksey Yeschenko commented on CASSANDRA-13740:
---

Sure. So what the current patch is doing is it does excise, and then, in 
essence, schedules another excise in {{RING_DELAY}}?

In other words, we simply don't trust the immediate excise, and rely on the 
follow-up one. In that case, to make sure that hints for writes that will time 
out at some later point get deleted from disk, shouldn't we just say "alright, 
let's just delay the first excise, instead of doing the immediate one and then 
another one"?

> Orphan hint file gets created while node is being removed from cluster
> --
>
> Key: CASSANDRA-13740
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13740
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, Hints
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 3.0.x, 3.11.x
>
> Attachments: 13740-3.0.15.txt, gossip_hang_test.py
>
>
> I have found this new issue during my test, whenever node is being removed 
> then hint file for that node gets written and stays inside the hint directory 
> forever. I debugged the code and found that it is due to the race condition 
> between [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
>  and [HintsWriteExecutor.java::closeWriter | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L106]
> . 
>  
> *Time t1* Node is down, as a result Hints are being written by 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L195]
> *Time t2* Node is removed from cluster as a result it calls 
> [HintsService.java-exciseStore | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L327]
>  which removes hint files for the node being removed
> *Time t3* Mutation stage keeps pumping Hints through [HintService.java::write 
> | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L145]
>  which again calls [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215]
>  and new orphan file gets created
> I was writing a new dtest for {CASSANDRA-13562, CASSANDRA-13308} and that 
> helped me reproduce this new bug. I will submit patch for this new dtest 
> later.
> I also tried following to check how this orphan hint file responds:
> 1. I tried {{nodetool truncatehints }} but it fails as node is no 
> longer part of the ring
> 2. I then tried {{nodetool truncatehints}}, that still doesn’t remove hint 
> file because it is not yet included in the [dispatchDequeue | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsStore.java#L53]
> Reproducible steps:
> Please find dTest python file {{gossip_hang_test.py}} attached which 
> reproduces this bug.
> Solution:
> This is due to race condition as mentioned above. Since 
> {{HintsWriteExecutor.java}} creates thread pool with only 1 worker, so 
> solution becomes little simple. Whenever we [HintService.java::excise | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L303]
>  a host, just store it in-memory, and check for already evicted host inside 
> [HintsWriteExecutor.java::flush | 
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsWriteExecutor.java#L215].
>  If already evicted host is found then ignore hints.
> Jaydeep



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14314) Fix argument passing for SSLContext in trunk

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400953#comment-16400953
 ] 

Jason Brown commented on CASSANDRA-14314:
-

[~djoshi3] and I spoke offline about this, and we uncovered some futher 
incorrectness with {{SSLFactory.getSslContext()}}. We'll have an updated patch 
soon.

> Fix argument passing for SSLContext in trunk
> 
>
> Key: CASSANDRA-14314
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14314
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
>
> Argument passing has a minor bug while creating the SSLContext. Audit and 
> make sure that the client & server SSL contexts are created at appropriate 
> locations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14302) Log when sstables are deleted

[Blake Eggleston (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400847#comment-16400847
 ] 

Blake Eggleston commented on CASSANDRA-14302:
-

ninja'd to debug as {{1d05bdab2329a8d30772e079ccd9e64ed80438eb}}

> Log when sstables are deleted
> -
>
> Key: CASSANDRA-14302
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14302
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
> Fix For: 4.0, 3.0.x, 3.11.x
>
>
> This was removed in 3.0 and is super helpful for debugging issues in prod



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[bdeggleston]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/253c003d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/253c003d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/253c003d

Branch: refs/heads/trunk
Commit: 253c003df7bb520225cce0cee5d6fbec3808f29f
Parents: 49c1a25 f57d12e
Author: Blake Eggleston 
Authored: Thu Mar 15 10:57:03 2018 -0700
Committer: Blake Eggleston 
Committed: Thu Mar 15 10:57:03 2018 -0700

--
 src/java/org/apache/cassandra/io/sstable/SSTable.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/253c003d/src/java/org/apache/cassandra/io/sstable/SSTable.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[3/6] cassandra git commit: ninja: change CASSANDRA-14118 log statement to DEBUG

[bdeggleston]

ninja: change CASSANDRA-14118 log statement to DEBUG


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1d05bdab
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1d05bdab
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1d05bdab

Branch: refs/heads/trunk
Commit: 1d05bdab2329a8d30772e079ccd9e64ed80438eb
Parents: fdc61cb
Author: Blake Eggleston 
Authored: Thu Mar 15 10:55:29 2018 -0700
Committer: Blake Eggleston 
Committed: Thu Mar 15 10:55:29 2018 -0700

--
 src/java/org/apache/cassandra/io/sstable/SSTable.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1d05bdab/src/java/org/apache/cassandra/io/sstable/SSTable.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTable.java 
b/src/java/org/apache/cassandra/io/sstable/SSTable.java
index a54e7e0..b5703fc 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTable.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTable.java
@@ -104,7 +104,7 @@ public abstract class SSTable
  */
 public static boolean delete(Descriptor desc, Set components)
 {
-logger.info("Deleting sstable: {}", desc);
+logger.debug("Deleting sstable: {}", desc);
 // remove the DATA component first if it exists
 if (components.contains(Component.DATA))
 FileUtils.deleteWithConfirm(desc.filenameFor(Component.DATA));


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[2/6] cassandra git commit: ninja: change CASSANDRA-14118 log statement to DEBUG

[bdeggleston]

ninja: change CASSANDRA-14118 log statement to DEBUG


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1d05bdab
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1d05bdab
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1d05bdab

Branch: refs/heads/cassandra-3.11
Commit: 1d05bdab2329a8d30772e079ccd9e64ed80438eb
Parents: fdc61cb
Author: Blake Eggleston 
Authored: Thu Mar 15 10:55:29 2018 -0700
Committer: Blake Eggleston 
Committed: Thu Mar 15 10:55:29 2018 -0700

--
 src/java/org/apache/cassandra/io/sstable/SSTable.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1d05bdab/src/java/org/apache/cassandra/io/sstable/SSTable.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTable.java 
b/src/java/org/apache/cassandra/io/sstable/SSTable.java
index a54e7e0..b5703fc 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTable.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTable.java
@@ -104,7 +104,7 @@ public abstract class SSTable
  */
 public static boolean delete(Descriptor desc, Set components)
 {
-logger.info("Deleting sstable: {}", desc);
+logger.debug("Deleting sstable: {}", desc);
 // remove the DATA component first if it exists
 if (components.contains(Component.DATA))
 FileUtils.deleteWithConfirm(desc.filenameFor(Component.DATA));


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[1/6] cassandra git commit: ninja: change CASSANDRA-14118 log statement to DEBUG

[bdeggleston]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 fdc61cb6a -> 1d05bdab2
  refs/heads/cassandra-3.11 19a547861 -> f57d12ee7
  refs/heads/trunk 49c1a256c -> 253c003df


ninja: change CASSANDRA-14118 log statement to DEBUG


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1d05bdab
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1d05bdab
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1d05bdab

Branch: refs/heads/cassandra-3.0
Commit: 1d05bdab2329a8d30772e079ccd9e64ed80438eb
Parents: fdc61cb
Author: Blake Eggleston 
Authored: Thu Mar 15 10:55:29 2018 -0700
Committer: Blake Eggleston 
Committed: Thu Mar 15 10:55:29 2018 -0700

--
 src/java/org/apache/cassandra/io/sstable/SSTable.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1d05bdab/src/java/org/apache/cassandra/io/sstable/SSTable.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTable.java 
b/src/java/org/apache/cassandra/io/sstable/SSTable.java
index a54e7e0..b5703fc 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTable.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTable.java
@@ -104,7 +104,7 @@ public abstract class SSTable
  */
 public static boolean delete(Descriptor desc, Set components)
 {
-logger.info("Deleting sstable: {}", desc);
+logger.debug("Deleting sstable: {}", desc);
 // remove the DATA component first if it exists
 if (components.contains(Component.DATA))
 FileUtils.deleteWithConfirm(desc.filenameFor(Component.DATA));


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[bdeggleston]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f57d12ee
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f57d12ee
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f57d12ee

Branch: refs/heads/trunk
Commit: f57d12ee7af1bf3b93d36b486749defd2204d066
Parents: 19a5478 1d05bda
Author: Blake Eggleston 
Authored: Thu Mar 15 10:55:46 2018 -0700
Committer: Blake Eggleston 
Committed: Thu Mar 15 10:55:46 2018 -0700

--
 src/java/org/apache/cassandra/io/sstable/SSTable.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f57d12ee/src/java/org/apache/cassandra/io/sstable/SSTable.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[bdeggleston]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f57d12ee
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f57d12ee
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f57d12ee

Branch: refs/heads/cassandra-3.11
Commit: f57d12ee7af1bf3b93d36b486749defd2204d066
Parents: 19a5478 1d05bda
Author: Blake Eggleston 
Authored: Thu Mar 15 10:55:46 2018 -0700
Committer: Blake Eggleston 
Committed: Thu Mar 15 10:55:46 2018 -0700

--
 src/java/org/apache/cassandra/io/sstable/SSTable.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f57d12ee/src/java/org/apache/cassandra/io/sstable/SSTable.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14118) Refactor write path

[Blake Eggleston (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400659#comment-16400659
 ] 

Blake Eggleston edited comment on CASSANDRA-14118 at 3/15/18 5:48 PM:
--

I figure I should explain what I meant by my last comment without making you 
wait for the review of CASSANDRA-14117. In the case of this ticket, I'd avoid 
hollowing out ColumnFamilyStore. I spoke with [~jasobrown] about this in person 
earlier this week, and the general structure we were thinking of would keep the 
core storage components that are used by multiple systems and operations in CFS 
(so sstables, View, etc). The CFS would also instantiate and return 
implementation specific "Handler" classes. StreamHandler, ReadHandler, 
WriteHandler, etc. Those classes would implement the logic specific to those 
operations by interacting with the storage components contained in CFS. Once 
the other tickets are completed, if we've done a good enough job of abstracting 
things, we should be able to move CFS to  CassandraColumnFamilyStore or 
something, and replace it with an interface that just provides those handler 
classes.

So in the case of this ticket, you probably just need to create a write handler 
class that implements {{apply}}, and do the same for {{Keyspace}}. WDYT?


was (Author: bdeggleston):
I figure I should explain what I meant by my last comment without making you 
wait for the review of CASSANDRA-14117. In the case of this ticket, I'd avoid 
hollowing out ColumnFamilyStore. I spoke with [~jasobrown] about this in person 
earlier this week, and the general structure we were thinking of would keep the 
core storage components that are used by multiple systems and operations in CFS 
(so sstables, View, etc). The CFS would also instantiate and return 
implementation specific "Handler" classes. StreamHandler, ReadHandler, 
WriteHandler, etc. Those classes would implement the logic specific to those 
operations by interacting with the storage components contained in CFS. Once 
the other tickets are completed, if we've done a good enough job of abstracting 
things, we should be able to move CFS to  CassandraColumnFamilyStore or 
something, and be replace it with an interface that just provides those handler 
classes.

So in the case of this ticket, you probably just need to create a write handler 
class that implements {{apply}}, and do the same for {{Keyspace}}. WDYT?

> Refactor write path
> ---
>
> Key: CASSANDRA-14118
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14118
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Dikang Gu
>Assignee: Dikang Gu
>Priority: Major
>
> As part of the pluggable storage engine effort, we'd like to modularize the 
> write path related code, make it to be independent from existing storage 
> engine implementation details.
> For now, refer to 
> https://docs.google.com/document/d/1suZlvhzgB6NIyBNpM9nxoHxz_Ri7qAm-UEO8v8AIFsc
>  for high level designs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14302) Log when sstables are deleted

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400707#comment-16400707
 ] 

Marcus Eriksson commented on CASSANDRA-14302:
-

compaction logs are debug in 3.0+ so this needs to be debug everywhere I think

> Log when sstables are deleted
> -
>
> Key: CASSANDRA-14302
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14302
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
> Fix For: 4.0, 3.0.x, 3.11.x
>
>
> This was removed in 3.0 and is super helpful for debugging issues in prod



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14302) Log when sstables are deleted

[Blake Eggleston (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400680#comment-16400680
 ] 

Blake Eggleston commented on CASSANDRA-14302:
-

[~krummas] do you mean in trunk, or 3.0 and 3.11 as well?

> Log when sstables are deleted
> -
>
> Key: CASSANDRA-14302
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14302
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
> Fix For: 4.0, 3.0.x, 3.11.x
>
>
> This was removed in 3.0 and is super helpful for debugging issues in prod



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14118) Refactor write path

[Blake Eggleston (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400659#comment-16400659
 ] 

Blake Eggleston edited comment on CASSANDRA-14118 at 3/15/18 4:26 PM:
--

I figure I should explain what I meant by my last comment without making you 
wait for the review of CASSANDRA-14117. In the case of this ticket, I'd avoid 
hollowing out ColumnFamilyStore. I spoke with [~jasobrown] about this in person 
earlier this week, and the general structure we were thinking of would keep the 
core storage components that are used by multiple systems and operations in CFS 
(so sstables, View, etc). The CFS would also instantiate and return 
implementation specific "Handler" classes. StreamHandler, ReadHandler, 
WriteHandler, etc. Those classes would implement the logic specific to those 
operations by interacting with the storage components contained in CFS. Once 
the other tickets are completed, if we've done a good enough job of abstracting 
things, we should be able to move CFS to  CassandraColumnFamilyStore or 
something, and be replace it with an interface that just provides those handler 
classes.

So in the case of this ticket, you probably just need to create a write handler 
class that implements {{apply}}, and do the same for {{Keyspace}}. WDYT?


was (Author: bdeggleston):
I figure I should explain what I meant by my last comment without making you 
wait for the review of CASSANDRA-14117. In the case of this ticket, I'd avoid 
hollowing out ColumnFamilyStore. I spoke with [~jasobrown] about this in person 
earlier this week, and the general structure we thinking of would keep the core 
storage components that are used by multiple systems and operations in CFS. The 
CFS would also instantiate and return implementation specific "Handler" 
classes. StreamHandler, ReadHandler, WriteHandler, etc. Those classes would 
implement the logic specific to those operations by interacting with the 
storage components contained in CFS. Once the other tickets are completed, if 
we've done a good enough job of abstracting things, we should be able to move 
CFS to 
 CassandraColumnFamilyStore or something, and be replace it with an interface 
that just provides those handler classes.

So in the case of this ticket, you probably just need to create a write handler 
class that implements {{apply}}, and do the same for {{Keyspace}}. WDYT?

> Refactor write path
> ---
>
> Key: CASSANDRA-14118
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14118
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Dikang Gu
>Assignee: Dikang Gu
>Priority: Major
>
> As part of the pluggable storage engine effort, we'd like to modularize the 
> write path related code, make it to be independent from existing storage 
> engine implementation details.
> For now, refer to 
> https://docs.google.com/document/d/1suZlvhzgB6NIyBNpM9nxoHxz_Ri7qAm-UEO8v8AIFsc
>  for high level designs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14118) Refactor write path

[Blake Eggleston (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400659#comment-16400659
 ] 

Blake Eggleston commented on CASSANDRA-14118:
-

I figure I should explain what I meant by my last comment without making you 
wait for the review of CASSANDRA-14117. In the case of this ticket, I'd avoid 
hollowing out ColumnFamilyStore. I spoke with [~jasobrown] about this in person 
earlier this week, and the general structure we thinking of would keep the core 
storage components that are used by multiple systems and operations in CFS. The 
CFS would also instantiate and return implementation specific "Handler" 
classes. StreamHandler, ReadHandler, WriteHandler, etc. Those classes would 
implement the logic specific to those operations by interacting with the 
storage components contained in CFS. Once the other tickets are completed, if 
we've done a good enough job of abstracting things, we should be able to move 
CFS to 
 CassandraColumnFamilyStore or something, and be replace it with an interface 
that just provides those handler classes.

So in the case of this ticket, you probably just need to create a write handler 
class that implements {{apply}}, and do the same for {{Keyspace}}. WDYT?

> Refactor write path
> ---
>
> Key: CASSANDRA-14118
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14118
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Dikang Gu
>Assignee: Dikang Gu
>Priority: Major
>
> As part of the pluggable storage engine effort, we'd like to modularize the 
> write path related code, make it to be independent from existing storage 
> engine implementation details.
> For now, refer to 
> https://docs.google.com/document/d/1suZlvhzgB6NIyBNpM9nxoHxz_Ri7qAm-UEO8v8AIFsc
>  for high level designs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14311) Allow Token-Aware drivers for range scans

[Avi Kivity (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400621#comment-16400621
 ] 

Avi Kivity commented on CASSANDRA-14311:


Thanks JIRA for converting my code into emojis. Is nowhere safe?

> Allow Token-Aware drivers for range scans
> -
>
> Key: CASSANDRA-14311
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14311
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Coordination
>Reporter: Avi Kivity
>Priority: Major
>
> Currently, range scans are not token aware. This means that an extra hop is 
> needed for most requests. Since range scans are usually data intensive, this 
> causes significant extra traffic.
>  
> Token awareness could be enabled by having the coordinator return the token 
> for the next (still unread) row in the response, so the driver can select a 
> next coordinator that owns this row.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14311) Allow Token-Aware drivers for range scans

[Avi Kivity (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400619#comment-16400619
 ] 

Avi Kivity commented on CASSANDRA-14311:


{quote}We would need to add some hints here about the token ranges covered by 
the query for the driver to use.
{quote}
There's no need for hints. You send the first page (which will likely miss the 
replica), and in addition to has_more_pages you also get a token for the next 
page.

 

We could also optimize the first page by providing more metadata. There are a 
few cases to consider:

 
 # SELECT ... FROM ... WHERE token(pk) >= ?
 # SELECT ... FROM ... WHERE token(pk) >= token(?)
 # SELECT ... FROM ... WHERE (no lower-bound specified)

(1 and 2 also need to support >).

 

If the metadata describes these cases, then we can send the first page query to 
a coordinator that is also a replica.

> Allow Token-Aware drivers for range scans
> -
>
> Key: CASSANDRA-14311
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14311
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Coordination
>Reporter: Avi Kivity
>Priority: Major
>
> Currently, range scans are not token aware. This means that an extra hop is 
> needed for most requests. Since range scans are usually data intensive, this 
> causes significant extra traffic.
>  
> Token awareness could be enabled by having the coordinator return the token 
> for the next (still unread) row in the response, so the driver can select a 
> next coordinator that owns this row.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14311) Allow Token-Aware drivers for range scans

[Avi Kivity (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400592#comment-16400592
 ] 

Avi Kivity commented on CASSANDRA-14311:


{quote}this allows querying multiple token ranges in parallel and gives even 
more benefits than just node hopping for the “next” page
{quote}
 

That changes the query semantics, not all users are prepared for parallel scan 
in a single thread. For sure, it is viable.

 

My proposal optimizes existing use case where you have a sequential query and 
want to keep it sequential.

> Allow Token-Aware drivers for range scans
> -
>
> Key: CASSANDRA-14311
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14311
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Coordination
>Reporter: Avi Kivity
>Priority: Major
>
> Currently, range scans are not token aware. This means that an extra hop is 
> needed for most requests. Since range scans are usually data intensive, this 
> causes significant extra traffic.
>  
> Token awareness could be enabled by having the coordinator return the token 
> for the next (still unread) row in the response, so the driver can select a 
> next coordinator that owns this row.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14311) Allow Token-Aware drivers for range scans

[Chris Lohfink (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400588#comment-16400588
 ] 

Chris Lohfink commented on CASSANDRA-14311:
---

> You could do this right now if you just add a new higher level querying 
> interface in a given driver.

That was my first reaction, adding some kinda table scan/bulk reader methods to 
the driver maybe worth a shot instead of being replicated in hadoop, spark and 
misc tools that do table scans.

> Allow Token-Aware drivers for range scans
> -
>
> Key: CASSANDRA-14311
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14311
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Coordination
>Reporter: Avi Kivity
>Priority: Major
>
> Currently, range scans are not token aware. This means that an extra hop is 
> needed for most requests. Since range scans are usually data intensive, this 
> causes significant extra traffic.
>  
> Token awareness could be enabled by having the coordinator return the token 
> for the next (still unread) row in the response, so the driver can select a 
> next coordinator that owns this row.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14311) Allow Token-Aware drivers for range scans

[Jeremiah Jordan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400571#comment-16400571
 ] 

Jeremiah Jordan commented on CASSANDRA-14311:
-

We would need to add some hints here about the token ranges covered by the 
query for the driver to use.
Another method which is used by the Spark connector and the Hadoop connector 
before it is for the client side to break up the query by token ranges at the 
start, this allows querying multiple token ranges in parallel and gives even 
more benefits than just node hopping for the “next” page. You could do this 
right now if you just add a new higher level querying interface in a given 
driver.

> Allow Token-Aware drivers for range scans
> -
>
> Key: CASSANDRA-14311
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14311
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Coordination
>Reporter: Avi Kivity
>Priority: Major
>
> Currently, range scans are not token aware. This means that an extra hop is 
> needed for most requests. Since range scans are usually data intensive, this 
> causes significant extra traffic.
>  
> Token awareness could be enabled by having the coordinator return the token 
> for the next (still unread) row in the response, so the driver can select a 
> next coordinator that owns this row.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Issue Comment Deleted] (CASSANDRA-14311) Allow Token-Aware drivers for range scans

[Chris Lohfink (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Lohfink updated CASSANDRA-14311:
--
Comment: was deleted

(was: The driver doesnt actually know if the request is a range scan and doesnt 
have a parser. Not quite sure what C* can do here. Maybe a different request 
interface in driver for table scans? I think this would be a better ticket for 
the driver project as I think people have looked into this before.)

> Allow Token-Aware drivers for range scans
> -
>
> Key: CASSANDRA-14311
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14311
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Coordination
>Reporter: Avi Kivity
>Priority: Major
>
> Currently, range scans are not token aware. This means that an extra hop is 
> needed for most requests. Since range scans are usually data intensive, this 
> causes significant extra traffic.
>  
> Token awareness could be enabled by having the coordinator return the token 
> for the next (still unread) row in the response, so the driver can select a 
> next coordinator that owns this row.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14311) Allow Token-Aware drivers for range scans

[Chris Lohfink (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400560#comment-16400560
 ] 

Chris Lohfink commented on CASSANDRA-14311:
---

The driver doesnt actually know if the request is a range scan and doesnt have 
a parser. Not quite sure what C* can do here. Maybe a different request 
interface in driver for table scans? I think this would be a better ticket for 
the driver project as I think people have looked into this before.

> Allow Token-Aware drivers for range scans
> -
>
> Key: CASSANDRA-14311
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14311
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Coordination
>Reporter: Avi Kivity
>Priority: Major
>
> Currently, range scans are not token aware. This means that an extra hop is 
> needed for most requests. Since range scans are usually data intensive, this 
> causes significant extra traffic.
>  
> Token awareness could be enabled by having the coordinator return the token 
> for the next (still unread) row in the response, so the driver can select a 
> next coordinator that owns this row.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14215) Cassandra does not respect hint window for CAS

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-14215:
--
   Resolution: Fixed
 Reviewer: Aleksey Yeschenko
Fix Version/s: 3.11.3
   3.0.17
   4.0
Reproduced In: 3.11.2, 3.0.16  (was: 3.0.16, 3.11.2)
   Status: Resolved  (was: Ready to Commit)

> Cassandra does not respect hint window for CAS
> --
>
> Key: CASSANDRA-14215
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14215
> Project: Cassandra
>  Issue Type: Bug
>  Components: Hints, Streaming and Messaging
>Reporter: Arijit Banerjee
>Assignee: Kurt Greaves
>Priority: Major
> Fix For: 4.0, 3.0.17, 3.11.3
>
>
> On Cassandra 3.0.9, it was observed that Cassandra continues to write hints 
> even though a node remains down (and does not come up) for longer than the 
> default 3 hour window.
>  
> After doing "nodetool setlogginglevel org.apache.cassandra TRACE", we see the 
> following log line in cassandra (debug) logs:
>  StorageProxy.java:2625 - Adding hints for [/10.0.100.84]
>  
> One possible code path seems to be:
> cas -> commitPaxos(proposal, consistencyForCommit, true); -> submitHint (in 
> StorageProxy.java)
>  
> The "true" parameter above explicitly states that a hint should be recorded 
> and ignores the time window calculation performed by the shouldHint method 
> invoked in other code paths. Is there a reason for this behavior?
>  
> Edit: There are actually two stacks that seem to be producing hints, the 
> "cas" and "syncWriteBatchedMutations" methods. I have posted them below.
>  
> A third issue seems to be that Cassandra seems to reset the timer which 
> counts how long a node has been down after a restart. Thus if Cassandra is 
> restarted on a good node, it continues to accumulate hints for a down node 
> over the next three hours.
>  
> {code:java}
> WARN [SharedPool-Worker-14] 2018-02-06 22:15:51,136 StorageProxy.java:2636 - 
> Adding hints for [/10.0.100.84] with stack trace: java.lang.Throwable: at 
> org.apache.cassandra.service.StorageProxy.stackTrace(StorageProxy.java:2608) 
> at 
> org.apache.cassandra.service.StorageProxy.submitHint(StorageProxy.java:2617) 
> at 
> org.apache.cassandra.service.StorageProxy.submitHint(StorageProxy.java:2603) 
> at 
> org.apache.cassandra.service.StorageProxy.commitPaxos(StorageProxy.java:540) 
> at org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:282) at 
> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:432)
>  at 
> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:407)
>  at 
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
>  at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:237) 
> at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222) 
> at 
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115)
>  at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513)
>  at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407)
>  at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
>  at 
> io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> at 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) at 
> java.lang.Thread.run(Thread.java:748) WARN
> {code}
> {code:java}
> [SharedPool-Worker-8] 2018-02-06 22:15:51,153 StorageProxy.java:2636 - Adding 
> hints for [/10.0.100.84] with stack trace: java.lang.Throwable: at 
> org.apache.cassandra.service.StorageProxy.stackTrace(StorageProxy.java:2608) 
> at 
> org.apache.cassandra.service.StorageProxy.submitHint(StorageProxy.java:2617) 
> at 
> org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:1247)
>  at 
> org.apache.cassandra.service.StorageProxy.syncWriteBatchedMutations(StorageProxy.java:1014)
>  at 
> org.apache.cassandra.service.StorageProxy.mutateAtomically(StorageProxy.java:899)
>  at 
> org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:834)
>  at 
> 

[jira] [Commented] (CASSANDRA-14215) Cassandra does not respect hint window for CAS

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400466#comment-16400466
 ] 

Aleksey Yeschenko commented on CASSANDRA-14215:
---

Committed to 3.0 as 
[fdc61cb6a74163402181d133519462f0ab1b504d|https://github.com/apache/cassandra/commit/fdc61cb6a74163402181d133519462f0ab1b504d]
 and merged into 3.11 and trunk. Thanks.

> Cassandra does not respect hint window for CAS
> --
>
> Key: CASSANDRA-14215
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14215
> Project: Cassandra
>  Issue Type: Bug
>  Components: Hints, Streaming and Messaging
>Reporter: Arijit Banerjee
>Assignee: Kurt Greaves
>Priority: Major
>
> On Cassandra 3.0.9, it was observed that Cassandra continues to write hints 
> even though a node remains down (and does not come up) for longer than the 
> default 3 hour window.
>  
> After doing "nodetool setlogginglevel org.apache.cassandra TRACE", we see the 
> following log line in cassandra (debug) logs:
>  StorageProxy.java:2625 - Adding hints for [/10.0.100.84]
>  
> One possible code path seems to be:
> cas -> commitPaxos(proposal, consistencyForCommit, true); -> submitHint (in 
> StorageProxy.java)
>  
> The "true" parameter above explicitly states that a hint should be recorded 
> and ignores the time window calculation performed by the shouldHint method 
> invoked in other code paths. Is there a reason for this behavior?
>  
> Edit: There are actually two stacks that seem to be producing hints, the 
> "cas" and "syncWriteBatchedMutations" methods. I have posted them below.
>  
> A third issue seems to be that Cassandra seems to reset the timer which 
> counts how long a node has been down after a restart. Thus if Cassandra is 
> restarted on a good node, it continues to accumulate hints for a down node 
> over the next three hours.
>  
> {code:java}
> WARN [SharedPool-Worker-14] 2018-02-06 22:15:51,136 StorageProxy.java:2636 - 
> Adding hints for [/10.0.100.84] with stack trace: java.lang.Throwable: at 
> org.apache.cassandra.service.StorageProxy.stackTrace(StorageProxy.java:2608) 
> at 
> org.apache.cassandra.service.StorageProxy.submitHint(StorageProxy.java:2617) 
> at 
> org.apache.cassandra.service.StorageProxy.submitHint(StorageProxy.java:2603) 
> at 
> org.apache.cassandra.service.StorageProxy.commitPaxos(StorageProxy.java:540) 
> at org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:282) at 
> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:432)
>  at 
> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:407)
>  at 
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
>  at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:237) 
> at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222) 
> at 
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115)
>  at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513)
>  at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407)
>  at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
>  at 
> io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> at 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) at 
> java.lang.Thread.run(Thread.java:748) WARN
> {code}
> {code:java}
> [SharedPool-Worker-8] 2018-02-06 22:15:51,153 StorageProxy.java:2636 - Adding 
> hints for [/10.0.100.84] with stack trace: java.lang.Throwable: at 
> org.apache.cassandra.service.StorageProxy.stackTrace(StorageProxy.java:2608) 
> at 
> org.apache.cassandra.service.StorageProxy.submitHint(StorageProxy.java:2617) 
> at 
> org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:1247)
>  at 
> org.apache.cassandra.service.StorageProxy.syncWriteBatchedMutations(StorageProxy.java:1014)
>  at 
> org.apache.cassandra.service.StorageProxy.mutateAtomically(StorageProxy.java:899)
>  at 
> org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:834)
>  at 
> org.apache.cassandra.cql3.statements.BatchStatement.executeWithoutConditions(BatchStatement.java:365)
>  at 
> 

[1/6] cassandra git commit: Respect max hint window when hinting for LWT

[aleksey]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 3d7471a56 -> fdc61cb6a
  refs/heads/cassandra-3.11 4c6a10af8 -> 19a547861
  refs/heads/trunk 92cde9b27 -> 49c1a256c


Respect max hint window when hinting for LWT

patch by Kurt Greaves; reviewed by Aleksey Yeschenko for CASSANDRA-14215


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fdc61cb6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fdc61cb6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fdc61cb6

Branch: refs/heads/cassandra-3.0
Commit: fdc61cb6a74163402181d133519462f0ab1b504d
Parents: 3d7471a
Author: kurt 
Authored: Tue Feb 20 00:59:00 2018 +
Committer: Aleksey Yeshchenko 
Committed: Thu Mar 15 17:13:56 2018 +0300

--
 CHANGES.txt | 8 +---
 src/java/org/apache/cassandra/service/StorageProxy.java | 6 +++---
 2 files changed, 8 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/fdc61cb6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 62b904f..6099b01 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,6 @@
 3.0.17
- * Adding missing WriteType enum values to v3,v4 and v5 spec (CASSANDRA-13697)
+ * Respect max hint window when hinting for LWT (CASSANDRA-14215)
+ * Adding missing WriteType enum values to v3, v4, and v5 spec 
(CASSANDRA-13697)
  * Don't regenerate bloomfilter and summaries on startup (CASSANDRA-11163)
  * Fix NPE when performing comparison against a null frozen in LWT 
(CASSANDRA-14087)
  * Log when SSTables are deleted (CASSANDRA-14302)
@@ -8,10 +9,11 @@
  * Chain commit log marker potential performance regression in batch commit 
mode (CASSANDRA-14194)
  * Fully utilise specified compaction threads (CASSANDRA-14210)
  * Pre-create deletion log records to finish compactions quicker 
(CASSANDRA-12763)
-Merged from 2.1:
- * CVE-2017-5929 Security vulnerability in Logback warning in NEWS.txt 
(CASSANDRA-14183)
 Merged from 2.2:
  * Backport circleci yaml (CASSANDRA-14240)
+Merged from 2.1:
+ * CVE-2017-5929 Security vulnerability in Logback warning in NEWS.txt 
(CASSANDRA-14183)
+
 
 3.0.16
  * Fix unit test failures in ViewComplexTest (CASSANDRA-14219)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/fdc61cb6/src/java/org/apache/cassandra/service/StorageProxy.java
--
diff --git a/src/java/org/apache/cassandra/service/StorageProxy.java 
b/src/java/org/apache/cassandra/service/StorageProxy.java
index 7a6bed4..bf58ef2 100644
--- a/src/java/org/apache/cassandra/service/StorageProxy.java
+++ b/src/java/org/apache/cassandra/service/StorageProxy.java
@@ -497,7 +497,7 @@ public class StorageProxy implements StorageProxyMBean
 return false;
 }
 
-private static void commitPaxos(Commit proposal, ConsistencyLevel 
consistencyLevel, boolean shouldHint) throws WriteTimeoutException
+private static void commitPaxos(Commit proposal, ConsistencyLevel 
consistencyLevel, boolean allowHints) throws WriteTimeoutException
 {
 boolean shouldBlock = consistencyLevel != ConsistencyLevel.ANY;
 Keyspace keyspace = Keyspace.open(proposal.update.metadata().ksName);
@@ -523,14 +523,14 @@ public class StorageProxy implements StorageProxyMBean
 if (canDoLocalRequest(destination))
 commitPaxosLocal(message, responseHandler);
 else
-MessagingService.instance().sendRR(message, 
destination, responseHandler, shouldHint);
+MessagingService.instance().sendRR(message, 
destination, responseHandler, allowHints && shouldHint(destination));
 }
 else
 {
 MessagingService.instance().sendOneWay(message, 
destination);
 }
 }
-else if (shouldHint)
+else if (allowHints && shouldHint(destination))
 {
 submitHint(proposal.makeMutation(), destination, null);
 }


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[aleksey]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/19a54786
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/19a54786
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/19a54786

Branch: refs/heads/trunk
Commit: 19a547861ad6c68e4ce584ee1b9ffee66bd5042d
Parents: 4c6a10a fdc61cb
Author: Aleksey Yeshchenko 
Authored: Thu Mar 15 17:17:23 2018 +0300
Committer: Aleksey Yeshchenko 
Committed: Thu Mar 15 17:17:23 2018 +0300

--
 CHANGES.txt | 3 ++-
 src/java/org/apache/cassandra/service/StorageProxy.java | 6 +++---
 2 files changed, 5 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/19a54786/CHANGES.txt
--
diff --cc CHANGES.txt
index fcd86ed,6099b01..bae967f
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,10 -1,6 +1,11 @@@
 -3.0.17
 +3.11.3
 + * SASI tokenizer for simple delimiter based entries (CASSANDRA-14247)
 + * Fix Loss of digits when doing CAST from varint/bigint to decimal 
(CASSANDRA-14170)
 + * RateBasedBackPressure unnecessarily invokes a lock on the Guava 
RateLimiter (CASSANDRA-14163)
 + * Fix wildcard GROUP BY queries (CASSANDRA-14209)
 +Merged from 3.0:
-  * Adding missing WriteType enum values to v3,v4 and v5 spec (CASSANDRA-13697)
+  * Respect max hint window when hinting for LWT (CASSANDRA-14215)
+  * Adding missing WriteType enum values to v3, v4, and v5 spec 
(CASSANDRA-13697)
   * Don't regenerate bloomfilter and summaries on startup (CASSANDRA-11163)
   * Fix NPE when performing comparison against a null frozen in LWT 
(CASSANDRA-14087)
   * Log when SSTables are deleted (CASSANDRA-14302)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/19a54786/src/java/org/apache/cassandra/service/StorageProxy.java
--
diff --cc src/java/org/apache/cassandra/service/StorageProxy.java
index 4dc05e3,bf58ef2..52dd8ab
--- a/src/java/org/apache/cassandra/service/StorageProxy.java
+++ b/src/java/org/apache/cassandra/service/StorageProxy.java
@@@ -522,7 -497,7 +522,7 @@@ public class StorageProxy implements St
  return false;
  }
  
- private static void commitPaxos(Commit proposal, ConsistencyLevel 
consistencyLevel, boolean shouldHint, long queryStartNanoTime) throws 
WriteTimeoutException
 -private static void commitPaxos(Commit proposal, ConsistencyLevel 
consistencyLevel, boolean allowHints) throws WriteTimeoutException
++private static void commitPaxos(Commit proposal, ConsistencyLevel 
consistencyLevel, boolean allowHints, long queryStartNanoTime) throws 
WriteTimeoutException
  {
  boolean shouldBlock = consistencyLevel != ConsistencyLevel.ANY;
  Keyspace keyspace = Keyspace.open(proposal.update.metadata().ksName);


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[3/6] cassandra git commit: Respect max hint window when hinting for LWT

[aleksey]

Respect max hint window when hinting for LWT

patch by Kurt Greaves; reviewed by Aleksey Yeschenko for CASSANDRA-14215


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fdc61cb6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fdc61cb6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fdc61cb6

Branch: refs/heads/trunk
Commit: fdc61cb6a74163402181d133519462f0ab1b504d
Parents: 3d7471a
Author: kurt 
Authored: Tue Feb 20 00:59:00 2018 +
Committer: Aleksey Yeshchenko 
Committed: Thu Mar 15 17:13:56 2018 +0300

--
 CHANGES.txt | 8 +---
 src/java/org/apache/cassandra/service/StorageProxy.java | 6 +++---
 2 files changed, 8 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/fdc61cb6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 62b904f..6099b01 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,6 @@
 3.0.17
- * Adding missing WriteType enum values to v3,v4 and v5 spec (CASSANDRA-13697)
+ * Respect max hint window when hinting for LWT (CASSANDRA-14215)
+ * Adding missing WriteType enum values to v3, v4, and v5 spec 
(CASSANDRA-13697)
  * Don't regenerate bloomfilter and summaries on startup (CASSANDRA-11163)
  * Fix NPE when performing comparison against a null frozen in LWT 
(CASSANDRA-14087)
  * Log when SSTables are deleted (CASSANDRA-14302)
@@ -8,10 +9,11 @@
  * Chain commit log marker potential performance regression in batch commit 
mode (CASSANDRA-14194)
  * Fully utilise specified compaction threads (CASSANDRA-14210)
  * Pre-create deletion log records to finish compactions quicker 
(CASSANDRA-12763)
-Merged from 2.1:
- * CVE-2017-5929 Security vulnerability in Logback warning in NEWS.txt 
(CASSANDRA-14183)
 Merged from 2.2:
  * Backport circleci yaml (CASSANDRA-14240)
+Merged from 2.1:
+ * CVE-2017-5929 Security vulnerability in Logback warning in NEWS.txt 
(CASSANDRA-14183)
+
 
 3.0.16
  * Fix unit test failures in ViewComplexTest (CASSANDRA-14219)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/fdc61cb6/src/java/org/apache/cassandra/service/StorageProxy.java
--
diff --git a/src/java/org/apache/cassandra/service/StorageProxy.java 
b/src/java/org/apache/cassandra/service/StorageProxy.java
index 7a6bed4..bf58ef2 100644
--- a/src/java/org/apache/cassandra/service/StorageProxy.java
+++ b/src/java/org/apache/cassandra/service/StorageProxy.java
@@ -497,7 +497,7 @@ public class StorageProxy implements StorageProxyMBean
 return false;
 }
 
-private static void commitPaxos(Commit proposal, ConsistencyLevel 
consistencyLevel, boolean shouldHint) throws WriteTimeoutException
+private static void commitPaxos(Commit proposal, ConsistencyLevel 
consistencyLevel, boolean allowHints) throws WriteTimeoutException
 {
 boolean shouldBlock = consistencyLevel != ConsistencyLevel.ANY;
 Keyspace keyspace = Keyspace.open(proposal.update.metadata().ksName);
@@ -523,14 +523,14 @@ public class StorageProxy implements StorageProxyMBean
 if (canDoLocalRequest(destination))
 commitPaxosLocal(message, responseHandler);
 else
-MessagingService.instance().sendRR(message, 
destination, responseHandler, shouldHint);
+MessagingService.instance().sendRR(message, 
destination, responseHandler, allowHints && shouldHint(destination));
 }
 else
 {
 MessagingService.instance().sendOneWay(message, 
destination);
 }
 }
-else if (shouldHint)
+else if (allowHints && shouldHint(destination))
 {
 submitHint(proposal.makeMutation(), destination, null);
 }


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[2/6] cassandra git commit: Respect max hint window when hinting for LWT

[aleksey]

Respect max hint window when hinting for LWT

patch by Kurt Greaves; reviewed by Aleksey Yeschenko for CASSANDRA-14215


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fdc61cb6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fdc61cb6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fdc61cb6

Branch: refs/heads/cassandra-3.11
Commit: fdc61cb6a74163402181d133519462f0ab1b504d
Parents: 3d7471a
Author: kurt 
Authored: Tue Feb 20 00:59:00 2018 +
Committer: Aleksey Yeshchenko 
Committed: Thu Mar 15 17:13:56 2018 +0300

--
 CHANGES.txt | 8 +---
 src/java/org/apache/cassandra/service/StorageProxy.java | 6 +++---
 2 files changed, 8 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/fdc61cb6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 62b904f..6099b01 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,6 @@
 3.0.17
- * Adding missing WriteType enum values to v3,v4 and v5 spec (CASSANDRA-13697)
+ * Respect max hint window when hinting for LWT (CASSANDRA-14215)
+ * Adding missing WriteType enum values to v3, v4, and v5 spec 
(CASSANDRA-13697)
  * Don't regenerate bloomfilter and summaries on startup (CASSANDRA-11163)
  * Fix NPE when performing comparison against a null frozen in LWT 
(CASSANDRA-14087)
  * Log when SSTables are deleted (CASSANDRA-14302)
@@ -8,10 +9,11 @@
  * Chain commit log marker potential performance regression in batch commit 
mode (CASSANDRA-14194)
  * Fully utilise specified compaction threads (CASSANDRA-14210)
  * Pre-create deletion log records to finish compactions quicker 
(CASSANDRA-12763)
-Merged from 2.1:
- * CVE-2017-5929 Security vulnerability in Logback warning in NEWS.txt 
(CASSANDRA-14183)
 Merged from 2.2:
  * Backport circleci yaml (CASSANDRA-14240)
+Merged from 2.1:
+ * CVE-2017-5929 Security vulnerability in Logback warning in NEWS.txt 
(CASSANDRA-14183)
+
 
 3.0.16
  * Fix unit test failures in ViewComplexTest (CASSANDRA-14219)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/fdc61cb6/src/java/org/apache/cassandra/service/StorageProxy.java
--
diff --git a/src/java/org/apache/cassandra/service/StorageProxy.java 
b/src/java/org/apache/cassandra/service/StorageProxy.java
index 7a6bed4..bf58ef2 100644
--- a/src/java/org/apache/cassandra/service/StorageProxy.java
+++ b/src/java/org/apache/cassandra/service/StorageProxy.java
@@ -497,7 +497,7 @@ public class StorageProxy implements StorageProxyMBean
 return false;
 }
 
-private static void commitPaxos(Commit proposal, ConsistencyLevel 
consistencyLevel, boolean shouldHint) throws WriteTimeoutException
+private static void commitPaxos(Commit proposal, ConsistencyLevel 
consistencyLevel, boolean allowHints) throws WriteTimeoutException
 {
 boolean shouldBlock = consistencyLevel != ConsistencyLevel.ANY;
 Keyspace keyspace = Keyspace.open(proposal.update.metadata().ksName);
@@ -523,14 +523,14 @@ public class StorageProxy implements StorageProxyMBean
 if (canDoLocalRequest(destination))
 commitPaxosLocal(message, responseHandler);
 else
-MessagingService.instance().sendRR(message, 
destination, responseHandler, shouldHint);
+MessagingService.instance().sendRR(message, 
destination, responseHandler, allowHints && shouldHint(destination));
 }
 else
 {
 MessagingService.instance().sendOneWay(message, 
destination);
 }
 }
-else if (shouldHint)
+else if (allowHints && shouldHint(destination))
 {
 submitHint(proposal.makeMutation(), destination, null);
 }


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[aleksey]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/19a54786
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/19a54786
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/19a54786

Branch: refs/heads/cassandra-3.11
Commit: 19a547861ad6c68e4ce584ee1b9ffee66bd5042d
Parents: 4c6a10a fdc61cb
Author: Aleksey Yeshchenko 
Authored: Thu Mar 15 17:17:23 2018 +0300
Committer: Aleksey Yeshchenko 
Committed: Thu Mar 15 17:17:23 2018 +0300

--
 CHANGES.txt | 3 ++-
 src/java/org/apache/cassandra/service/StorageProxy.java | 6 +++---
 2 files changed, 5 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/19a54786/CHANGES.txt
--
diff --cc CHANGES.txt
index fcd86ed,6099b01..bae967f
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,10 -1,6 +1,11 @@@
 -3.0.17
 +3.11.3
 + * SASI tokenizer for simple delimiter based entries (CASSANDRA-14247)
 + * Fix Loss of digits when doing CAST from varint/bigint to decimal 
(CASSANDRA-14170)
 + * RateBasedBackPressure unnecessarily invokes a lock on the Guava 
RateLimiter (CASSANDRA-14163)
 + * Fix wildcard GROUP BY queries (CASSANDRA-14209)
 +Merged from 3.0:
-  * Adding missing WriteType enum values to v3,v4 and v5 spec (CASSANDRA-13697)
+  * Respect max hint window when hinting for LWT (CASSANDRA-14215)
+  * Adding missing WriteType enum values to v3, v4, and v5 spec 
(CASSANDRA-13697)
   * Don't regenerate bloomfilter and summaries on startup (CASSANDRA-11163)
   * Fix NPE when performing comparison against a null frozen in LWT 
(CASSANDRA-14087)
   * Log when SSTables are deleted (CASSANDRA-14302)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/19a54786/src/java/org/apache/cassandra/service/StorageProxy.java
--
diff --cc src/java/org/apache/cassandra/service/StorageProxy.java
index 4dc05e3,bf58ef2..52dd8ab
--- a/src/java/org/apache/cassandra/service/StorageProxy.java
+++ b/src/java/org/apache/cassandra/service/StorageProxy.java
@@@ -522,7 -497,7 +522,7 @@@ public class StorageProxy implements St
  return false;
  }
  
- private static void commitPaxos(Commit proposal, ConsistencyLevel 
consistencyLevel, boolean shouldHint, long queryStartNanoTime) throws 
WriteTimeoutException
 -private static void commitPaxos(Commit proposal, ConsistencyLevel 
consistencyLevel, boolean allowHints) throws WriteTimeoutException
++private static void commitPaxos(Commit proposal, ConsistencyLevel 
consistencyLevel, boolean allowHints, long queryStartNanoTime) throws 
WriteTimeoutException
  {
  boolean shouldBlock = consistencyLevel != ConsistencyLevel.ANY;
  Keyspace keyspace = Keyspace.open(proposal.update.metadata().ksName);


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[aleksey]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/49c1a256
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/49c1a256
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/49c1a256

Branch: refs/heads/trunk
Commit: 49c1a256c28e913e1e2af25dd308c8835291e7d7
Parents: 92cde9b 19a5478
Author: Aleksey Yeshchenko 
Authored: Thu Mar 15 17:20:09 2018 +0300
Committer: Aleksey Yeshchenko 
Committed: Thu Mar 15 17:20:09 2018 +0300

--
 CHANGES.txt | 3 ++-
 src/java/org/apache/cassandra/service/StorageProxy.java | 6 +++---
 2 files changed, 5 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/49c1a256/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/49c1a256/src/java/org/apache/cassandra/service/StorageProxy.java
--
diff --cc src/java/org/apache/cassandra/service/StorageProxy.java
index b512722,52dd8ab..a47c4c6
--- a/src/java/org/apache/cassandra/service/StorageProxy.java
+++ b/src/java/org/apache/cassandra/service/StorageProxy.java
@@@ -586,14 -522,14 +586,14 @@@ public class StorageProxy implements St
  return false;
  }
  
- private static void commitPaxos(Commit proposal, ConsistencyLevel 
consistencyLevel, boolean shouldHint, long queryStartNanoTime) throws 
WriteTimeoutException
+ private static void commitPaxos(Commit proposal, ConsistencyLevel 
consistencyLevel, boolean allowHints, long queryStartNanoTime) throws 
WriteTimeoutException
  {
  boolean shouldBlock = consistencyLevel != ConsistencyLevel.ANY;
 -Keyspace keyspace = Keyspace.open(proposal.update.metadata().ksName);
 +Keyspace keyspace = 
Keyspace.open(proposal.update.metadata().keyspace);
  
  Token tk = proposal.update.partitionKey().getToken();
 -List naturalEndpoints = 
StorageService.instance.getNaturalEndpoints(keyspace.getName(), tk);
 -Collection pendingEndpoints = 
StorageService.instance.getTokenMetadata().pendingEndpointsFor(tk, 
keyspace.getName());
 +List naturalEndpoints = 
StorageService.instance.getNaturalEndpoints(keyspace.getName(), tk);
 +Collection pendingEndpoints = 
StorageService.instance.getTokenMetadata().pendingEndpointsFor(tk, 
keyspace.getName());
  
  AbstractWriteResponseHandler responseHandler = null;
  if (shouldBlock)
@@@ -622,16 -558,9 +622,16 @@@
  MessagingService.instance().sendOneWay(message, 
destination);
  }
  }
 -else if (allowHints && shouldHint(destination))
 +else
  {
 -submitHint(proposal.makeMutation(), destination, null);
 +if (responseHandler != null)
 +{
 +responseHandler.expired();
 +}
- if (shouldHint)
++if (allowHints && shouldHint(destination))
 +{
 +submitHint(proposal.makeMutation(), destination, null);
 +}
  }
  }
  


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14215) Cassandra does not respect hint window for CAS

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-14215:
--
Status: Ready to Commit  (was: Patch Available)

> Cassandra does not respect hint window for CAS
> --
>
> Key: CASSANDRA-14215
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14215
> Project: Cassandra
>  Issue Type: Bug
>  Components: Hints, Streaming and Messaging
>Reporter: Arijit Banerjee
>Assignee: Kurt Greaves
>Priority: Major
>
> On Cassandra 3.0.9, it was observed that Cassandra continues to write hints 
> even though a node remains down (and does not come up) for longer than the 
> default 3 hour window.
>  
> After doing "nodetool setlogginglevel org.apache.cassandra TRACE", we see the 
> following log line in cassandra (debug) logs:
>  StorageProxy.java:2625 - Adding hints for [/10.0.100.84]
>  
> One possible code path seems to be:
> cas -> commitPaxos(proposal, consistencyForCommit, true); -> submitHint (in 
> StorageProxy.java)
>  
> The "true" parameter above explicitly states that a hint should be recorded 
> and ignores the time window calculation performed by the shouldHint method 
> invoked in other code paths. Is there a reason for this behavior?
>  
> Edit: There are actually two stacks that seem to be producing hints, the 
> "cas" and "syncWriteBatchedMutations" methods. I have posted them below.
>  
> A third issue seems to be that Cassandra seems to reset the timer which 
> counts how long a node has been down after a restart. Thus if Cassandra is 
> restarted on a good node, it continues to accumulate hints for a down node 
> over the next three hours.
>  
> {code:java}
> WARN [SharedPool-Worker-14] 2018-02-06 22:15:51,136 StorageProxy.java:2636 - 
> Adding hints for [/10.0.100.84] with stack trace: java.lang.Throwable: at 
> org.apache.cassandra.service.StorageProxy.stackTrace(StorageProxy.java:2608) 
> at 
> org.apache.cassandra.service.StorageProxy.submitHint(StorageProxy.java:2617) 
> at 
> org.apache.cassandra.service.StorageProxy.submitHint(StorageProxy.java:2603) 
> at 
> org.apache.cassandra.service.StorageProxy.commitPaxos(StorageProxy.java:540) 
> at org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:282) at 
> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:432)
>  at 
> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:407)
>  at 
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
>  at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:237) 
> at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222) 
> at 
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115)
>  at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513)
>  at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407)
>  at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
>  at 
> io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> at 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) at 
> java.lang.Thread.run(Thread.java:748) WARN
> {code}
> {code:java}
> [SharedPool-Worker-8] 2018-02-06 22:15:51,153 StorageProxy.java:2636 - Adding 
> hints for [/10.0.100.84] with stack trace: java.lang.Throwable: at 
> org.apache.cassandra.service.StorageProxy.stackTrace(StorageProxy.java:2608) 
> at 
> org.apache.cassandra.service.StorageProxy.submitHint(StorageProxy.java:2617) 
> at 
> org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:1247)
>  at 
> org.apache.cassandra.service.StorageProxy.syncWriteBatchedMutations(StorageProxy.java:1014)
>  at 
> org.apache.cassandra.service.StorageProxy.mutateAtomically(StorageProxy.java:899)
>  at 
> org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:834)
>  at 
> org.apache.cassandra.cql3.statements.BatchStatement.executeWithoutConditions(BatchStatement.java:365)
>  at 
> org.apache.cassandra.cql3.statements.BatchStatement.execute(BatchStatement.java:343)
>  at 
> org.apache.cassandra.cql3.statements.BatchStatement.execute(BatchStatement.java:329)
>  at 
> 

[jira] [Commented] (CASSANDRA-14302) Log when sstables are deleted

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400202#comment-16400202
 ] 

Marcus Eriksson commented on CASSANDRA-14302:
-

this should probably go to DEBUG - looking at system.log right now is quite 
confusing since it looks like the node is doing nothing but deleting files

> Log when sstables are deleted
> -
>
> Key: CASSANDRA-14302
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14302
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
> Fix For: 4.0, 3.0.x, 3.11.x
>
>
> This was removed in 3.0 and is super helpful for debugging issues in prod



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14247) SASI tokenizer for simple delimiter based entries

[mck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mck updated CASSANDRA-14247:

Fix Version/s: (was: 3.11.x)
   3.11.3

> SASI tokenizer for simple delimiter based entries
> -
>
> Key: CASSANDRA-14247
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14247
> Project: Cassandra
>  Issue Type: Improvement
>  Components: sasi
>Reporter: mck
>Assignee: mck
>Priority: Major
>  Labels: sasi
> Fix For: 4.0, 3.11.3
>
>
> Currently SASI offers only two tokenizer options:
>  - NonTokenizerAnalyser
>  - StandardAnalyzer
> The latter is built upon Snowball, powerful for human languages but overkill 
> for simple tokenization.
> A simple tokenizer is proposed here. The need for this arose as a workaround 
> of CASSANDRA-11182, and to avoid the disk usage explosion when having to 
> resort to {{CONTAINS}}. See https://github.com/openzipkin/zipkin/issues/1861
> Example use of this would be:
> {code}
> CREATE CUSTOM INDEX span_annotation_query_idx 
> ON zipkin2.span (annotation_query) USING 
> 'org.apache.cassandra.index.sasi.SASIIndex' 
> WITH OPTIONS = {
> 'analyzer_class': 
> 'org.apache.cassandra.index.sasi.analyzer.DelimiterAnalyzer', 
> 'delimiter': '░',
> 'case_sensitive': 'true', 
> 'mode': 'prefix', 
> 'analyzed': 'true'};
> {code}
> Original credit for this work goes to https://github.com/zuochangan



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14247) SASI tokenizer for simple delimiter based entries

[mck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mck updated CASSANDRA-14247:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed.

> SASI tokenizer for simple delimiter based entries
> -
>
> Key: CASSANDRA-14247
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14247
> Project: Cassandra
>  Issue Type: Improvement
>  Components: sasi
>Reporter: mck
>Assignee: mck
>Priority: Major
>  Labels: sasi
> Fix For: 4.0, 3.11.3
>
>
> Currently SASI offers only two tokenizer options:
>  - NonTokenizerAnalyser
>  - StandardAnalyzer
> The latter is built upon Snowball, powerful for human languages but overkill 
> for simple tokenization.
> A simple tokenizer is proposed here. The need for this arose as a workaround 
> of CASSANDRA-11182, and to avoid the disk usage explosion when having to 
> resort to {{CONTAINS}}. See https://github.com/openzipkin/zipkin/issues/1861
> Example use of this would be:
> {code}
> CREATE CUSTOM INDEX span_annotation_query_idx 
> ON zipkin2.span (annotation_query) USING 
> 'org.apache.cassandra.index.sasi.SASIIndex' 
> WITH OPTIONS = {
> 'analyzer_class': 
> 'org.apache.cassandra.index.sasi.analyzer.DelimiterAnalyzer', 
> 'delimiter': '░',
> 'case_sensitive': 'true', 
> 'mode': 'prefix', 
> 'analyzed': 'true'};
> {code}
> Original credit for this work goes to https://github.com/zuochangan



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[5/7] cassandra git commit: SASI tokenizer for simple delimiter based entries

[mck]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/4c6a10af/test/resources/tokenization/world_cities_a.csv
--
diff --git a/test/resources/tokenization/world_cities_a.csv 
b/test/resources/tokenization/world_cities_a.csv
new file mode 100644
index 000..1bb7121
--- /dev/null
+++ b/test/resources/tokenization/world_cities_a.csv
@@ -0,0 +1 @@
+aaba,aaba kebire,aaba serhire,aabaan weyn,aabaan 
yare,aabarak,aabasen,aabasene,aabassiye,aabateine,aabauerschaft,aabayat,aabb ej 
jinnate,aabb el khazne,aabba,aabbade,aabbassiye,aabboud,aabboudiye,aabd ech 
cheikh,aabd el fattah,aabdekliye,aabdet 
biche,aabdille,aabdilli,aabdine,aabdou,aabeidat,aabeidate,aabenraa,aaber beit 
sif,aabey,aabjaerg,aabjerg,aabjorgan,aabla,aable,aabo,aabol,aabom,aaboud,aabow,aabra,aabreita,aabreite,aabrine,aabtine,aaby,aabybro,aabyhoj,aabyskov,aach,aach-linz,aachach,aacharheu
 tachleuk,aacharne,aachchet charqiye,aache,aachen,aachen-rothe erde,aacheq 
omar,aachini,aachiran,aachour,aachqane,aachqout,aachqoute,aad la 
macta,aada,aadal,aadalsbruk,aadami,aadan harmey,aadan nuur,aadan waraabow,aadan 
yabaal,aadaouane,aadassiye,aadbel,aadchit,aadchit el 
qoussair,aadchite,aaddaye,aaderup,aadi noia,aadland,aadliye,aadloun,aadma,aadma 
asundus,aadmoun,aadneram,aadneskaar,aadorf,aadorp,aadouane,aadoui,aadouiye,aadouss,aadra,aadzi,aaen,aaenget,aaerne,aafarnes,aafoss,aafrine,a
 
afrite,aafsadiq,aafsdiq,aafsouniye,aag,aagaard,aageby,aagedal,aagerup,aagoubt,aagskardet,aagskaret,aagtdorn,aagtdorp,aagtekerke,aaguil,aahire,aaichiye,aaidamoun,aaien,aaigem,aaiha,aaijen,aailiyine,aaimar,aaintoura,aaintourine,aaita,aaita
 el foukhar,aaita ez 
zott,aaitanit,aaitaniya,aaium,aaiun,aaiyat,aajaja,aajaltoun,aajami,aajeilate,aajeltoun,aajjaz,aajjel,aajjoura,aajpur,aajuolahti,aakaa,aakae,aakaer,aakakir,aakar,aakaru,aakb
 el 
bellene,aaker,aakeroen,aakerviken,aakfor,aakiba,aakil,aakip,aakirkeby,aakjaer,aakkar
 el 
aatiqa,aakkari,aakkour,aakla,aaklangberget,aaknes,aakobar,aakoinen,aakra,aakrak,aakre,aakre
 
asundus,aakrehamn,aakvaag,aakvik,aakyor,aal,aala,aalalech,aalamdar,aalampur,aalanden,aalane,aalanganjar,aalatsivik,aalbaek,aalbeek,aalbeke,aalborg,aalbroek,aalbu,aalburg,aalden,aaldering,aalderink,aaldonk,aale,aalebaek,aalebaek
 
huse,aalefjaer,aalen,aaleskov,aalestrup,aalesund,aaley,aalfang,aalfoten,aalgaard,aalgana,aali
 badrane,aali bajliye,aali enn nahri,aali farro,aali koz,aali 
 
qorane,aalia,aalidji,aalio,aalissa,aalita,aaliyate,aaliye,aaljoki,aalkaer,aalkorb,aallaakh,aallani,aalling,aalma,aalma
 ech chaab,aalmane,aalmane ed 
daiaa,aalmate,aalmine,aalmo,aalo,aalokke,aalose,aaloua,aalouannk,aalougoum,aalouine,aalouk,aalouq
 charqi,aalouq 
rharbi,aalqine,aalsbo,aalset,aalsgaard,aalsgaarde,aalsmeer,aalsmeerderbrug,aalso,aalsrode,aalst,aalst
 sint-pieter,aalst-saint 
pierre,aalstad,aalstrup,aalsum,aaltebei,aalten,aalter,aalterbei,aalterburg,aalum,aalvik,aalykke,aalzum,aam,aam
 tam,aamair el buied,aamar,aamar el beikate,aamar el bikat,aamara,aamaret el 
bikat,aamarne,aamarne faouqani,aamarne 
tahtani,aamatour,aamayer,aamba,aamchite,aamdal,aamdals 
vaerk,aamelfot,aamer,aaminaay,aamire,aamland,aamli,aammariye,aammatour,aammiq,aammoun,aammourine,aamnes,aamot,aamot-modum,aamoud,aamouda,aamoudi,aamoudiye,aamoyhamn,aamqa,aamqiye,aamra,aamret
 el freij,aamriye,aamse,aamsi,aamsosen,aamzit,aan,aan t holven,aan t 
hoolven,aan de berg,aan de bergen,aan de bunders,aan de doorns,aan de 
 kippen,aan de luyssen,aan de maas,aan de popelaar,aan de school,aan de 
steenoven,aan de vreenen,aan de vrenen,aan de wolfsberg,aan de zuwe,aan den 
nieuwen vijver,aan den steenoven,aan het broek,aan 
reijans,aana,aana-sira,aanaar,aanadane,aanadane ech 
chih,aanane,aananib,aanaye,aanaz,aanbal,aancud,aandal,aandalen,aandara,aandenk,aandepoppelaar,aandervaag,aandqet,aandu,aaneby,aanekoski,aaneland,aaneq
 al haoua,aaneq baiarra,aaneq el 
haoua,aanerne,aanes,aangel,aangenaam,aangewysd,aangstad,aanhimaki,aanhou,aanislag,aaniss,aanjar,aankibane,aankomst,aannabe,aannabiye,aannaqiyet
 miri,aannaqiyet 
tchiftlik,aannaya,aannaze,aannbal,aanndaqatt,aannerud,aannestad,aannjara,aannkaoui,aannkibane,aannouz,aanntariye,aannzaouiye,aanosusura,aanout,aanoute,aanqoun,aanschot,aanshu,aanstad,aansvik,aantaara,aantoura,aanum,aanwas,aaouachiye,aaouadi,aaouainat,aaouaini,aaouainiye,aaouaj,aaouamiye,aaouar,aaouara,aaouas,aaouass,aaoudet
 el qalamoun,aaoueichiye,aaoueidja,aaoueijet jine,aaoueiline,aaoueina,aaoueinat
 e,aaoueinate kebire,aaoueinate serhire,aaoueine,aaoueinet er 
rihane,aaoueini,aaoueira,aaoueira 
charqiye,aaoueisse,aaouenia,aaouifate,aaouint--maazouza,aaouj ej jenah,aaouj el 
jenah,aaouje,aaoukar,aaoumar ed 
dine,aaouniye,aaoura,aaousiene,aapaasdal,aapaasland,aapajarven 
asutusalue,aapajarvi,aapajoki,aapdraai,aappilattoq,aappilattorq,aapua,aaqabet 
hairouna,aaqareb es safi,aaqarib es safi,aaqb el ballane,aaqbet roummane,aaqbet 
sebaail,aaqeibe,aaqerbate,aaqerbe,aaqiba,aaqile,aaqlain,aaqliya,aaqliye,aaqmata,aaqoula,aaqoulet
 el 

[3/7] cassandra git commit: SASI tokenizer for simple delimiter based entries

[mck]

SASI tokenizer for simple delimiter based entries

Patch by Mick Semb Wever; reviewed by Michael Kjellman for CASSANDRA-14247


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4c6a10af
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4c6a10af
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4c6a10af

Branch: refs/heads/cassandra-3.11
Commit: 4c6a10af8bd5247348d5445e6fd3a2f141e50bc1
Parents: 5a662ea
Author: Mick Semb Wever 
Authored: Wed Feb 21 21:16:05 2018 +1100
Committer: Mick Semb Wever 
Committed: Thu Mar 15 17:41:50 2018 +1100

--
 CHANGES.txt |   1 +
 .../index/sasi/analyzer/DelimiterAnalyzer.java  | 103 +++
 .../analyzer/DelimiterTokenizingOptions.java|  71 +
 test/resources/tokenization/world_cities_a.csv  |   1 +
 .../sasi/analyzer/DelimiterAnalyzerTest.java| 282 +++
 5 files changed, 458 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/4c6a10af/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 8f913cd..fcd86ed 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.11.3
+ * SASI tokenizer for simple delimiter based entries (CASSANDRA-14247)
  * Fix Loss of digits when doing CAST from varint/bigint to decimal 
(CASSANDRA-14170)
  * RateBasedBackPressure unnecessarily invokes a lock on the Guava RateLimiter 
(CASSANDRA-14163)
  * Fix wildcard GROUP BY queries (CASSANDRA-14209)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/4c6a10af/src/java/org/apache/cassandra/index/sasi/analyzer/DelimiterAnalyzer.java
--
diff --git 
a/src/java/org/apache/cassandra/index/sasi/analyzer/DelimiterAnalyzer.java 
b/src/java/org/apache/cassandra/index/sasi/analyzer/DelimiterAnalyzer.java
new file mode 100644
index 000..794a8b9
--- /dev/null
+++ b/src/java/org/apache/cassandra/index/sasi/analyzer/DelimiterAnalyzer.java
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.index.sasi.analyzer;
+
+import java.nio.CharBuffer;
+import java.nio.ByteBuffer;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Preconditions;
+
+import org.apache.cassandra.db.marshal.AbstractType;
+import org.apache.cassandra.db.marshal.AsciiType;
+import org.apache.cassandra.db.marshal.UTF8Type;
+import org.apache.cassandra.utils.AbstractIterator;
+
+@Beta
+public class DelimiterAnalyzer extends AbstractAnalyzer
+{
+
+private static final Map VALID_ANALYZABLE_TYPES = 
new HashMap()
+{{
+put(UTF8Type.instance, StandardCharsets.UTF_8);
+put(AsciiType.instance, StandardCharsets.US_ASCII);
+}};
+
+private char delimiter;
+private Charset charset;
+private Iterator iter;
+
+public DelimiterAnalyzer()
+{
+}
+
+public ByteBuffer next()
+{
+return iter.next();
+}
+
+public void init(Map options, AbstractType validator)
+{
+DelimiterTokenizingOptions tokenizingOptions = 
DelimiterTokenizingOptions.buildFromMap(options);
+delimiter = tokenizingOptions.getDelimiter();
+
+if (!VALID_ANALYZABLE_TYPES.containsKey(validator))
+throw new IllegalArgumentException(String.format("Only text types 
supported, got %s", validator));
+
+charset = VALID_ANALYZABLE_TYPES.get(validator);
+}
+
+public boolean hasNext()
+{
+return iter.hasNext();
+}
+
+public void reset(ByteBuffer input)
+{
+Preconditions.checkNotNull(input);
+final CharBuffer cb = charset.decode(input);
+
+this.iter = new AbstractIterator() {
+protected 

[6/7] cassandra git commit: SASI tokenizer for simple delimiter based entries

[mck]

SASI tokenizer for simple delimiter based entries

Patch by Mick Semb Wever; reviewed by Michael Kjellman for CASSANDRA-14247


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4c6a10af
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4c6a10af
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4c6a10af

Branch: refs/heads/trunk
Commit: 4c6a10af8bd5247348d5445e6fd3a2f141e50bc1
Parents: 5a662ea
Author: Mick Semb Wever 
Authored: Wed Feb 21 21:16:05 2018 +1100
Committer: Mick Semb Wever 
Committed: Thu Mar 15 17:41:50 2018 +1100

--
 CHANGES.txt |   1 +
 .../index/sasi/analyzer/DelimiterAnalyzer.java  | 103 +++
 .../analyzer/DelimiterTokenizingOptions.java|  71 +
 test/resources/tokenization/world_cities_a.csv  |   1 +
 .../sasi/analyzer/DelimiterAnalyzerTest.java| 282 +++
 5 files changed, 458 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/4c6a10af/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 8f913cd..fcd86ed 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.11.3
+ * SASI tokenizer for simple delimiter based entries (CASSANDRA-14247)
  * Fix Loss of digits when doing CAST from varint/bigint to decimal 
(CASSANDRA-14170)
  * RateBasedBackPressure unnecessarily invokes a lock on the Guava RateLimiter 
(CASSANDRA-14163)
  * Fix wildcard GROUP BY queries (CASSANDRA-14209)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/4c6a10af/src/java/org/apache/cassandra/index/sasi/analyzer/DelimiterAnalyzer.java
--
diff --git 
a/src/java/org/apache/cassandra/index/sasi/analyzer/DelimiterAnalyzer.java 
b/src/java/org/apache/cassandra/index/sasi/analyzer/DelimiterAnalyzer.java
new file mode 100644
index 000..794a8b9
--- /dev/null
+++ b/src/java/org/apache/cassandra/index/sasi/analyzer/DelimiterAnalyzer.java
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.index.sasi.analyzer;
+
+import java.nio.CharBuffer;
+import java.nio.ByteBuffer;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+
+import com.google.common.annotations.Beta;
+import com.google.common.base.Preconditions;
+
+import org.apache.cassandra.db.marshal.AbstractType;
+import org.apache.cassandra.db.marshal.AsciiType;
+import org.apache.cassandra.db.marshal.UTF8Type;
+import org.apache.cassandra.utils.AbstractIterator;
+
+@Beta
+public class DelimiterAnalyzer extends AbstractAnalyzer
+{
+
+private static final Map VALID_ANALYZABLE_TYPES = 
new HashMap()
+{{
+put(UTF8Type.instance, StandardCharsets.UTF_8);
+put(AsciiType.instance, StandardCharsets.US_ASCII);
+}};
+
+private char delimiter;
+private Charset charset;
+private Iterator iter;
+
+public DelimiterAnalyzer()
+{
+}
+
+public ByteBuffer next()
+{
+return iter.next();
+}
+
+public void init(Map options, AbstractType validator)
+{
+DelimiterTokenizingOptions tokenizingOptions = 
DelimiterTokenizingOptions.buildFromMap(options);
+delimiter = tokenizingOptions.getDelimiter();
+
+if (!VALID_ANALYZABLE_TYPES.containsKey(validator))
+throw new IllegalArgumentException(String.format("Only text types 
supported, got %s", validator));
+
+charset = VALID_ANALYZABLE_TYPES.get(validator);
+}
+
+public boolean hasNext()
+{
+return iter.hasNext();
+}
+
+public void reset(ByteBuffer input)
+{
+Preconditions.checkNotNull(input);
+final CharBuffer cb = charset.decode(input);
+
+this.iter = new AbstractIterator() {
+protected ByteBuffer 

[7/7] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[mck]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/92cde9b2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/92cde9b2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/92cde9b2

Branch: refs/heads/trunk
Commit: 92cde9b27c341ebb03a8d6be2d8d32b0ba548db6
Parents: d4dfbb5 4c6a10a
Author: Mick Semb Wever 
Authored: Thu Mar 15 17:42:31 2018 +1100
Committer: Mick Semb Wever 
Committed: Thu Mar 15 17:45:10 2018 +1100

--
 CHANGES.txt |   1 +
 doc/SASI.md |  21 ++
 .../index/sasi/analyzer/DelimiterAnalyzer.java  | 103 +++
 .../analyzer/DelimiterTokenizingOptions.java|  71 +
 test/resources/tokenization/world_cities_a.csv  |   1 +
 .../sasi/analyzer/DelimiterAnalyzerTest.java| 282 +++
 6 files changed, 479 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/92cde9b2/CHANGES.txt
--
diff --cc CHANGES.txt
index f2b9f7c,fcd86ed..c049991
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,206 -1,5 +1,207 @@@
 +4.0
 + * Forced incremental repairs should promote sstables if they can 
(CASSANDRA-14294)
 + * Use Murmur3 for validation compactions (CASSANDRA-14002)
 + * Comma at the end of the seed list is interpretated as localhost 
(CASSANDRA-14285)
 + * Refactor read executor and response resolver, abstract read repair 
(CASSANDRA-14058)
 + * Add optional startup delay to wait until peers are ready (CASSANDRA-13993)
 + * Add a few options to nodetool verify (CASSANDRA-14201)
 + * CVE-2017-5929 Security vulnerability and redefine default log rotation 
policy (CASSANDRA-14183)
 + * Use JVM default SSL validation algorithm instead of custom default 
(CASSANDRA-13259)
 + * Better document in code InetAddressAndPort usage post 7544, incorporate 
port into UUIDGen node (CASSANDRA-14226)
 + * Fix sstablemetadata date string for minLocalDeletionTime (CASSANDRA-14132)
 + * Make it possible to change neverPurgeTombstones during runtime 
(CASSANDRA-14214)
 + * Remove GossipDigestSynVerbHandler#doSort() (CASSANDRA-14174)
 + * Add nodetool clientlist (CASSANDRA-13665)
 + * Revert ProtocolVersion changes from CASSANDRA-7544 (CASSANDRA-14211)
 + * Non-disruptive seed node list reload (CASSANDRA-14190)
 + * Nodetool tablehistograms to print statics for all the tables 
(CASSANDRA-14185)
 + * Migrate dtests to use pytest and python3 (CASSANDRA-14134)
 + * Allow storage port to be configurable per node (CASSANDRA-7544)
 + * Make sub-range selection for non-frozen collections return null instead of 
empty (CASSANDRA-14182)
 + * BloomFilter serialization format should not change byte ordering 
(CASSANDRA-9067)
 + * Remove unused on-heap BloomFilter implementation (CASSANDRA-14152)
 + * Delete temp test files on exit (CASSANDRA-14153)
 + * Make PartitionUpdate and Mutation immutable (CASSANDRA-13867)
 + * Fix CommitLogReplayer exception for CDC data (CASSANDRA-14066)
 + * Fix cassandra-stress startup failure (CASSANDRA-14106)
 + * Remove initialDirectories from CFS (CASSANDRA-13928)
 + * Fix trivial log format error (CASSANDRA-14015)
 + * Allow sstabledump to do a json object per partition (CASSANDRA-13848)
 + * Add option to optimise merkle tree comparison across replicas 
(CASSANDRA-3200)
 + * Remove unused and deprecated methods from AbstractCompactionStrategy 
(CASSANDRA-14081)
 + * Fix Distribution.average in cassandra-stress (CASSANDRA-14090)
 + * Support a means of logging all queries as they were invoked 
(CASSANDRA-13983)
 + * Presize collections (CASSANDRA-13760)
 + * Add GroupCommitLogService (CASSANDRA-13530)
 + * Parallelize initial materialized view build (CASSANDRA-12245)
 + * Fix flaky SecondaryIndexManagerTest.assert[Not]MarkedAsBuilt 
(CASSANDRA-13965)
 + * Make LWTs send resultset metadata on every request (CASSANDRA-13992)
 + * Fix flaky indexWithFailedInitializationIsNotQueryableAfterPartialRebuild 
(CASSANDRA-13963)
 + * Introduce leaf-only iterator (CASSANDRA-9988)
 + * Upgrade Guava to 23.3 and Airline to 0.8 (CASSANDRA-13997)
 + * Allow only one concurrent call to StatusLogger (CASSANDRA-12182)
 + * Refactoring to specialised functional interfaces (CASSANDRA-13982)
 + * Speculative retry should allow more friendly params (CASSANDRA-13876)
 + * Throw exception if we send/receive repair messages to incompatible nodes 
(CASSANDRA-13944)
 + * Replace usages of MessageDigest with Guava's Hasher (CASSANDRA-13291)
 + * Add nodetool cmd to print hinted handoff window (CASSANDRA-13728)
 + * Fix some alerts raised by static analysis (CASSANDRA-13799)
 + * Checksum sstable metadata (CASSANDRA-13321, CASSANDRA-13593)
 + * Add result set metadata 

[1/7] cassandra git commit: SASI tokenizer for simple delimiter based entries

[mck]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.11 5a662ea3f -> 4c6a10af8
  refs/heads/trunk d4dfbb5c6 -> 92cde9b27


http://git-wip-us.apache.org/repos/asf/cassandra/blob/4c6a10af/test/unit/org/apache/cassandra/index/sasi/analyzer/DelimiterAnalyzerTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/index/sasi/analyzer/DelimiterAnalyzerTest.java 
b/test/unit/org/apache/cassandra/index/sasi/analyzer/DelimiterAnalyzerTest.java
new file mode 100644
index 000..15bbb84
--- /dev/null
+++ 
b/test/unit/org/apache/cassandra/index/sasi/analyzer/DelimiterAnalyzerTest.java
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.index.sasi.analyzer;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+
+import org.apache.cassandra.db.marshal.Int32Type;
+import org.apache.cassandra.db.marshal.UTF8Type;
+import org.apache.cassandra.utils.ByteBufferUtil;
+import org.apache.commons.io.IOUtils;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+
+public class DelimiterAnalyzerTest
+{
+
+@Test
+public void caseSensitiveAnalizer() throws Exception
+{
+DelimiterAnalyzer analyzer = new DelimiterAnalyzer();
+
+analyzer.init(
+new HashMap()
+{{
+put(DelimiterTokenizingOptions.DELIMITER, " ");
+}},
+UTF8Type.instance);
+
+String testString = "Nip it in the bud";
+ByteBuffer toAnalyze = ByteBuffer.wrap(testString.getBytes());
+analyzer.reset(toAnalyze);
+StringBuilder output = new StringBuilder();
+while (analyzer.hasNext())
+output.append(ByteBufferUtil.string(analyzer.next()) + 
(analyzer.hasNext() ? ' ' : ""));
+
+Assert.assertTrue(testString.equals(output.toString()));
+Assert.assertFalse(testString.toLowerCase().equals(output.toString()));
+}
+
+@Test(expected = IllegalArgumentException.class)
+public void ensureIncompatibleInputSkipped() throws Exception
+{
+new DelimiterAnalyzer().init(new HashMap(), Int32Type.instance);
+}
+
+@Test
+public void testTokenizationLoremIpsum() throws Exception
+{
+ByteBuffer bb = ByteBuffer.wrap(IOUtils.toByteArray(
+
DelimiterAnalyzerTest.class.getClassLoader().getResourceAsStream("tokenization/lorem_ipsum.txt")));
+
+DelimiterAnalyzer tokenizer = new DelimiterAnalyzer();
+
+tokenizer.init(
+new HashMap()
+{{
+put(DelimiterTokenizingOptions.DELIMITER, " ");
+}},
+UTF8Type.instance);
+
+List tokens = new ArrayList<>();
+tokenizer.reset(bb);
+while (tokenizer.hasNext())
+tokens.add(tokenizer.next());
+
+assertEquals(69, tokens.size());
+
+}
+
+@Test
+public void testTokenizationJaJp1() throws Exception
+{
+ByteBuffer bb = ByteBuffer.wrap(IOUtils.toByteArray(
+
DelimiterAnalyzerTest.class.getClassLoader().getResourceAsStream("tokenization/ja_jp_1.txt")));
+
+DelimiterAnalyzer tokenizer = new DelimiterAnalyzer();
+
+tokenizer.init(
+new HashMap()
+{{
+put(DelimiterTokenizingOptions.DELIMITER, "。");
+}},
+UTF8Type.instance);
+
+tokenizer.reset(bb);
+List tokens = new ArrayList<>();
+while (tokenizer.hasNext())
+tokens.add(tokenizer.next());
+
+assertEquals(4, tokens.size());
+}
+
+@Test
+public void testTokenizationJaJp2() throws Exception
+{
+ByteBuffer bb = ByteBuffer.wrap(IOUtils.toByteArray(
+
DelimiterAnalyzerTest.class.getClassLoader().getResourceAsStream("tokenization/ja_jp_2.txt")));
+
+DelimiterAnalyzer tokenizer = new DelimiterAnalyzer();
+
+tokenizer.init(
+new HashMap()
+{{
+put(DelimiterTokenizingOptions.DELIMITER, "。");
+}},

[2/7] cassandra git commit: SASI tokenizer for simple delimiter based entries

[mck]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/4c6a10af/test/resources/tokenization/world_cities_a.csv
--
diff --git a/test/resources/tokenization/world_cities_a.csv 
b/test/resources/tokenization/world_cities_a.csv
new file mode 100644
index 000..1bb7121
--- /dev/null
+++ b/test/resources/tokenization/world_cities_a.csv
@@ -0,0 +1 @@
+aaba,aaba kebire,aaba serhire,aabaan weyn,aabaan 
yare,aabarak,aabasen,aabasene,aabassiye,aabateine,aabauerschaft,aabayat,aabb ej 
jinnate,aabb el khazne,aabba,aabbade,aabbassiye,aabboud,aabboudiye,aabd ech 
cheikh,aabd el fattah,aabdekliye,aabdet 
biche,aabdille,aabdilli,aabdine,aabdou,aabeidat,aabeidate,aabenraa,aaber beit 
sif,aabey,aabjaerg,aabjerg,aabjorgan,aabla,aable,aabo,aabol,aabom,aaboud,aabow,aabra,aabreita,aabreite,aabrine,aabtine,aaby,aabybro,aabyhoj,aabyskov,aach,aach-linz,aachach,aacharheu
 tachleuk,aacharne,aachchet charqiye,aache,aachen,aachen-rothe erde,aacheq 
omar,aachini,aachiran,aachour,aachqane,aachqout,aachqoute,aad la 
macta,aada,aadal,aadalsbruk,aadami,aadan harmey,aadan nuur,aadan waraabow,aadan 
yabaal,aadaouane,aadassiye,aadbel,aadchit,aadchit el 
qoussair,aadchite,aaddaye,aaderup,aadi noia,aadland,aadliye,aadloun,aadma,aadma 
asundus,aadmoun,aadneram,aadneskaar,aadorf,aadorp,aadouane,aadoui,aadouiye,aadouss,aadra,aadzi,aaen,aaenget,aaerne,aafarnes,aafoss,aafrine,a
 
afrite,aafsadiq,aafsdiq,aafsouniye,aag,aagaard,aageby,aagedal,aagerup,aagoubt,aagskardet,aagskaret,aagtdorn,aagtdorp,aagtekerke,aaguil,aahire,aaichiye,aaidamoun,aaien,aaigem,aaiha,aaijen,aailiyine,aaimar,aaintoura,aaintourine,aaita,aaita
 el foukhar,aaita ez 
zott,aaitanit,aaitaniya,aaium,aaiun,aaiyat,aajaja,aajaltoun,aajami,aajeilate,aajeltoun,aajjaz,aajjel,aajjoura,aajpur,aajuolahti,aakaa,aakae,aakaer,aakakir,aakar,aakaru,aakb
 el 
bellene,aaker,aakeroen,aakerviken,aakfor,aakiba,aakil,aakip,aakirkeby,aakjaer,aakkar
 el 
aatiqa,aakkari,aakkour,aakla,aaklangberget,aaknes,aakobar,aakoinen,aakra,aakrak,aakre,aakre
 
asundus,aakrehamn,aakvaag,aakvik,aakyor,aal,aala,aalalech,aalamdar,aalampur,aalanden,aalane,aalanganjar,aalatsivik,aalbaek,aalbeek,aalbeke,aalborg,aalbroek,aalbu,aalburg,aalden,aaldering,aalderink,aaldonk,aale,aalebaek,aalebaek
 
huse,aalefjaer,aalen,aaleskov,aalestrup,aalesund,aaley,aalfang,aalfoten,aalgaard,aalgana,aali
 badrane,aali bajliye,aali enn nahri,aali farro,aali koz,aali 
 
qorane,aalia,aalidji,aalio,aalissa,aalita,aaliyate,aaliye,aaljoki,aalkaer,aalkorb,aallaakh,aallani,aalling,aalma,aalma
 ech chaab,aalmane,aalmane ed 
daiaa,aalmate,aalmine,aalmo,aalo,aalokke,aalose,aaloua,aalouannk,aalougoum,aalouine,aalouk,aalouq
 charqi,aalouq 
rharbi,aalqine,aalsbo,aalset,aalsgaard,aalsgaarde,aalsmeer,aalsmeerderbrug,aalso,aalsrode,aalst,aalst
 sint-pieter,aalst-saint 
pierre,aalstad,aalstrup,aalsum,aaltebei,aalten,aalter,aalterbei,aalterburg,aalum,aalvik,aalykke,aalzum,aam,aam
 tam,aamair el buied,aamar,aamar el beikate,aamar el bikat,aamara,aamaret el 
bikat,aamarne,aamarne faouqani,aamarne 
tahtani,aamatour,aamayer,aamba,aamchite,aamdal,aamdals 
vaerk,aamelfot,aamer,aaminaay,aamire,aamland,aamli,aammariye,aammatour,aammiq,aammoun,aammourine,aamnes,aamot,aamot-modum,aamoud,aamouda,aamoudi,aamoudiye,aamoyhamn,aamqa,aamqiye,aamra,aamret
 el freij,aamriye,aamse,aamsi,aamsosen,aamzit,aan,aan t holven,aan t 
hoolven,aan de berg,aan de bergen,aan de bunders,aan de doorns,aan de 
 kippen,aan de luyssen,aan de maas,aan de popelaar,aan de school,aan de 
steenoven,aan de vreenen,aan de vrenen,aan de wolfsberg,aan de zuwe,aan den 
nieuwen vijver,aan den steenoven,aan het broek,aan 
reijans,aana,aana-sira,aanaar,aanadane,aanadane ech 
chih,aanane,aananib,aanaye,aanaz,aanbal,aancud,aandal,aandalen,aandara,aandenk,aandepoppelaar,aandervaag,aandqet,aandu,aaneby,aanekoski,aaneland,aaneq
 al haoua,aaneq baiarra,aaneq el 
haoua,aanerne,aanes,aangel,aangenaam,aangewysd,aangstad,aanhimaki,aanhou,aanislag,aaniss,aanjar,aankibane,aankomst,aannabe,aannabiye,aannaqiyet
 miri,aannaqiyet 
tchiftlik,aannaya,aannaze,aannbal,aanndaqatt,aannerud,aannestad,aannjara,aannkaoui,aannkibane,aannouz,aanntariye,aannzaouiye,aanosusura,aanout,aanoute,aanqoun,aanschot,aanshu,aanstad,aansvik,aantaara,aantoura,aanum,aanwas,aaouachiye,aaouadi,aaouainat,aaouaini,aaouainiye,aaouaj,aaouamiye,aaouar,aaouara,aaouas,aaouass,aaoudet
 el qalamoun,aaoueichiye,aaoueidja,aaoueijet jine,aaoueiline,aaoueina,aaoueinat
 e,aaoueinate kebire,aaoueinate serhire,aaoueine,aaoueinet er 
rihane,aaoueini,aaoueira,aaoueira 
charqiye,aaoueisse,aaouenia,aaouifate,aaouint--maazouza,aaouj ej jenah,aaouj el 
jenah,aaouje,aaoukar,aaoumar ed 
dine,aaouniye,aaoura,aaousiene,aapaasdal,aapaasland,aapajarven 
asutusalue,aapajarvi,aapajoki,aapdraai,aappilattoq,aappilattorq,aapua,aaqabet 
hairouna,aaqareb es safi,aaqarib es safi,aaqb el ballane,aaqbet roummane,aaqbet 
sebaail,aaqeibe,aaqerbate,aaqerbe,aaqiba,aaqile,aaqlain,aaqliya,aaqliye,aaqmata,aaqoula,aaqoulet
 el 

[4/7] cassandra git commit: SASI tokenizer for simple delimiter based entries

[mck]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/4c6a10af/test/unit/org/apache/cassandra/index/sasi/analyzer/DelimiterAnalyzerTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/index/sasi/analyzer/DelimiterAnalyzerTest.java 
b/test/unit/org/apache/cassandra/index/sasi/analyzer/DelimiterAnalyzerTest.java
new file mode 100644
index 000..15bbb84
--- /dev/null
+++ 
b/test/unit/org/apache/cassandra/index/sasi/analyzer/DelimiterAnalyzerTest.java
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.index.sasi.analyzer;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+
+import org.apache.cassandra.db.marshal.Int32Type;
+import org.apache.cassandra.db.marshal.UTF8Type;
+import org.apache.cassandra.utils.ByteBufferUtil;
+import org.apache.commons.io.IOUtils;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+
+public class DelimiterAnalyzerTest
+{
+
+@Test
+public void caseSensitiveAnalizer() throws Exception
+{
+DelimiterAnalyzer analyzer = new DelimiterAnalyzer();
+
+analyzer.init(
+new HashMap()
+{{
+put(DelimiterTokenizingOptions.DELIMITER, " ");
+}},
+UTF8Type.instance);
+
+String testString = "Nip it in the bud";
+ByteBuffer toAnalyze = ByteBuffer.wrap(testString.getBytes());
+analyzer.reset(toAnalyze);
+StringBuilder output = new StringBuilder();
+while (analyzer.hasNext())
+output.append(ByteBufferUtil.string(analyzer.next()) + 
(analyzer.hasNext() ? ' ' : ""));
+
+Assert.assertTrue(testString.equals(output.toString()));
+Assert.assertFalse(testString.toLowerCase().equals(output.toString()));
+}
+
+@Test(expected = IllegalArgumentException.class)
+public void ensureIncompatibleInputSkipped() throws Exception
+{
+new DelimiterAnalyzer().init(new HashMap(), Int32Type.instance);
+}
+
+@Test
+public void testTokenizationLoremIpsum() throws Exception
+{
+ByteBuffer bb = ByteBuffer.wrap(IOUtils.toByteArray(
+
DelimiterAnalyzerTest.class.getClassLoader().getResourceAsStream("tokenization/lorem_ipsum.txt")));
+
+DelimiterAnalyzer tokenizer = new DelimiterAnalyzer();
+
+tokenizer.init(
+new HashMap()
+{{
+put(DelimiterTokenizingOptions.DELIMITER, " ");
+}},
+UTF8Type.instance);
+
+List tokens = new ArrayList<>();
+tokenizer.reset(bb);
+while (tokenizer.hasNext())
+tokens.add(tokenizer.next());
+
+assertEquals(69, tokens.size());
+
+}
+
+@Test
+public void testTokenizationJaJp1() throws Exception
+{
+ByteBuffer bb = ByteBuffer.wrap(IOUtils.toByteArray(
+
DelimiterAnalyzerTest.class.getClassLoader().getResourceAsStream("tokenization/ja_jp_1.txt")));
+
+DelimiterAnalyzer tokenizer = new DelimiterAnalyzer();
+
+tokenizer.init(
+new HashMap()
+{{
+put(DelimiterTokenizingOptions.DELIMITER, "。");
+}},
+UTF8Type.instance);
+
+tokenizer.reset(bb);
+List tokens = new ArrayList<>();
+while (tokenizer.hasNext())
+tokens.add(tokenizer.next());
+
+assertEquals(4, tokens.size());
+}
+
+@Test
+public void testTokenizationJaJp2() throws Exception
+{
+ByteBuffer bb = ByteBuffer.wrap(IOUtils.toByteArray(
+
DelimiterAnalyzerTest.class.getClassLoader().getResourceAsStream("tokenization/ja_jp_2.txt")));
+
+DelimiterAnalyzer tokenizer = new DelimiterAnalyzer();
+
+tokenizer.init(
+new HashMap()
+{{
+put(DelimiterTokenizingOptions.DELIMITER, "。");
+}},
+UTF8Type.instance);
+
+tokenizer.reset(bb);
+List tokens = new ArrayList<>();
+while 

[jira] [Comment Edited] (CASSANDRA-5836) Seed nodes should be able to bootstrap without manual intervention

[Oleksandr Shulgin (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-5836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400082#comment-16400082
 ] 

Oleksandr Shulgin edited comment on CASSANDRA-5836 at 3/15/18 8:35 AM:
---

{quote}system.available_ranges works off keyspaces, so rebuild will still work 
fine as long as you didn't add RF before provisioning the DC (e.g you didn't 
bootstrap the NTS keyspaces){quote}

You are correct, I had a false assumption here.  But then I don't see at all 
where does the recommendation to set {{auto_bootstrap=false}} for new DC come 
from?  I believed the reason was that {{nodetool rebuild}} won't work 
otherwise, but it's not the case apparently.

If we can simply drop this recommendation from the docs that would be a great 
thing, IMO.  By following the doc in its current form it is not unlikely that 
one can accidentally add some nodes with {{auto_bootstrap=false}} to *existing* 
DC, simply by messing up the DC suffix parameter.  With the default setting of 
{{auto_bootstrap} such a configuration error is mostly harmless and is easy to 
rollback.

{quote}> At the same time, new cluster startup process can be arbitrarily 
complex
No, it can't. {quote}

Unfortunately, it already is.  For example, look at our home-grown automation 
code to create new Cassandra clusters on AWS: 
https://github.com/zalando-stups/planb-cassandra/blob/master/planb/create_cluster.py
  That's already close to 1,000 lines of Python.

{quote}Cassandra is hard enough to use as it is, and we really shouldn't be 
making operations more complex.{quote}

Creating a new cluster is the operation with the least possible potential 
impact of all, and you do it only once in a lifetime of a cluster.  I would go 
as far as saying it doesn't even belong to "ops".  Restarts, upgrades, 
bootstrapping new nodes and DCs: these are the operations and we shouldn't make 
"introduction to Cassandra" easier at the cost of making *these* more complex 
or risky.

{quote}Far more logical to be able to say that "All nodes will respect the 
auto_bootstrap setting regardless of their configuration". The only caveat is 
that the first node won't bootstrap,..{quote}

That's already a contradiction, don't you think?  And more precisely it should 
be spelled as "if a node *believes* it is the very first one".  A big question 
to me still: can this be done in the code reliably?

{quote}... but to users this is irrelevant and they don't need to know about 
it.{quote}

This attitude is exactly what makes Cassandra hard to use in my experience. :(  
I cannot even count the number of times when I had to dive deeply into the 
source code trying to figure some detail which was not properly documented, 
because the devs thought the same: users don't need to know about it...



was (Author: oshulgin):
{quote}system.available_ranges works off keyspaces, so rebuild will still work 
fine as long as you didn't add RF before provisioning the DC (e.g you didn't 
bootstrap the NTS keyspaces){quote}

You are correct, I had a false assumption here.  But then I don't see at all 
where does the recommendation to set {{auto_bootstrap=false}} for new DC come 
from?  I believed the reason was that {{nodetool rebuild}} won't work 
otherwise, but it's not the case apparently.

If we can simply drop this recommendation from the docs that would be a great 
thing, IMO.  By following the doc in its current form it is not unlikely that 
one can accidentally add some nodes with {{auto_bootstrap=false}} to *existing* 
DC, simply by messing up the DC suffix parameter.  With the default setting of 
{{auto_bootstrap} such a configuration error is mostly harmless and is easy to 
rollback.

{quote}> At the same time, new cluster startup process can be arbitrarily 
complex
No, it can't. {quote}

Unfortunately, it already is.  For example, look at our home-grown automation 
code to create new Cassandra clusters on AWS: 
https://github.com/zalando-stups/planb-cassandra/blob/master/planb/create_cluster.py
  That's already close to 1,000 lines of Python.

{quote}Cassandra is hard enough to use as it is, and we really shouldn't be 
making operations more complex.{quote}

Creating a new cluster is the operation with the least possible potential 
impact of all, and you do it only once in a lifetime of a cluster.  I would go 
as far as saying it doesn't even belong to "ops".  Restarts, upgrades, 
bootstrapping new nodes and DCs: these are the operations and we shouldn't make 
"introduction to Cassandra" easier at the cost of making *these* more complex 
or risky.

{quote}Far more logical to be able to say that "All nodes will respect the 
auto_bootstrap setting regardless of their configuration". The only caveat is 
that the first node won't bootstrap,..{quote}

That's already a contradiction, don't you think?  And more precisely it should 
be spelled as "if a node *believes* it is the very 

[jira] [Commented] (CASSANDRA-5836) Seed nodes should be able to bootstrap without manual intervention

[Oleksandr Shulgin (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-5836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400082#comment-16400082
 ] 

Oleksandr Shulgin commented on CASSANDRA-5836:
--

{quote}system.available_ranges works off keyspaces, so rebuild will still work 
fine as long as you didn't add RF before provisioning the DC (e.g you didn't 
bootstrap the NTS keyspaces){quote}

You are correct, I had a false assumption here.  But then I don't see at all 
where does the recommendation to set {{auto_bootstrap=false}} for new DC come 
from?  I believed the reason was that {{nodetool rebuild}} won't work 
otherwise, but it's not the case apparently.

If we can simply drop this recommendation from the docs that would be a great 
thing, IMO.  By following the doc in its current form it is not unlikely that 
one can accidentally add some nodes with {{auto_bootstrap=false}} to *existing* 
DC, simply by messing up the DC suffix parameter.  With the default setting of 
{{auto_bootstrap} such a configuration error is mostly harmless and is easy to 
rollback.

{quote}> At the same time, new cluster startup process can be arbitrarily 
complex
No, it can't. {quote}

Unfortunately, it already is.  For example, look at our home-grown automation 
code to create new Cassandra clusters on AWS: 
https://github.com/zalando-stups/planb-cassandra/blob/master/planb/create_cluster.py
  That's already close to 1,000 lines of Python.

{quote}Cassandra is hard enough to use as it is, and we really shouldn't be 
making operations more complex.{quote}

Creating a new cluster is the operation with the least possible potential 
impact of all, and you do it only once in a lifetime of a cluster.  I would go 
as far as saying it doesn't even belong to "ops".  Restarts, upgrades, 
bootstrapping new nodes and DCs: these are the operations and we shouldn't make 
"introduction to Cassandra" easier at the cost of making *these* more complex 
or risky.

{quote}Far more logical to be able to say that "All nodes will respect the 
auto_bootstrap setting regardless of their configuration". The only caveat is 
that the first node won't bootstrap,..{quote}

That's already a contradiction, don't you think?  And more precisely it should 
be spelled as "if a node *believes* it is the very first one".  A big question 
to me still: can this be done reliably?

{quote}... but to users this is irrelevant and they don't need to know about 
it.{quote}

This attitude is exactly what makes Cassandra hard to use in my experience. :(  
I cannot even count the number of times when I had to dive deeply into the 
source code trying to figure some detail which was not properly documented, 
because the devs thought the same: users don't need to know about it...


> Seed nodes should be able to bootstrap without manual intervention
> --
>
> Key: CASSANDRA-5836
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5836
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Bill Hathaway
>Priority: Minor
>
> The current logic doesn't allow a seed node to be bootstrapped.  If a user 
> wants to bootstrap a node configured as a seed (for example to replace a seed 
> node via replace_token), they first need to remove the node's own IP from the 
> seed list, and then start the bootstrap process.  This seems like an 
> unnecessary step since a node never uses itself as a seed.
> I think it would be a better experience if the logic was changed to allow a 
> seed node to bootstrap without manual intervention when there are other seed 
> nodes up in a ring.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14315) ThrottledUnfilteredIterator failed on UnfilteredRowIterator with only partition level info

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400048#comment-16400048
 ] 

ZhaoYang commented on CASSANDRA-14315:
--

| [trunk|https://github.com/jasonstack/cassandra/commits/throttle-trunk]|
| [dtest|https://github.com/jasonstack/cassandra-dtest/commits/throttle-mv] |

Changes:
1. when batch size is 0, don't throttle and return itself. (opt out)
2. when the original UnfilteredRowIterator has only partition level info, 
return itself once.
3. removed the assertion "UnfilteredPartitionIterator should not contain empty 
partitions"

> ThrottledUnfilteredIterator failed on UnfilteredRowIterator with only 
> partition level info
> --
>
> Key: CASSANDRA-14315
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14315
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Major
> Fix For: 4.0
>
>
> When repairing base table with MV, in order to avoid OOM, Cassandra-13299 
> added ThrottledUnfilteredIterator to split large partition into small chunks, 
> but it didn't handle partition without unfiltered properly.
> {code:title=repro}
> // create cell tombstone, range tombstone, partition deletion
> createTable("CREATE TABLE %s (pk int, ck1 int, ck2 int, v1 int, v2 int, 
> PRIMARY KEY (pk, ck1, ck2))");
> // partition deletion
> execute("DELETE FROM %s USING TIMESTAMP 160 WHERE pk=1");
> // flush and generate 1 sstable
> ColumnFamilyStore cfs = 
> Keyspace.open(keyspace()).getColumnFamilyStore(currentTable());
> cfs.forceBlockingFlush();
> cfs.disableAutoCompaction();
> cfs.forceMajorCompaction();
> assertEquals(1, cfs.getLiveSSTables().size());
> SSTableReader reader = cfs.getLiveSSTables().iterator().next();
> try (ISSTableScanner scanner = reader.getScanner();
> CloseableIterator throttled = 
> ThrottledUnfilteredIterator.throttle(scanner, 100))
> {
> assertTrue(throttled.hasNext());
> UnfilteredRowIterator iterator = throttled.next();
> assertFalse(throttled.hasNext());
> assertFalse(iterator.hasNext());
> assertEquals(iterator.partitionLevelDeletion().markedForDeleteAt(), 160);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14315) ThrottledUnfilteredIterator failed on UnfilteredRowIterator with only partition level info

[ZhaoYang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhaoYang updated CASSANDRA-14315:
-
Environment: (was: | 
[trunk|https://github.com/jasonstack/cassandra/commits/throttle-trunk]|
| [dtest|https://github.com/jasonstack/cassandra-dtest/commits/throttle-mv] |

Changes:
1. when batch size is 0, don't throttle and return itself. (opt out)
2. when the original UnfilteredRowIterator has only partition level info, 
return itself once.
3. removed the assertion "UnfilteredPartitionIterator should not contain empty 
partitions")

> ThrottledUnfilteredIterator failed on UnfilteredRowIterator with only 
> partition level info
> --
>
> Key: CASSANDRA-14315
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14315
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Major
> Fix For: 4.0
>
>
> When repairing base table with MV, in order to avoid OOM, Cassandra-13299 
> added ThrottledUnfilteredIterator to split large partition into small chunks, 
> but it didn't handle partition without unfiltered properly.
> {code:title=repro}
> // create cell tombstone, range tombstone, partition deletion
> createTable("CREATE TABLE %s (pk int, ck1 int, ck2 int, v1 int, v2 int, 
> PRIMARY KEY (pk, ck1, ck2))");
> // partition deletion
> execute("DELETE FROM %s USING TIMESTAMP 160 WHERE pk=1");
> // flush and generate 1 sstable
> ColumnFamilyStore cfs = 
> Keyspace.open(keyspace()).getColumnFamilyStore(currentTable());
> cfs.forceBlockingFlush();
> cfs.disableAutoCompaction();
> cfs.forceMajorCompaction();
> assertEquals(1, cfs.getLiveSSTables().size());
> SSTableReader reader = cfs.getLiveSSTables().iterator().next();
> try (ISSTableScanner scanner = reader.getScanner();
> CloseableIterator throttled = 
> ThrottledUnfilteredIterator.throttle(scanner, 100))
> {
> assertTrue(throttled.hasNext());
> UnfilteredRowIterator iterator = throttled.next();
> assertFalse(throttled.hasNext());
> assertFalse(iterator.hasNext());
> assertEquals(iterator.partitionLevelDeletion().markedForDeleteAt(), 160);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14315) ThrottledUnfilteredIterator failed on UnfilteredRowIterator with only partition level info

[ZhaoYang (JIRA)]

ZhaoYang created CASSANDRA-14315:


 Summary: ThrottledUnfilteredIterator failed on 
UnfilteredRowIterator with only partition level info
 Key: CASSANDRA-14315
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14315
 Project: Cassandra
  Issue Type: Bug
  Components: Materialized Views
 Environment: | 
[trunk|https://github.com/jasonstack/cassandra/commits/throttle-trunk]|
| [dtest|https://github.com/jasonstack/cassandra-dtest/commits/throttle-mv] |

Changes:
1. when batch size is 0, don't throttle and return itself. (opt out)
2. when the original UnfilteredRowIterator has only partition level info, 
return itself once.
3. removed the assertion "UnfilteredPartitionIterator should not contain empty 
partitions"
Reporter: ZhaoYang
Assignee: ZhaoYang
 Fix For: 4.0


When repairing base table with MV, in order to avoid OOM, Cassandra-13299 added 
ThrottledUnfilteredIterator to split large partition into small chunks, but it 
didn't handle partition without unfiltered properly.

{code:title=repro}
// create cell tombstone, range tombstone, partition deletion
createTable("CREATE TABLE %s (pk int, ck1 int, ck2 int, v1 int, v2 int, PRIMARY 
KEY (pk, ck1, ck2))");
// partition deletion
execute("DELETE FROM %s USING TIMESTAMP 160 WHERE pk=1");

// flush and generate 1 sstable
ColumnFamilyStore cfs = 
Keyspace.open(keyspace()).getColumnFamilyStore(currentTable());
cfs.forceBlockingFlush();
cfs.disableAutoCompaction();
cfs.forceMajorCompaction();

assertEquals(1, cfs.getLiveSSTables().size());
SSTableReader reader = cfs.getLiveSSTables().iterator().next();

try (ISSTableScanner scanner = reader.getScanner();
CloseableIterator throttled = 
ThrottledUnfilteredIterator.throttle(scanner, 100))
{
assertTrue(throttled.hasNext());
UnfilteredRowIterator iterator = throttled.next();
assertFalse(throttled.hasNext());
assertFalse(iterator.hasNext());
assertEquals(iterator.partitionLevelDeletion().markedForDeleteAt(), 160);
}
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-9452) Remove configuration of storage-conf from tools

[Vinay Chella (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399987#comment-16399987
 ] 

Vinay Chella commented on CASSANDRA-9452:
-

[~jolynch]  [~jjirsa]  Cleaned up unused {{test/resources/functions}} folder. 
Updated patch is attached as CASSANDRA-14092-trunk_v1.txt

Here is the 
[circleci|https://circleci.com/gh/vinaykumarchella/cassandra/tree/CASSANDRA-9452]
 

> Remove configuration of storage-conf from tools
> ---
>
> Key: CASSANDRA-9452
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9452
> Project: Cassandra
>  Issue Type: Task
>  Components: Configuration, Testing, Tools
>Reporter: Mike Adamson
>Assignee: Vinay Chella
>Priority: Minor
>  Labels: lhf
> Fix For: 4.x
>
> Attachments: CASSANDRA-14092-trunk_v1.txt, CASSANDRA-9452-trunk.txt
>
>
> The following files still making reference to storage-config and/or 
> storage-conf.xml
> * ./build.xml
> * ./bin/nodetool
> * ./bin/sstablekeys
> * ./test/resources/functions/configure_cassandra.sh
> * ./test/resources/functions/install_cassandra.sh
> * ./tools/bin/json2sstable
> * ./tools/bin/sstable2json
> * ./tools/bin/sstablelevelreset



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-9452) Remove configuration of storage-conf from tools

[Vinay Chella (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinay Chella updated CASSANDRA-9452:

Attachment: CASSANDRA-14092-trunk_v1.txt

> Remove configuration of storage-conf from tools
> ---
>
> Key: CASSANDRA-9452
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9452
> Project: Cassandra
>  Issue Type: Task
>  Components: Configuration, Testing, Tools
>Reporter: Mike Adamson
>Assignee: Vinay Chella
>Priority: Minor
>  Labels: lhf
> Fix For: 4.x
>
> Attachments: CASSANDRA-14092-trunk_v1.txt, CASSANDRA-9452-trunk.txt
>
>
> The following files still making reference to storage-config and/or 
> storage-conf.xml
> * ./build.xml
> * ./bin/nodetool
> * ./bin/sstablekeys
> * ./test/resources/functions/configure_cassandra.sh
> * ./test/resources/functions/install_cassandra.sh
> * ./tools/bin/json2sstable
> * ./tools/bin/sstable2json
> * ./tools/bin/sstablelevelreset



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14247) SASI tokenizer for simple delimiter based entries

[Michael Kjellman (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399968#comment-16399968
 ] 

Michael Kjellman commented on CASSANDRA-14247:
--

docs look awesome!! ship it!

> SASI tokenizer for simple delimiter based entries
> -
>
> Key: CASSANDRA-14247
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14247
> Project: Cassandra
>  Issue Type: Improvement
>  Components: sasi
>Reporter: mck
>Assignee: mck
>Priority: Major
>  Labels: sasi
> Fix For: 4.0, 3.11.x
>
>
> Currently SASI offers only two tokenizer options:
>  - NonTokenizerAnalyser
>  - StandardAnalyzer
> The latter is built upon Snowball, powerful for human languages but overkill 
> for simple tokenization.
> A simple tokenizer is proposed here. The need for this arose as a workaround 
> of CASSANDRA-11182, and to avoid the disk usage explosion when having to 
> resort to {{CONTAINS}}. See https://github.com/openzipkin/zipkin/issues/1861
> Example use of this would be:
> {code}
> CREATE CUSTOM INDEX span_annotation_query_idx 
> ON zipkin2.span (annotation_query) USING 
> 'org.apache.cassandra.index.sasi.SASIIndex' 
> WITH OPTIONS = {
> 'analyzer_class': 
> 'org.apache.cassandra.index.sasi.analyzer.DelimiterAnalyzer', 
> 'delimiter': '░',
> 'case_sensitive': 'true', 
> 'mode': 'prefix', 
> 'analyzed': 'true'};
> {code}
> Original credit for this work goes to https://github.com/zuochangan



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14247) SASI tokenizer for simple delimiter based entries

[mck (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399964#comment-16399964
 ] 

mck commented on CASSANDRA-14247:
-

{quote}i'll add the relevant section to doc/SASI.md{quote}

Done. Added docs update to just {{trunk_14247}} branch.

> SASI tokenizer for simple delimiter based entries
> -
>
> Key: CASSANDRA-14247
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14247
> Project: Cassandra
>  Issue Type: Improvement
>  Components: sasi
>Reporter: mck
>Assignee: mck
>Priority: Major
>  Labels: sasi
> Fix For: 4.0, 3.11.x
>
>
> Currently SASI offers only two tokenizer options:
>  - NonTokenizerAnalyser
>  - StandardAnalyzer
> The latter is built upon Snowball, powerful for human languages but overkill 
> for simple tokenization.
> A simple tokenizer is proposed here. The need for this arose as a workaround 
> of CASSANDRA-11182, and to avoid the disk usage explosion when having to 
> resort to {{CONTAINS}}. See https://github.com/openzipkin/zipkin/issues/1861
> Example use of this would be:
> {code}
> CREATE CUSTOM INDEX span_annotation_query_idx 
> ON zipkin2.span (annotation_query) USING 
> 'org.apache.cassandra.index.sasi.SASIIndex' 
> WITH OPTIONS = {
> 'analyzer_class': 
> 'org.apache.cassandra.index.sasi.analyzer.DelimiterAnalyzer', 
> 'delimiter': '░',
> 'case_sensitive': 'true', 
> 'mode': 'prefix', 
> 'analyzed': 'true'};
> {code}
> Original credit for this work goes to https://github.com/zuochangan



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org