Expiring Columns Callback
Hi all,
We have a need to be able to notify downstream systems when certain columns
have expired. Given the way Cassandra does distributed deletes and
tombstones it seems like compaction time is only time that this is fairly
predictable but the notifications will 'lag' a bit from when the columns
have actually expired.
Attached is a patch (against a 1.1.x branch) with what I am currently
working with. I am in the middle of getting a decent unit test together.
I wonder if this is even a sound approach, however. So, given the attached
patch I have a few questions:
1. What would be the best place to put a callback mechanism for column
notifications (given the context of a row key as well) where these expired
columns would be known about as soon as possible?
2. What about the "at compaction time" approach I have attached? I
currently have the mechanism plugged into the "PrecompactedRow.java" class
that will notify a custom compaction strategy if so desired. As I
mentioned I am still working out a suitable unit test and am still
struggling with the flushes and timing involved (and perhaps caches?).
I'm grateful for any guidance you might have. Better places to hook this in
for compaction? A more sound approach for earlier notifications? etc...
Regards,
Greg
From d0ad33821676b3895af5973aae2426bc53c0b25f Mon Sep 17 00:00:00 2001
From: Greg Cooper
Date: Fri, 25 Oct 2013 14:25:36 -0700
Subject: [PATCH] Hook into the system for notifications when columns are
removed during compaction
---
.../db/compaction/IColumnRemovalListener.java | 43 +
.../cassandra/db/compaction/PrecompactedRow.java | 42 ++--
.../RemovalAwareLeveledCompactionStrategy.java | 51
3 files changed, 131 insertions(+), 5 deletions(-)
create mode 100644 src/java/org/apache/cassandra/db/compaction/IColumnRemovalListener.java
create mode 100644 src/java/org/apache/cassandra/db/compaction/RemovalAwareLeveledCompactionStrategy.java
diff --git a/src/java/org/apache/cassandra/db/compaction/IColumnRemovalListener.java b/src/java/org/apache/cassandra/db/compaction/IColumnRemovalListener.java
new file mode 100644
index 000..81fb350
--- /dev/null
+++ b/src/java/org/apache/cassandra/db/compaction/IColumnRemovalListener.java
@@ -0,0 +1,43 @@
+package org.apache.cassandra.db.compaction;
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+
+import org.apache.cassandra.db.DecoratedKey;
+import org.apache.cassandra.db.IColumn;
+
+import java.util.Set;
+
+/**
+ * Implementations of this hear about when columns go away from a node during compaction. It is sometimes
+ * useful to know when data is no longer managed by Cassandra for external systems.
+ * Created by gcooper on 10/25/13.
+ */
+public interface IColumnRemovalListener {
+/**
+ * This method will be called when columns are compacted from a row. The implementation of this is a custom
+ * CompactionStrategy class that is provided at runtime.
+ *
+ * @param key The key for the row where the columns were removed. (Helpful for context)
+ * @param removedCols Columns that went away during compaction.
+ */
+void colsRemoved(DecoratedKey key, Set removedCols);
+}
diff --git a/src/java/org/apache/cassandra/db/compaction/PrecompactedRow.java b/src/java/org/apache/cassandra/db/compaction/PrecompactedRow.java
index 8ed21ca..e69c8a1 100644
--- a/src/java/org/apache/cassandra/db/compaction/PrecompactedRow.java
+++ b/src/java/org/apache/cassandra/db/compaction/PrecompactedRow.java
@@ -24,17 +24,16 @@ package org.apache.cassandra.db.compaction;
import java.io.DataOutput;
import java.io.IOError;
import java.io.IOException;
+import java.nio.ByteBuffer;
import java.security.MessageDigest;
+import java.util.HashSet;
import java.util.List;
+import java.util.Set;
+import org.apache.cassandra.db.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import org.apache.cassandra.db.ColumnFamily;
-import org.apache.cassandra.db.ColumnFamilyStore;
-import org.apache.cassandra.db.ColumnIndexer;
-import org.apache.cassandra.db.CounterColumn;
-import org.apache.cassandra.db.DecoratedKey;
import org.apache.cassandra.io.sstable.SSTa
Re: Authentication in cassandra binary protocol v2
Only AUTH_SUCCESS will be sent on correct authentication. The fact READY mention CREDENTIALS is indeed a left-over of the v1 doc. I'll fix the spec ant try to clarify this a bit somehow. -- Sylvain On Mon, Oct 28, 2013 at 8:06 PM, Mathieu D'Amours wrote: > I think figured it out wrong initially. I thought AUTH_CHALLENGE was the > message the server sends right after STARTUP. If I understand correctly > a server configured with the PasswordAuthenticator is going to expect this > flow: > > C -> [STARTUP] > S -> [AUTHENTICATE] "PasswordAuthenticator" > C -> [AUTH_RESPONSE] "usernamepassword" > > Given correct credentials, is C* going to send both of these message one > after the other? > > S -> [AUTH_SUCCESS] > S -> [READY] > > The documentation about READY seem to contain artifacts from v1 (the > CREDENTIALS message): > > > Indicates that the server is ready to process queries. This message will > be > > sent by the server either after a STARTUP message if no authentication is > > required, or after a successful CREDENTIALS message. > > > Thank again, > > Le Oct 28, 2013 à 2:48 PM, Sylvain Lebresne a > écrit : > > > What information are you looking for? As the comment says, the details > are > > authenticaticator specific. So you were right to look into > > PasswordAuthenticator in particular, and to be more precise you'll want > to > > look at > PasswordAuthenticator.PlainTextSaslAuthenticator.evaluateResponse() > > for that that specific authenticator expect (basically the username and > > password as UTF8). > > > > -- > > Sylvain > > > > > > On Mon, Oct 28, 2013 at 7:15 PM, Mathieu D'Amours >wrote: > > > >> Hello, > >> > >> I stumbled upon this description in the binary protocol specs [4.2.7. > >> AUTH_CHALLENGE]: > >> > >>> The body of this message is a single [bytes] token. The details of what > >> this > >>> token contains (and when it can be null/empty, if ever) depends on the > >> actual > >>> authenticator used. > >> > >> > >> I looked in C* builtin authenticator classes, `AllAllowAuthenticator` > and > >> `PasswordAuthenticator`, but couldn't find this sort of > >> information. Could someone point me in the right direction? > >> > >> Thanks in advance, > >> > >> Mathieu > >
Re: Implementing a driver for cassandra native protocol v2
Also, you might see if you could join efforts with Aleksey on https://github.com/iamaleksey/seestar as well :) Or at least exchange ideas. On 28 Oct 2013, at 14:17, Sylvain Lebresne wrote: > On Sat, Oct 26, 2013 at 8:07 PM, Mathieu D'Amours wrote: > >> Hello, >> >> I'm currently implementing an Erlang driver for Cassandra's binary >> protocol v2. Most of it is straightforward, but I came across this part in >> [4.1.4. QUERY]: >> >>> is a [byte] whose bits define the options for this query and >>>in particular influence what the remainder of the message contains. >>>A flag is set if the bit corresponding to its `mask` is set. >> Supported >>>flags are, given there mask: >>> 0x01 : ... >>> 0x02 : ... >>> 0x03 : ... >>> 0x04 : ... >> >> Does that mean that the 0x01 flag is set if `1 << 0` is set? and 0x03 is >> set if `1 << 2` is set? >> > > This is what it means, but really that's a typo. The "mask" values should > read 0x01, 0x02, 0x04, 0x08 and 0x10. I'll fix the spec. > > -- > Sylvain > > >> >> Thanks in advance >> >> Mathieu
