Updated Branches: refs/heads/trunk 7a6fbc1b2 -> 883c34b7c
Reenable ALTER TABLE DROP with new semantics patch by Aleksey Yeschenko; reviewed by Jonathan Ellis for CASSANDRA-3919 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/883c34b7 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/883c34b7 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/883c34b7 Branch: refs/heads/trunk Commit: 883c34b7c513eb36f2a45d591db706fa7e36e145 Parents: 7a6fbc1 Author: Aleksey Yeschenko <[email protected]> Authored: Wed Apr 17 17:30:38 2013 +0300 Committer: Aleksey Yeschenko <[email protected]> Committed: Wed Apr 17 17:30:38 2013 +0300 ---------------------------------------------------------------------- NEWS.txt | 3 + doc/cql3/CQL.textile | 8 ++- pylib/cqlshlib/helptopics.py | 16 ++++ .../org/apache/cassandra/config/CFMetaData.java | 56 +++++++++++++-- src/java/org/apache/cassandra/cql3/Cql.g | 2 +- .../org/apache/cassandra/cql3/QueryProcessor.java | 2 +- .../cql3/statements/AlterTableStatement.java | 3 + .../org/apache/cassandra/db/ColumnFamilyStore.java | 35 +++++++++- .../apache/cassandra/db/marshal/CompositeType.java | 7 ++ 9 files changed, 120 insertions(+), 12 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/cassandra/blob/883c34b7/NEWS.txt ---------------------------------------------------------------------- diff --git a/NEWS.txt b/NEWS.txt index 1c68de6..ea4ae1e 100644 --- a/NEWS.txt +++ b/NEWS.txt @@ -33,6 +33,9 @@ Operations - Disabling autocompactions by setting min/max compaction threshold to 0 has been deprecated, instead, use the nodetool commands 'disableautocompaction' and 'enableautocompaction' or set the compaction strategy option enabled = false + - ALTER TABLE DROP has been reenabled for CQL3 tables and has new semantics now. + See https://cassandra.apache.org/doc/cql3/CQL.html#alterTableStmt and + https://issues.apache.org/jira/browse/CASSANDRA-3919 for details. 1.2.4 http://git-wip-us.apache.org/repos/asf/cassandra/blob/883c34b7/doc/cql3/CQL.textile ---------------------------------------------------------------------- diff --git a/doc/cql3/CQL.textile b/doc/cql3/CQL.textile index 83ed82a..a079303 100644 --- a/doc/cql3/CQL.textile +++ b/doc/cql3/CQL.textile @@ -338,6 +338,7 @@ bc(syntax).. <instruction> ::= ALTER <identifier> TYPE <type> | ADD <identifier> <type> + | DROP <identifier> | WITH <option> ( AND <option> )* p. __Sample:__ @@ -358,10 +359,9 @@ The @ALTER@ statement is used to manipulate table definitions. It allows to add The @<tablename>@ is the table name optionally preceded by the keyspace name. The @<instruction>@ defines the alteration to perform: * @ALTER@: Update the type of a given defined column. Note that the type of the "clustering keys":#createTablepartitionClustering cannot be modified as it induces the on-disk ordering of rows. Columns on which a "secondary index":#createIndexStmt is defined have the same restriction. Other columns are free from those restrictions (no validation of existing data is performed), but it is usually a bad idea to change the type to a non-compatible one, unless no data have been inserted for that column yet, as this could confuse CQL drivers/tools. * @ADD@: Adds a new column to the table. The @<identifier>@ for the new column must not conflict with an existing column. Moreover, columns cannot be added to tables defined with the @COMPACT STORAGE@ option. +* @DROP@: Removes a column from the table. Dropped columns will immediately become unavailable in the queries and will not be included in compacted sstables in the future. If a column is readded, queries won't return values written before the column was last dropped. It is assumed that timestamps represent actual time, so if this is not your case, you should NOT readd previously dropped columns. Columns can't be dropped from tables defined with the @COMPACT STORAGE@ option. * @WITH@: Allows to update the options of the table. The "supported @<option>@":#createTableOptions (and syntax) are the same as for the @CREATE TABLE@ statement except that @COMPACT STORAGE@ is not supported. Note that setting any @compaction@ sub-options has the effect of erasing all previous @compaction@ options, so you need to re-specify all the sub-options if you want to keep them. The same note applies to the set of @compression@ sub-options. -Dropping a column is no yet supported but is on "the roadmap":https://issues.apache.org/jira/browse/CASSANDRA-3919. In the meantime, a declared but unused column has no impact on performance nor uses any storage. - h3(#dropTableStmt). DROP TABLE __Syntax:__ @@ -1045,6 +1045,10 @@ h2(#changes). Changes The following describes the addition/changes brought for each version of CQL. +h3. 3.1.0 + +* "ALTER TABLE":#alterTableStmt @DROP@ option has been reenabled for CQL3 tables and has new semantics now: the space formerly used by dropped columns will now be eventually reclaimed (post-compaction). You should not readd previously dropped columns unless you use timestamps with microsecond precision (see "CASSANDRA-3919":https://issues.apache.org/jira/browse/CASSANDRA-3919 for more details). + h3. 3.0.2 * Type validation for the "constants":#constants has been fixed. For instance, the implementation used to allow @'2'@ as a valid value for an @int@ column (interpreting it has the equivalent of @2@), or @42@ as a valid @blob@ value (in which case @42@ was interpreted as an hexadecimal representation of the blob). This is no longer the case, type validation of constants is now more strict. See the "data types":#types section for details on which constant is allowed for which type. http://git-wip-us.apache.org/repos/asf/cassandra/blob/883c34b7/pylib/cqlshlib/helptopics.py ---------------------------------------------------------------------- diff --git a/pylib/cqlshlib/helptopics.py b/pylib/cqlshlib/helptopics.py index 775c7de..a5b9e48 100644 --- a/pylib/cqlshlib/helptopics.py +++ b/pylib/cqlshlib/helptopics.py @@ -898,6 +898,22 @@ class CQL3HelpTopics(CQLHelpTopics): Currently, COUNT is the only function supported by CQL. """ + def help_alter_drop(self): + print """ + ALTER TABLE: dropping a typed column + + ALTER TABLE addamsFamily DROP gender; + + An ALTER TABLE ... DROP statement removes the type of a column + from the column family metadata. Dropped columns will immediately + become unavailable in the queries and will not be included in + compacted sstables in the future. If a column is readded, queries + won't return values written before the column was last dropped. + It is assumed that timestamps represent actual time, so if this + is not your case, you should NOT readd previously dropped columns. + Columns can't be dropped from tables defined with COMPACT STORAGE. + """ + def help_create(self): super(CQL3HelpTopics, self).help_create() print " HELP CREATE_USER;" http://git-wip-us.apache.org/repos/asf/cassandra/blob/883c34b7/src/java/org/apache/cassandra/config/CFMetaData.java ---------------------------------------------------------------------- diff --git a/src/java/org/apache/cassandra/config/CFMetaData.java b/src/java/org/apache/cassandra/config/CFMetaData.java index 31b16a0..2f27b87 100644 --- a/src/java/org/apache/cassandra/config/CFMetaData.java +++ b/src/java/org/apache/cassandra/config/CFMetaData.java @@ -36,6 +36,7 @@ import org.apache.commons.lang.builder.ToStringBuilder; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import org.apache.cassandra.cql3.ColumnNameBuilder; import org.apache.cassandra.cql3.CFDefinition; import org.apache.cassandra.cql3.QueryProcessor; import org.apache.cassandra.cql3.UntypedResultSet; @@ -115,6 +116,7 @@ public final class CFMetaData + "strategy_class text," + "strategy_options text" + ") WITH COMPACT STORAGE AND COMMENT='keyspace definitions' AND gc_grace_seconds=8640"); + public static final CFMetaData SchemaColumnFamiliesCf = compile(9, "CREATE TABLE " + SystemTable.SCHEMA_COLUMNFAMILIES_CF + "(" + "keyspace_name text," + "columnfamily_name text," @@ -146,8 +148,10 @@ public final class CFMetaData + "default_write_consistency text," + "speculative_retry text," + "populate_io_cache_on_flush boolean," + + "dropped_columns map<text, bigint>," + "PRIMARY KEY (keyspace_name, columnfamily_name)" + ") WITH COMMENT='ColumnFamily definitions' AND gc_grace_seconds=8640"); + public static final CFMetaData SchemaColumnsCf = compile(10, "CREATE TABLE " + SystemTable.SCHEMA_COLUMNS_CF + "(" + "keyspace_name text," + "columnfamily_name text," @@ -363,6 +367,7 @@ public final class CFMetaData private volatile int defaultTimeToLive = DEFAULT_DEFAULT_TIME_TO_LIVE; private volatile SpeculativeRetry speculativeRetry = DEFAULT_SPECULATIVE_RETRY; private volatile boolean populateIoCacheOnFlush = DEFAULT_POPULATE_IO_CACHE_ON_FLUSH; + private volatile Map<ByteBuffer, Long> droppedColumns = new HashMap<ByteBuffer, Long>(); /* * All CQL3 columns definition are stored in the column_metadata map. @@ -407,6 +412,7 @@ public final class CFMetaData public CFMetaData defaultTimeToLive(int prop) {defaultTimeToLive = prop; return this;} public CFMetaData speculativeRetry(SpeculativeRetry prop) {speculativeRetry = prop; return this;} public CFMetaData populateIoCacheOnFlush(boolean prop) {populateIoCacheOnFlush = prop; return this;} + public CFMetaData droppedColumns(Map<ByteBuffer, Long> cols) {droppedColumns = cols; return this;} public CFMetaData(String keyspace, String name, ColumnFamilyType type, AbstractType<?> comp, AbstractType<?> subcc) { @@ -468,11 +474,6 @@ public final class CFMetaData return UUID.nameUUIDFromBytes(ArrayUtils.addAll(ksName.getBytes(), cfName.getBytes())); } - private void init() - { - updateCfDef(); // init cqlCfDef - } - private static CFMetaData newSystemMetadata(String keyspace, String cfName, int oldCfId, String comment, AbstractType<?> comparator, AbstractType<?> subcc) { ColumnFamilyType type = subcc == null ? ColumnFamilyType.Standard : ColumnFamilyType.Super; @@ -555,7 +556,8 @@ public final class CFMetaData .indexInterval(oldCFMD.indexInterval) .speculativeRetry(oldCFMD.speculativeRetry) .memtableFlushPeriod(oldCFMD.memtableFlushPeriod) - .populateIoCacheOnFlush(oldCFMD.populateIoCacheOnFlush); + .populateIoCacheOnFlush(oldCFMD.populateIoCacheOnFlush) + .droppedColumns(oldCFMD.droppedColumns); } /** @@ -711,6 +713,11 @@ public final class CFMetaData return defaultTimeToLive; } + public Map<ByteBuffer, Long> getDroppedColumns() + { + return droppedColumns; + } + public boolean equals(Object obj) { if (obj == this) @@ -749,6 +756,7 @@ public final class CFMetaData .append(indexInterval, rhs.indexInterval) .append(speculativeRetry, rhs.speculativeRetry) .append(populateIoCacheOnFlush, rhs.populateIoCacheOnFlush) + .append(droppedColumns, rhs.droppedColumns) .isEquals(); } @@ -780,6 +788,7 @@ public final class CFMetaData .append(indexInterval) .append(speculativeRetry) .append(populateIoCacheOnFlush) + .append(droppedColumns) .toHashCode(); } @@ -950,6 +959,9 @@ public final class CFMetaData speculativeRetry = cfm.speculativeRetry; populateIoCacheOnFlush = cfm.populateIoCacheOnFlush; + if (!cfm.droppedColumns.isEmpty()) + droppedColumns = cfm.droppedColumns; + MapDifference<ByteBuffer, ColumnDefinition> columnDiff = Maps.difference(column_metadata, cfm.column_metadata); // columns that are no longer needed for (ColumnDefinition cd : columnDiff.entriesOnlyOnLeft().values()) @@ -1442,6 +1454,9 @@ public final class CFMetaData cf.addColumn(DeletedColumn.create(ldt, timestamp, cfName, "compaction_strategy_options")); cf.addColumn(DeletedColumn.create(ldt, timestamp, cfName, "index_interval")); + for (Map.Entry<ByteBuffer, Long> entry : droppedColumns.entrySet()) + cf.addColumn(new DeletedColumn(makeDroppedColumnName(entry.getKey()), ldt, timestamp)); + for (ColumnDefinition cd : column_metadata.values()) cd.deleteFromSchema(rm, cfName, getColumnDefinitionComparator(cd), timestamp); @@ -1506,6 +1521,9 @@ public final class CFMetaData cf.addColumn(Column.create(indexInterval, timestamp, cfName, "index_interval")); cf.addColumn(Column.create(speculativeRetry.toString(), timestamp, cfName, "speculative_retry")); + for (Map.Entry<ByteBuffer, Long> entry : droppedColumns.entrySet()) + cf.addColumn(new Column(makeDroppedColumnName(entry.getKey()), LongType.instance.decompose(entry.getValue()), timestamp)); + // Save the CQL3 metadata "the old way" for compatibility sake cf.addColumn(Column.create(aliasesToJson(partitionKeyColumns), timestamp, cfName, "key_aliases")); cf.addColumn(Column.create(aliasesToJson(clusteringKeyColumns), timestamp, cfName, "column_aliases")); @@ -1574,6 +1592,9 @@ public final class CFMetaData if (result.has("value_alias")) cfm.addColumnMetadataFromAliases(Collections.<ByteBuffer>singletonList(result.getBytes("value_alias")), cfm.defaultValidator, ColumnDefinition.Type.COMPACT_VALUE); + if (result.has("dropped_columns")) + cfm.droppedColumns(convertDroppedColumns(result.getMap("dropped_columns", UTF8Type.instance, LongType.instance))); + return cfm; } catch (SyntaxException e) @@ -1641,6 +1662,22 @@ public final class CFMetaData return rawAliases; } + private static Map<ByteBuffer, Long> convertDroppedColumns(Map<String, Long> raw) + { + Map<ByteBuffer, Long> converted = Maps.newHashMap(); + for (Map.Entry<String, Long> entry : raw.entrySet()) + converted.put(UTF8Type.instance.decompose(entry.getKey()), entry.getValue()); + return converted; + } + + private ByteBuffer makeDroppedColumnName(ByteBuffer column) + { + ColumnNameBuilder builder = SchemaColumnFamiliesCf.cqlCfDef.getColumnNameBuilder(); + builder.add(UTF8Type.instance.decompose(cfName)); + builder.add(UTF8Type.instance.decompose("dropped_columns")); + return builder.add(column).build(); + } + /** * Convert current metadata into schema mutation * @@ -1721,6 +1758,12 @@ public final class CFMetaData return removed; } + public void recordColumnDrop(ColumnDefinition def) + { + assert def.componentIndex != null; + droppedColumns.put(def.name, FBUtilities.timestampMicros()); + } + public void renameColumn(ByteBuffer from, String strFrom, ByteBuffer to, String strTo) throws InvalidRequestException { ColumnDefinition def = column_metadata.get(from); @@ -1950,6 +1993,7 @@ public final class CFMetaData .append("speculative_retry", speculativeRetry) .append("indexInterval", indexInterval) .append("populateIoCacheOnFlush", populateIoCacheOnFlush) + .append("droppedColumns", droppedColumns) .toString(); } } http://git-wip-us.apache.org/repos/asf/cassandra/blob/883c34b7/src/java/org/apache/cassandra/cql3/Cql.g ---------------------------------------------------------------------- diff --git a/src/java/org/apache/cassandra/cql3/Cql.g b/src/java/org/apache/cassandra/cql3/Cql.g index a91e529..774a0e8 100644 --- a/src/java/org/apache/cassandra/cql3/Cql.g +++ b/src/java/org/apache/cassandra/cql3/Cql.g @@ -484,7 +484,7 @@ alterTableStatement returns [AlterTableStatement expr] : K_ALTER K_COLUMNFAMILY cf=columnFamilyName ( K_ALTER id=cident K_TYPE v=comparatorType { type = AlterTableStatement.Type.ALTER; } | K_ADD id=cident v=comparatorType { type = AlterTableStatement.Type.ADD; } - // | K_DROP id=cident { type = AlterTableStatement.Type.DROP; } + | K_DROP id=cident { type = AlterTableStatement.Type.DROP; } | K_WITH properties[props] { type = AlterTableStatement.Type.OPTS; } | K_RENAME { type = AlterTableStatement.Type.RENAME; } id1=cident K_TO toId1=cident { renames.put(id1, toId1); } http://git-wip-us.apache.org/repos/asf/cassandra/blob/883c34b7/src/java/org/apache/cassandra/cql3/QueryProcessor.java ---------------------------------------------------------------------- diff --git a/src/java/org/apache/cassandra/cql3/QueryProcessor.java b/src/java/org/apache/cassandra/cql3/QueryProcessor.java index e6c4589..61b0b50 100644 --- a/src/java/org/apache/cassandra/cql3/QueryProcessor.java +++ b/src/java/org/apache/cassandra/cql3/QueryProcessor.java @@ -40,7 +40,7 @@ import org.apache.cassandra.utils.SemanticVersion; public class QueryProcessor { - public static final SemanticVersion CQL_VERSION = new SemanticVersion("3.0.1"); + public static final SemanticVersion CQL_VERSION = new SemanticVersion("3.1.0"); private static final Logger logger = LoggerFactory.getLogger(QueryProcessor.class); http://git-wip-us.apache.org/repos/asf/cassandra/blob/883c34b7/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java ---------------------------------------------------------------------- diff --git a/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java b/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java index ef15691..945d202 100644 --- a/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java +++ b/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java @@ -155,6 +155,8 @@ public class AlterTableStatement extends SchemaAlteringStatement case DROP: if (cfDef.isCompact) throw new InvalidRequestException("Cannot drop columns from a compact CF"); + if (!cfDef.isComposite) + throw new InvalidRequestException("Cannot drop columns from a non-CQL3 CF"); if (name == null) throw new InvalidRequestException(String.format("Column %s was not found in table %s", columnName, columnFamily())); @@ -172,6 +174,7 @@ public class AlterTableStatement extends SchemaAlteringStatement } assert toDelete != null; cfm.removeColumnDefinition(toDelete); + cfm.recordColumnDrop(toDelete); break; } break; http://git-wip-us.apache.org/repos/asf/cassandra/blob/883c34b7/src/java/org/apache/cassandra/db/ColumnFamilyStore.java ---------------------------------------------------------------------- diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java index 3d930bd..8903a46 100644 --- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java +++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java @@ -57,6 +57,7 @@ import org.apache.cassandra.db.filter.QueryFilter; import org.apache.cassandra.db.index.SecondaryIndex; import org.apache.cassandra.db.index.SecondaryIndexManager; import org.apache.cassandra.db.marshal.AbstractType; +import org.apache.cassandra.db.marshal.CompositeType; import org.apache.cassandra.dht.*; import org.apache.cassandra.dht.Range; import org.apache.cassandra.exceptions.ConfigurationException; @@ -910,7 +911,8 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean // remove columns if // (a) the column itself is gcable or // (b) the column is shadowed by a CF tombstone - if (c.getLocalDeletionTime() < gcBefore || cf.deletionInfo().isDeleted(c)) + // (c) the column has been dropped from the CF schema (CQL3 tables only) + if (c.getLocalDeletionTime() < gcBefore || cf.deletionInfo().isDeleted(c) || isDroppedColumn(c, cf.metadata())) { iter.remove(); indexer.remove(c); @@ -923,6 +925,28 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean removeDeletedColumnsOnly(cf, gcBefore, SecondaryIndexManager.nullUpdater); } + // returns true if + // 1. this column has been dropped from schema and + // 2. if it has been re-added since then, this particular column was inserted before the last drop + private static boolean isDroppedColumn(Column c, CFMetaData meta) + { + if (meta.getDroppedColumns().isEmpty()) + return false; + Long droppedAt = meta.getDroppedColumns().get(((CompositeType) meta.comparator).extractLastComponent(c.name())); + return droppedAt != null && c.timestamp() <= droppedAt; + } + + private void removeDroppedColumns(ColumnFamily cf) + { + if (cf == null) + return; + + Iterator<Column> iter = cf.iterator(); + while (iter.hasNext()) + if (isDroppedColumn(iter.next(), metadata)) + iter.remove(); + } + /** * @param sstables * @return sstables whose key range overlaps with that of the given sstables, not including itself. @@ -1287,8 +1311,9 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean return null; result = removeDeletedCF(cf, gcBefore); - } + + removeDroppedColumns(result); } finally { @@ -1540,6 +1565,8 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean data.addAll(cf, HeapAllocator.instance); } + removeDroppedColumns(data); + if (!filter.isSatisfiedBy(rawRow.key.key, data, null)) continue; @@ -1547,6 +1574,10 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean // cut the resultset back to what was requested, if necessary data = filter.prune(data); } + else + { + removeDroppedColumns(data); + } rows.add(new Row(rawRow.key, data)); matched++; http://git-wip-us.apache.org/repos/asf/cassandra/blob/883c34b7/src/java/org/apache/cassandra/db/marshal/CompositeType.java ---------------------------------------------------------------------- diff --git a/src/java/org/apache/cassandra/db/marshal/CompositeType.java b/src/java/org/apache/cassandra/db/marshal/CompositeType.java index 18d1dae..7333e0e 100644 --- a/src/java/org/apache/cassandra/db/marshal/CompositeType.java +++ b/src/java/org/apache/cassandra/db/marshal/CompositeType.java @@ -146,6 +146,13 @@ public class CompositeType extends AbstractCompositeType return null; } + // Extract CQL3 column name from the full column name. + public ByteBuffer extractLastComponent(ByteBuffer bb) + { + int idx = types.get(types.size() - 1) instanceof ColumnToCollectionType ? types.size() - 2 : types.size() - 1; + return extractComponent(bb, idx); + } + @Override public int componentsCount() {
