yifan-c commented on code in PR #4458:
URL: https://github.com/apache/cassandra/pull/4458#discussion_r2487296008
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -49,6 +49,20 @@ public interface CompressionDictionary extends AutoCloseable
*/
byte[] rawDictionary();
+ /**
+ * Get checkum of this dictionary.
Review Comment:
typo:
```suggestion
* Get checksum of this dictionary.
```
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -281,4 +315,60 @@ public String toString()
'}';
}
}
+
+ /**
+ * The purpose of lightweight dictionary is to not carry the actual
dictionary bytes for performance reasons.
+ * Handy for situations when retrieval from the database does not need to
contain dictionary
+ * or the instatiation of a proper dictionary object is not desirable or
unnecessary for other,
+ * mostly performance-related, reasons.
+ */
+ class LightweightCompressionDictionary implements CompressionDictionary
+ {
+ private final DictId dictId;
+ private final int checksum;
+ private final int size;
+
+ public LightweightCompressionDictionary(DictId dictId, int checksum,
int size)
+ {
+ this.dictId = dictId;
+ this.checksum = checksum;
+ this.size = size;
+ }
+
+ @Override
+ public DictId dictId()
+ {
+ return dictId;
+ }
+
+ @Override
+ public byte[] rawDictionary()
+ {
+ return null;
Review Comment:
In my opinion, returning null loosen the API contract by permitting `null`
return value. Caller has to consider the null case. How about throwing
`UnsupportedOperationException` instead? So the API remains return non-null
values only.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryDetailsTabularData.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Arrays;
+import javax.management.openmbean.ArrayType;
+import javax.management.openmbean.CompositeData;
+import javax.management.openmbean.CompositeDataSupport;
+import javax.management.openmbean.CompositeType;
+import javax.management.openmbean.OpenDataException;
+import javax.management.openmbean.OpenType;
+import javax.management.openmbean.SimpleType;
+import javax.management.openmbean.TabularDataSupport;
+import javax.management.openmbean.TabularType;
+
+import static java.lang.String.format;
+
+public class CompressionDictionaryDetailsTabularData
+{
+ public static final String KEYSPACE_NAME = "Keyspace";
+ public static final String TABLE_NAME = "Table";
+ public static final String DICT_ID_NAME = "DictId";
+ public static final String DICT_NAME = "Dict";
+ public static final String KIND_NAME = "Kind";
+ public static final String CHECKSUM_NAME = "Checksum";
+ public static final String SIZE_NAME = "Size";
+
+
+ private static final String[] ITEM_NAMES = new String[]{ KEYSPACE_NAME,
+ TABLE_NAME,
+ DICT_ID_NAME,
+ DICT_NAME,
+ KIND_NAME,
+ CHECKSUM_NAME,
+ SIZE_NAME };
+
+ private static final String[] ITEM_DESCS = new String[]{ "keyspace",
+ "table",
+ "dictionary_id",
+
"dictionary_bytes",
+ "kind",
+ "checksum",
+ "size" };
+
+ private static final String TYPE_NAME = "DictionaryDetails";
+ private static final String ROW_DESC = "DictionaryDetails";
+ private static final OpenType<?>[] ITEM_TYPES;
+ private static final CompositeType COMPOSITE_TYPE;
+ public static final TabularType TABULAR_TYPE;
+
+ static
+ {
+ try
+ {
+ ITEM_TYPES = new OpenType[]{ SimpleType.STRING, // keyspace
+ SimpleType.STRING, // table
+ SimpleType.LONG, // dict id
+ new
ArrayType<String[]>(SimpleType.BYTE, true), // dict bytes
+ SimpleType.STRING, // kind
+ SimpleType.INTEGER, // checksum
+ SimpleType.INTEGER }; // size of dict
bytes
+
+ COMPOSITE_TYPE = new CompositeType(TYPE_NAME, ROW_DESC,
ITEM_NAMES, ITEM_DESCS, ITEM_TYPES);
+ TABULAR_TYPE = new TabularType(TYPE_NAME, ROW_DESC,
COMPOSITE_TYPE, ITEM_NAMES);
+ }
+ catch (OpenDataException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static void from(String keyspace,
+ String table,
+ CompressionDictionary dictionary,
+ TabularDataSupport result)
+ {
+ result.put(from(keyspace, table, dictionary));
+ }
+
+ public static CompositeData from(String keyspace, String table,
CompressionDictionary dictionary)
+ {
+ try
+ {
+ return new CompositeDataSupport(COMPOSITE_TYPE,
+ ITEM_NAMES,
+ new Object[]
+ {
+ keyspace,
+ table,
+ dictionary.dictId().id,
+ dictionary.rawDictionary(),
+ dictionary.kind().name(),
+ dictionary.checksum(),
+ dictionary.size(),
+ });
+ }
+ catch (OpenDataException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static CompositeData from(CompressionDictionaryPojo pojo)
+ {
+ try
+ {
+ return new CompositeDataSupport(COMPOSITE_TYPE,
+ ITEM_NAMES,
+ new Object[]
+ {
+ pojo.keyspace,
+ pojo.table,
+ pojo.dictId,
+ pojo.dict,
+ pojo.kind,
+ pojo.dictChecksum,
+ pojo.dictLength
+ });
+ }
+ catch (OpenDataException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /**
+ * Deserializes data to convenience object to work further with.
+ *
+ * @param compositeData data to create pojo from
+ * @return deserialized composite data to convenience object
+ * @throws IllegalArgumentException if values in deserialized object are
invalid.
+ * @see CompressionDictionaryPojo#validate()
+ */
+ public static CompressionDictionaryPojo from(CompositeData compositeData)
+ {
+ String keyspace = (String)
compositeData.get(CompressionDictionaryDetailsTabularData.KEYSPACE_NAME);
+ String table = (String)
compositeData.get(CompressionDictionaryDetailsTabularData.TABLE_NAME);
+ long dictId = (Long)
compositeData.get(CompressionDictionaryDetailsTabularData.DICT_ID_NAME);
+ byte[] dictionaryBytes = (byte[])
compositeData.get(CompressionDictionaryDetailsTabularData.DICT_NAME);
+ String kind = (String)
compositeData.get(CompressionDictionaryDetailsTabularData.KIND_NAME);
+ int checksum = (Integer)
compositeData.get(CompressionDictionaryDetailsTabularData.CHECKSUM_NAME);
+ int size = (Integer)
compositeData.get(CompressionDictionaryDetailsTabularData.SIZE_NAME);
+
+ CompressionDictionaryPojo pojo = new CompressionDictionaryPojo();
+ pojo.keyspace = keyspace;
+ pojo.table = table;
+ pojo.dictId = dictId;
+ pojo.dict = dictionaryBytes;
+ pojo.kind = kind;
+ pojo.dictChecksum = checksum;
+ pojo.dictLength = size;
+
+ pojo.validate();
+
+ return pojo;
+ }
+
+ public static class CompressionDictionaryPojo
+ {
+ public String keyspace;
+ public String table;
+ public long dictId;
+ public byte[] dict;
+ public String kind;
+ public int dictChecksum;
+ public int dictLength;
+
+ /**
+ * Dictionary is valid if, keyspace and table are specified,
dictionary id is strictly positive integer,
+ * dictionary byte array is not nor not empty,
Review Comment:
not sure what does "dictionary byte array is not nor not empty" mean.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -244,6 +251,68 @@ public CompositeData getTrainingState()
return dictionaryTrainer.getTrainingState().toCompositeData();
}
+ @Override
+ public TabularData listDictionaries(String keyspace, String table)
+ {
+ List<LightweightCompressionDictionary> dictionaries =
SystemDistributedKeyspace.retrieveLightweightCompressionDictionaries(keyspace,
table);
+ TabularDataSupport tableData = new
TabularDataSupport(CompressionDictionaryDetailsTabularData.TABULAR_TYPE);
+
+ if (dictionaries == null)
+ {
+ return tableData;
+ }
+
+ for (LightweightCompressionDictionary dictionary : dictionaries)
+ {
+ CompressionDictionaryDetailsTabularData.from(keyspace, table,
dictionary, tableData);
+ }
+
+ return tableData;
+ }
+
+ @Override
+ public CompositeData getDictionary(String keyspace, String table)
+ {
+ CompressionDictionary compressionDictionary =
SystemDistributedKeyspace.retrieveLatestCompressionDictionary(keyspaceName,
tableName);
+ if (compressionDictionary == null)
+ return null;
+
+ return CompressionDictionaryDetailsTabularData.from(keyspace, table,
compressionDictionary);
+ }
+
+ @Override
+ public CompositeData getDictionary(String keyspace, String table, long
dictId)
+ {
+ CompressionDictionary compressionDictionary =
SystemDistributedKeyspace.retrieveCompressionDictionary(keyspaceName,
tableName, dictId);
+ if (compressionDictionary == null)
+ return null;
+
+ return CompressionDictionaryDetailsTabularData.from(keyspace, table,
compressionDictionary);
+ }
+
+ @Override
+ public void importDictionary(CompositeData compositeData)
Review Comment:
Should this method be `synchornized`? We would like to avoid importing
concurrently.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryDetailsTabularData.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Arrays;
+import javax.management.openmbean.ArrayType;
+import javax.management.openmbean.CompositeData;
+import javax.management.openmbean.CompositeDataSupport;
+import javax.management.openmbean.CompositeType;
+import javax.management.openmbean.OpenDataException;
+import javax.management.openmbean.OpenType;
+import javax.management.openmbean.SimpleType;
+import javax.management.openmbean.TabularDataSupport;
+import javax.management.openmbean.TabularType;
+
+import static java.lang.String.format;
+
+public class CompressionDictionaryDetailsTabularData
+{
+ public static final String KEYSPACE_NAME = "Keyspace";
+ public static final String TABLE_NAME = "Table";
+ public static final String DICT_ID_NAME = "DictId";
+ public static final String DICT_NAME = "Dict";
+ public static final String KIND_NAME = "Kind";
+ public static final String CHECKSUM_NAME = "Checksum";
+ public static final String SIZE_NAME = "Size";
+
+
+ private static final String[] ITEM_NAMES = new String[]{ KEYSPACE_NAME,
+ TABLE_NAME,
+ DICT_ID_NAME,
+ DICT_NAME,
+ KIND_NAME,
+ CHECKSUM_NAME,
+ SIZE_NAME };
+
+ private static final String[] ITEM_DESCS = new String[]{ "keyspace",
+ "table",
+ "dictionary_id",
+
"dictionary_bytes",
+ "kind",
+ "checksum",
+ "size" };
+
+ private static final String TYPE_NAME = "DictionaryDetails";
+ private static final String ROW_DESC = "DictionaryDetails";
+ private static final OpenType<?>[] ITEM_TYPES;
+ private static final CompositeType COMPOSITE_TYPE;
+ public static final TabularType TABULAR_TYPE;
+
+ static
+ {
+ try
+ {
+ ITEM_TYPES = new OpenType[]{ SimpleType.STRING, // keyspace
+ SimpleType.STRING, // table
+ SimpleType.LONG, // dict id
+ new
ArrayType<String[]>(SimpleType.BYTE, true), // dict bytes
+ SimpleType.STRING, // kind
+ SimpleType.INTEGER, // checksum
+ SimpleType.INTEGER }; // size of dict
bytes
+
+ COMPOSITE_TYPE = new CompositeType(TYPE_NAME, ROW_DESC,
ITEM_NAMES, ITEM_DESCS, ITEM_TYPES);
+ TABULAR_TYPE = new TabularType(TYPE_NAME, ROW_DESC,
COMPOSITE_TYPE, ITEM_NAMES);
+ }
+ catch (OpenDataException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static void from(String keyspace,
+ String table,
+ CompressionDictionary dictionary,
+ TabularDataSupport result)
+ {
+ result.put(from(keyspace, table, dictionary));
+ }
+
+ public static CompositeData from(String keyspace, String table,
CompressionDictionary dictionary)
+ {
+ try
+ {
+ return new CompositeDataSupport(COMPOSITE_TYPE,
+ ITEM_NAMES,
+ new Object[]
+ {
+ keyspace,
+ table,
+ dictionary.dictId().id,
+ dictionary.rawDictionary(),
+ dictionary.kind().name(),
+ dictionary.checksum(),
+ dictionary.size(),
+ });
+ }
+ catch (OpenDataException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static CompositeData from(CompressionDictionaryPojo pojo)
+ {
+ try
+ {
+ return new CompositeDataSupport(COMPOSITE_TYPE,
+ ITEM_NAMES,
+ new Object[]
+ {
+ pojo.keyspace,
+ pojo.table,
+ pojo.dictId,
+ pojo.dict,
+ pojo.kind,
+ pojo.dictChecksum,
+ pojo.dictLength
+ });
+ }
+ catch (OpenDataException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /**
+ * Deserializes data to convenience object to work further with.
+ *
+ * @param compositeData data to create pojo from
+ * @return deserialized composite data to convenience object
+ * @throws IllegalArgumentException if values in deserialized object are
invalid.
+ * @see CompressionDictionaryPojo#validate()
+ */
+ public static CompressionDictionaryPojo from(CompositeData compositeData)
+ {
+ String keyspace = (String)
compositeData.get(CompressionDictionaryDetailsTabularData.KEYSPACE_NAME);
+ String table = (String)
compositeData.get(CompressionDictionaryDetailsTabularData.TABLE_NAME);
+ long dictId = (Long)
compositeData.get(CompressionDictionaryDetailsTabularData.DICT_ID_NAME);
+ byte[] dictionaryBytes = (byte[])
compositeData.get(CompressionDictionaryDetailsTabularData.DICT_NAME);
+ String kind = (String)
compositeData.get(CompressionDictionaryDetailsTabularData.KIND_NAME);
+ int checksum = (Integer)
compositeData.get(CompressionDictionaryDetailsTabularData.CHECKSUM_NAME);
+ int size = (Integer)
compositeData.get(CompressionDictionaryDetailsTabularData.SIZE_NAME);
+
+ CompressionDictionaryPojo pojo = new CompressionDictionaryPojo();
+ pojo.keyspace = keyspace;
+ pojo.table = table;
+ pojo.dictId = dictId;
+ pojo.dict = dictionaryBytes;
+ pojo.kind = kind;
+ pojo.dictChecksum = checksum;
+ pojo.dictLength = size;
+
+ pojo.validate();
+
+ return pojo;
+ }
+
+ public static class CompressionDictionaryPojo
+ {
+ public String keyspace;
+ public String table;
+ public long dictId;
+ public byte[] dict;
+ public String kind;
+ public int dictChecksum;
+ public int dictLength;
+
+ /**
+ * Dictionary is valid if, keyspace and table are specified,
dictionary id is strictly positive integer,
+ * dictionary byte array is not nor not empty,
+ * kind corresponds to {@code Kind}, checksum and size are bigger than
0.
+ */
+ public void validate()
+ {
+ if (keyspace == null)
+ throw new IllegalArgumentException("keyspace not specified");
+ if (table == null)
+ throw new IllegalArgumentException("table not specified");
+ if (dictId <= 0)
+ throw new IllegalArgumentException("Provided dictionary id is
lower than 0, it is '" + dictId + "'.'");
+ if (dict == null || dict.length == 0)
+ throw new IllegalArgumentException("Provided dictionary byte
array is null or empty.");
+ if (kind == null)
+ throw new IllegalArgumentException("Provided kind is null.");
+
+ CompressionDictionary.Kind dictionarykind;
Review Comment:
nit: snake-case
```suggestion
CompressionDictionary.Kind dictionaryKind;
```
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryDetailsTabularData.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Arrays;
+import javax.management.openmbean.ArrayType;
+import javax.management.openmbean.CompositeData;
+import javax.management.openmbean.CompositeDataSupport;
+import javax.management.openmbean.CompositeType;
+import javax.management.openmbean.OpenDataException;
+import javax.management.openmbean.OpenType;
+import javax.management.openmbean.SimpleType;
+import javax.management.openmbean.TabularDataSupport;
+import javax.management.openmbean.TabularType;
+
+import static java.lang.String.format;
+
+public class CompressionDictionaryDetailsTabularData
+{
+ public static final String KEYSPACE_NAME = "Keyspace";
+ public static final String TABLE_NAME = "Table";
+ public static final String DICT_ID_NAME = "DictId";
+ public static final String DICT_NAME = "Dict";
+ public static final String KIND_NAME = "Kind";
+ public static final String CHECKSUM_NAME = "Checksum";
+ public static final String SIZE_NAME = "Size";
+
+
+ private static final String[] ITEM_NAMES = new String[]{ KEYSPACE_NAME,
+ TABLE_NAME,
+ DICT_ID_NAME,
+ DICT_NAME,
+ KIND_NAME,
+ CHECKSUM_NAME,
+ SIZE_NAME };
+
+ private static final String[] ITEM_DESCS = new String[]{ "keyspace",
+ "table",
+ "dictionary_id",
+
"dictionary_bytes",
+ "kind",
+ "checksum",
+ "size" };
+
+ private static final String TYPE_NAME = "DictionaryDetails";
+ private static final String ROW_DESC = "DictionaryDetails";
+ private static final OpenType<?>[] ITEM_TYPES;
+ private static final CompositeType COMPOSITE_TYPE;
+ public static final TabularType TABULAR_TYPE;
+
+ static
+ {
+ try
+ {
+ ITEM_TYPES = new OpenType[]{ SimpleType.STRING, // keyspace
+ SimpleType.STRING, // table
+ SimpleType.LONG, // dict id
+ new
ArrayType<String[]>(SimpleType.BYTE, true), // dict bytes
+ SimpleType.STRING, // kind
+ SimpleType.INTEGER, // checksum
+ SimpleType.INTEGER }; // size of dict
bytes
+
+ COMPOSITE_TYPE = new CompositeType(TYPE_NAME, ROW_DESC,
ITEM_NAMES, ITEM_DESCS, ITEM_TYPES);
+ TABULAR_TYPE = new TabularType(TYPE_NAME, ROW_DESC,
COMPOSITE_TYPE, ITEM_NAMES);
+ }
+ catch (OpenDataException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static void from(String keyspace,
+ String table,
+ CompressionDictionary dictionary,
+ TabularDataSupport result)
+ {
+ result.put(from(keyspace, table, dictionary));
+ }
+
+ public static CompositeData from(String keyspace, String table,
CompressionDictionary dictionary)
+ {
+ try
+ {
+ return new CompositeDataSupport(COMPOSITE_TYPE,
+ ITEM_NAMES,
+ new Object[]
+ {
+ keyspace,
+ table,
+ dictionary.dictId().id,
+ dictionary.rawDictionary(),
+ dictionary.kind().name(),
+ dictionary.checksum(),
+ dictionary.size(),
+ });
+ }
+ catch (OpenDataException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static CompositeData from(CompressionDictionaryPojo pojo)
+ {
+ try
+ {
+ return new CompositeDataSupport(COMPOSITE_TYPE,
+ ITEM_NAMES,
+ new Object[]
+ {
+ pojo.keyspace,
+ pojo.table,
+ pojo.dictId,
+ pojo.dict,
+ pojo.kind,
+ pojo.dictChecksum,
+ pojo.dictLength
+ });
+ }
+ catch (OpenDataException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /**
+ * Deserializes data to convenience object to work further with.
+ *
+ * @param compositeData data to create pojo from
+ * @return deserialized composite data to convenience object
+ * @throws IllegalArgumentException if values in deserialized object are
invalid.
+ * @see CompressionDictionaryPojo#validate()
+ */
+ public static CompressionDictionaryPojo from(CompositeData compositeData)
+ {
+ String keyspace = (String)
compositeData.get(CompressionDictionaryDetailsTabularData.KEYSPACE_NAME);
+ String table = (String)
compositeData.get(CompressionDictionaryDetailsTabularData.TABLE_NAME);
+ long dictId = (Long)
compositeData.get(CompressionDictionaryDetailsTabularData.DICT_ID_NAME);
+ byte[] dictionaryBytes = (byte[])
compositeData.get(CompressionDictionaryDetailsTabularData.DICT_NAME);
+ String kind = (String)
compositeData.get(CompressionDictionaryDetailsTabularData.KIND_NAME);
+ int checksum = (Integer)
compositeData.get(CompressionDictionaryDetailsTabularData.CHECKSUM_NAME);
+ int size = (Integer)
compositeData.get(CompressionDictionaryDetailsTabularData.SIZE_NAME);
+
+ CompressionDictionaryPojo pojo = new CompressionDictionaryPojo();
+ pojo.keyspace = keyspace;
+ pojo.table = table;
+ pojo.dictId = dictId;
+ pojo.dict = dictionaryBytes;
+ pojo.kind = kind;
+ pojo.dictChecksum = checksum;
+ pojo.dictLength = size;
+
+ pojo.validate();
+
+ return pojo;
+ }
+
+ public static class CompressionDictionaryPojo
+ {
+ public String keyspace;
+ public String table;
+ public long dictId;
+ public byte[] dict;
+ public String kind;
+ public int dictChecksum;
+ public int dictLength;
+
+ /**
+ * Dictionary is valid if, keyspace and table are specified,
dictionary id is strictly positive integer,
+ * dictionary byte array is not nor not empty,
+ * kind corresponds to {@code Kind}, checksum and size are bigger than
0.
+ */
+ public void validate()
+ {
+ if (keyspace == null)
+ throw new IllegalArgumentException("keyspace not specified");
+ if (table == null)
+ throw new IllegalArgumentException("table not specified");
+ if (dictId <= 0)
+ throw new IllegalArgumentException("Provided dictionary id is
lower than 0, it is '" + dictId + "'.'");
+ if (dict == null || dict.length == 0)
+ throw new IllegalArgumentException("Provided dictionary byte
array is null or empty.");
+ if (kind == null)
+ throw new IllegalArgumentException("Provided kind is null.");
+
+ CompressionDictionary.Kind dictionarykind;
+
+ try
+ {
+ dictionarykind = CompressionDictionary.Kind.valueOf(kind);
+ }
+ catch (IllegalArgumentException ex)
+ {
+ throw new IllegalArgumentException("There is no such
dictionary kind like '" + kind + "'. Available kinds: " +
Arrays.asList(CompressionDictionary.Kind.values()));
+ }
+
+ if (dictChecksum <= 0)
+ throw new IllegalArgumentException("Checksum has to be
strictly positive number, it is '" + dictChecksum + "'.");
Review Comment:
I think the checksum could be negative. The checksum is from crc32, which
produce 32 bits. In signed integer, half of the values are negative.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManagerMBean.java:
##########
@@ -44,4 +46,42 @@ public interface CompressionDictionaryManagerMBean
* @return CompositeData representing {@link TrainingState}
*/
CompositeData getTrainingState();
+
+ /**
+ * Lists dictionaries for given keyspace and table. Returned dictionaries
+ * do not contain raw dictionary bytes.
+ *
+ * @param keyspace keyspace to get dictionaries from
+ * @param table table to get dictionaries from
+ * @return dictionaries for given keyspace and table
+ */
+ TabularData listDictionaries(String keyspace, String table);
+
+ /**
+ * Get latest compression dictionary.
+ *
+ * @param keyspace keyspace to get dictionary for
+ * @param table table to get dictionary for
+ * @return CompositeData representing dictionary or null of not found
Review Comment:
typo:
```suggestion
* @return CompositeData representing dictionary or null if not found
```
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -49,6 +49,20 @@ public interface CompressionDictionary extends AutoCloseable
*/
byte[] rawDictionary();
+ /**
+ * Get checkum of this dictionary.
+ *
+ * @return checksum of this dictionary
+ */
+ int checksum();
+
+ /**
+ * Get size of raw dictionary.
+ *
+ * @return size of raw dictionary, in bytes
+ */
+ int size();
Review Comment:
Why is this method needed in addition to `byte[] rawDictionary();`?
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -281,4 +315,60 @@ public String toString()
'}';
}
}
+
+ /**
+ * The purpose of lightweight dictionary is to not carry the actual
dictionary bytes for performance reasons.
+ * Handy for situations when retrieval from the database does not need to
contain dictionary
+ * or the instatiation of a proper dictionary object is not desirable or
unnecessary for other,
+ * mostly performance-related, reasons.
+ */
+ class LightweightCompressionDictionary implements CompressionDictionary
Review Comment:
Giving it another thought, should it implement `CompressionDictionary`? What
about just have a data class that contains 3 fields? I can see that
`LightweightCompressionDictionary` type is used specifically, e.g.
`SystemDistributedKeyspace#retrieveLightweightLatestCompressionDictionary`
returns `LightweightCompressionDictionary`, not `CompressionDictionary`.
By having a specific data type would give better type safety and caller may
not need to worry about the unnecessary methods (that are marked unsupported or
returning null, etc.)
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -281,4 +315,60 @@ public String toString()
'}';
}
}
+
+ /**
+ * The purpose of lightweight dictionary is to not carry the actual
dictionary bytes for performance reasons.
+ * Handy for situations when retrieval from the database does not need to
contain dictionary
+ * or the instatiation of a proper dictionary object is not desirable or
unnecessary for other,
Review Comment:
typo:
```suggestion
* or the instantiation of a proper dictionary object is not desirable
or unnecessary for other,
```
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManagerMBean.java:
##########
@@ -44,4 +46,42 @@ public interface CompressionDictionaryManagerMBean
* @return CompositeData representing {@link TrainingState}
*/
CompositeData getTrainingState();
+
+ /**
+ * Lists dictionaries for given keyspace and table. Returned dictionaries
+ * do not contain raw dictionary bytes.
+ *
+ * @param keyspace keyspace to get dictionaries from
+ * @param table table to get dictionaries from
+ * @return dictionaries for given keyspace and table
+ */
+ TabularData listDictionaries(String keyspace, String table);
Review Comment:
1. Please name the method (this and the other new ones) as
`listCompressionDictionaries` or simply `list` if you consider dictionaries is
implied already implied by the mbean name. I intentionally stayed about from
using "dictionary" alone in the initial patch, because the word "dictionary" is
too general.
2. I do not think you need to supply `String keyspace, String table`. The
MBean, like CFS MBean, is per table. The keyspace and table names are already
known.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryDetailsTabularData.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Arrays;
+import javax.management.openmbean.ArrayType;
+import javax.management.openmbean.CompositeData;
+import javax.management.openmbean.CompositeDataSupport;
+import javax.management.openmbean.CompositeType;
+import javax.management.openmbean.OpenDataException;
+import javax.management.openmbean.OpenType;
+import javax.management.openmbean.SimpleType;
+import javax.management.openmbean.TabularDataSupport;
+import javax.management.openmbean.TabularType;
+
+import static java.lang.String.format;
+
+public class CompressionDictionaryDetailsTabularData
+{
+ public static final String KEYSPACE_NAME = "Keyspace";
+ public static final String TABLE_NAME = "Table";
+ public static final String DICT_ID_NAME = "DictId";
+ public static final String DICT_NAME = "Dict";
+ public static final String KIND_NAME = "Kind";
+ public static final String CHECKSUM_NAME = "Checksum";
+ public static final String SIZE_NAME = "Size";
+
+
+ private static final String[] ITEM_NAMES = new String[]{ KEYSPACE_NAME,
+ TABLE_NAME,
+ DICT_ID_NAME,
+ DICT_NAME,
+ KIND_NAME,
+ CHECKSUM_NAME,
+ SIZE_NAME };
+
+ private static final String[] ITEM_DESCS = new String[]{ "keyspace",
+ "table",
+ "dictionary_id",
+
"dictionary_bytes",
+ "kind",
+ "checksum",
+ "size" };
+
+ private static final String TYPE_NAME = "DictionaryDetails";
+ private static final String ROW_DESC = "DictionaryDetails";
+ private static final OpenType<?>[] ITEM_TYPES;
+ private static final CompositeType COMPOSITE_TYPE;
+ public static final TabularType TABULAR_TYPE;
+
+ static
+ {
+ try
+ {
+ ITEM_TYPES = new OpenType[]{ SimpleType.STRING, // keyspace
+ SimpleType.STRING, // table
+ SimpleType.LONG, // dict id
+ new
ArrayType<String[]>(SimpleType.BYTE, true), // dict bytes
+ SimpleType.STRING, // kind
+ SimpleType.INTEGER, // checksum
+ SimpleType.INTEGER }; // size of dict
bytes
+
+ COMPOSITE_TYPE = new CompositeType(TYPE_NAME, ROW_DESC,
ITEM_NAMES, ITEM_DESCS, ITEM_TYPES);
+ TABULAR_TYPE = new TabularType(TYPE_NAME, ROW_DESC,
COMPOSITE_TYPE, ITEM_NAMES);
+ }
+ catch (OpenDataException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static void from(String keyspace,
+ String table,
+ CompressionDictionary dictionary,
+ TabularDataSupport result)
+ {
+ result.put(from(keyspace, table, dictionary));
+ }
+
+ public static CompositeData from(String keyspace, String table,
CompressionDictionary dictionary)
+ {
+ try
+ {
+ return new CompositeDataSupport(COMPOSITE_TYPE,
+ ITEM_NAMES,
+ new Object[]
+ {
+ keyspace,
+ table,
+ dictionary.dictId().id,
+ dictionary.rawDictionary(),
+ dictionary.kind().name(),
+ dictionary.checksum(),
+ dictionary.size(),
+ });
+ }
+ catch (OpenDataException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static CompositeData from(CompressionDictionaryPojo pojo)
+ {
+ try
+ {
+ return new CompositeDataSupport(COMPOSITE_TYPE,
+ ITEM_NAMES,
+ new Object[]
+ {
+ pojo.keyspace,
+ pojo.table,
+ pojo.dictId,
+ pojo.dict,
+ pojo.kind,
+ pojo.dictChecksum,
+ pojo.dictLength
+ });
+ }
+ catch (OpenDataException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /**
+ * Deserializes data to convenience object to work further with.
+ *
+ * @param compositeData data to create pojo from
+ * @return deserialized composite data to convenience object
+ * @throws IllegalArgumentException if values in deserialized object are
invalid.
+ * @see CompressionDictionaryPojo#validate()
+ */
+ public static CompressionDictionaryPojo from(CompositeData compositeData)
+ {
+ String keyspace = (String)
compositeData.get(CompressionDictionaryDetailsTabularData.KEYSPACE_NAME);
+ String table = (String)
compositeData.get(CompressionDictionaryDetailsTabularData.TABLE_NAME);
+ long dictId = (Long)
compositeData.get(CompressionDictionaryDetailsTabularData.DICT_ID_NAME);
+ byte[] dictionaryBytes = (byte[])
compositeData.get(CompressionDictionaryDetailsTabularData.DICT_NAME);
+ String kind = (String)
compositeData.get(CompressionDictionaryDetailsTabularData.KIND_NAME);
+ int checksum = (Integer)
compositeData.get(CompressionDictionaryDetailsTabularData.CHECKSUM_NAME);
+ int size = (Integer)
compositeData.get(CompressionDictionaryDetailsTabularData.SIZE_NAME);
+
+ CompressionDictionaryPojo pojo = new CompressionDictionaryPojo();
+ pojo.keyspace = keyspace;
+ pojo.table = table;
+ pojo.dictId = dictId;
+ pojo.dict = dictionaryBytes;
+ pojo.kind = kind;
+ pojo.dictChecksum = checksum;
+ pojo.dictLength = size;
+
+ pojo.validate();
+
+ return pojo;
+ }
+
+ public static class CompressionDictionaryPojo
+ {
+ public String keyspace;
+ public String table;
+ public long dictId;
+ public byte[] dict;
+ public String kind;
+ public int dictChecksum;
+ public int dictLength;
+
+ /**
+ * Dictionary is valid if, keyspace and table are specified,
dictionary id is strictly positive integer,
+ * dictionary byte array is not nor not empty,
+ * kind corresponds to {@code Kind}, checksum and size are bigger than
0.
+ */
+ public void validate()
+ {
+ if (keyspace == null)
+ throw new IllegalArgumentException("keyspace not specified");
+ if (table == null)
+ throw new IllegalArgumentException("table not specified");
+ if (dictId <= 0)
+ throw new IllegalArgumentException("Provided dictionary id is
lower than 0, it is '" + dictId + "'.'");
+ if (dict == null || dict.length == 0)
+ throw new IllegalArgumentException("Provided dictionary byte
array is null or empty.");
+ if (kind == null)
+ throw new IllegalArgumentException("Provided kind is null.");
+
+ CompressionDictionary.Kind dictionarykind;
+
+ try
+ {
+ dictionarykind = CompressionDictionary.Kind.valueOf(kind);
+ }
+ catch (IllegalArgumentException ex)
+ {
+ throw new IllegalArgumentException("There is no such
dictionary kind like '" + kind + "'. Available kinds: " +
Arrays.asList(CompressionDictionary.Kind.values()));
+ }
+
+ if (dictChecksum <= 0)
+ throw new IllegalArgumentException("Checksum has to be
strictly positive number, it is '" + dictChecksum + "'.");
+ if (dictLength <= 0)
+ throw new IllegalArgumentException("Size has to be strictly
positive number, it is '" + dictLength + "'.");
+ if (dict.length != dictLength)
+ throw new IllegalArgumentException("The length of the provided
dictionary array (" + dict.length + ") does not equal to provided lenght value
(" + dictLength + ").");
Review Comment:
typo:
```suggestion
throw new IllegalArgumentException("The length of the
provided dictionary array (" + dict.length + ") does not equal to provided
length value (" + dictLength + ").");
```
##########
src/java/org/apache/cassandra/schema/SystemDistributedKeyspace.java:
##########
@@ -458,18 +459,45 @@ public static CompressionDictionary
retrieveLatestCompressionDictionary(String k
}
/**
- * Retrieves a specific compression dictionary for a given keyspace and
table.
+ * Retrieves the latest compression dictionary for a given keyspace and
table
+ * backed by {@link LightweightCompressionDictionary} object.
+ *
+ * @param keyspaceName the keyspace name to retrieve the dictionary for
+ * @param tableName the table name to retrieve the dictionary for
+ * @return the latest compression dictionary for the specified keyspace
and table,
+ * or null if no dictionary exists or if an error occurs during
retrieval
+ */
+ @Nullable
Review Comment:
👍 on adding the annotation.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -244,6 +251,68 @@ public CompositeData getTrainingState()
return dictionaryTrainer.getTrainingState().toCompositeData();
}
+ @Override
+ public TabularData listDictionaries(String keyspace, String table)
+ {
+ List<LightweightCompressionDictionary> dictionaries =
SystemDistributedKeyspace.retrieveLightweightCompressionDictionaries(keyspace,
table);
Review Comment:
We should use the field variables of the enclosing
`CompressionDictionaryManager` instance.
```
private final String keyspaceName;
private final String tableName;
```
##########
src/java/org/apache/cassandra/tools/nodetool/CompressionDictionary.java:
##########
@@ -119,4 +136,168 @@ private static void displayProgress(TrainingState
trainingState, long startTime,
status, sampleCount, sampleSizeMB, elapsedSeconds);
}
}
+
+ @Command(name = "list", description = "List available dictionaries of
specific keyspace and table.")
+ public static class List extends AbstractCommand
Review Comment:
nit: Could we rename the command, so that the full qualified name
`java.util.List` can be avoided?
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManagerMBean.java:
##########
@@ -44,4 +46,42 @@ public interface CompressionDictionaryManagerMBean
* @return CompositeData representing {@link TrainingState}
*/
CompositeData getTrainingState();
+
+ /**
+ * Lists dictionaries for given keyspace and table. Returned dictionaries
+ * do not contain raw dictionary bytes.
+ *
+ * @param keyspace keyspace to get dictionaries from
+ * @param table table to get dictionaries from
+ * @return dictionaries for given keyspace and table
+ */
+ TabularData listDictionaries(String keyspace, String table);
+
+ /**
+ * Get latest compression dictionary.
+ *
+ * @param keyspace keyspace to get dictionary for
+ * @param table table to get dictionary for
+ * @return CompositeData representing dictionary or null of not found
+ */
+ @Nullable
+ CompositeData getDictionary(String keyspace, String table);
+
+ /**
+ * Get compression dictionary.
+ *
+ * @param keyspace keyspace to get dictionary for
+ * @param table table to get dictionary for
+ * @param dictId id of dictionary to get
+ * @return CompositeData representing dictionary or null of not found
+ */
+ @Nullable
+ CompositeData getDictionary(String keyspace, String table, long dictId);
Review Comment:
`String keyspace, String table` are redundant.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManagerMBean.java:
##########
@@ -44,4 +46,42 @@ public interface CompressionDictionaryManagerMBean
* @return CompositeData representing {@link TrainingState}
*/
CompositeData getTrainingState();
+
+ /**
+ * Lists dictionaries for given keyspace and table. Returned dictionaries
+ * do not contain raw dictionary bytes.
+ *
+ * @param keyspace keyspace to get dictionaries from
+ * @param table table to get dictionaries from
+ * @return dictionaries for given keyspace and table
+ */
+ TabularData listDictionaries(String keyspace, String table);
+
+ /**
+ * Get latest compression dictionary.
+ *
+ * @param keyspace keyspace to get dictionary for
+ * @param table table to get dictionary for
+ * @return CompositeData representing dictionary or null of not found
+ */
+ @Nullable
+ CompositeData getDictionary(String keyspace, String table);
Review Comment:
`String keyspace, String table` are redundant.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryDetailsTabularData.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Arrays;
+import javax.management.openmbean.ArrayType;
+import javax.management.openmbean.CompositeData;
+import javax.management.openmbean.CompositeDataSupport;
+import javax.management.openmbean.CompositeType;
+import javax.management.openmbean.OpenDataException;
+import javax.management.openmbean.OpenType;
+import javax.management.openmbean.SimpleType;
+import javax.management.openmbean.TabularDataSupport;
+import javax.management.openmbean.TabularType;
+
+import static java.lang.String.format;
+
+public class CompressionDictionaryDetailsTabularData
+{
+ public static final String KEYSPACE_NAME = "Keyspace";
+ public static final String TABLE_NAME = "Table";
+ public static final String DICT_ID_NAME = "DictId";
+ public static final String DICT_NAME = "Dict";
+ public static final String KIND_NAME = "Kind";
+ public static final String CHECKSUM_NAME = "Checksum";
+ public static final String SIZE_NAME = "Size";
+
+
+ private static final String[] ITEM_NAMES = new String[]{ KEYSPACE_NAME,
+ TABLE_NAME,
+ DICT_ID_NAME,
+ DICT_NAME,
+ KIND_NAME,
+ CHECKSUM_NAME,
+ SIZE_NAME };
+
+ private static final String[] ITEM_DESCS = new String[]{ "keyspace",
+ "table",
+ "dictionary_id",
+
"dictionary_bytes",
+ "kind",
+ "checksum",
+ "size" };
+
+ private static final String TYPE_NAME = "DictionaryDetails";
+ private static final String ROW_DESC = "DictionaryDetails";
+ private static final OpenType<?>[] ITEM_TYPES;
+ private static final CompositeType COMPOSITE_TYPE;
+ public static final TabularType TABULAR_TYPE;
+
+ static
+ {
+ try
+ {
+ ITEM_TYPES = new OpenType[]{ SimpleType.STRING, // keyspace
+ SimpleType.STRING, // table
+ SimpleType.LONG, // dict id
+ new
ArrayType<String[]>(SimpleType.BYTE, true), // dict bytes
+ SimpleType.STRING, // kind
+ SimpleType.INTEGER, // checksum
+ SimpleType.INTEGER }; // size of dict
bytes
+
+ COMPOSITE_TYPE = new CompositeType(TYPE_NAME, ROW_DESC,
ITEM_NAMES, ITEM_DESCS, ITEM_TYPES);
+ TABULAR_TYPE = new TabularType(TYPE_NAME, ROW_DESC,
COMPOSITE_TYPE, ITEM_NAMES);
+ }
+ catch (OpenDataException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static void from(String keyspace,
+ String table,
+ CompressionDictionary dictionary,
+ TabularDataSupport result)
+ {
+ result.put(from(keyspace, table, dictionary));
+ }
+
+ public static CompositeData from(String keyspace, String table,
CompressionDictionary dictionary)
+ {
+ try
+ {
+ return new CompositeDataSupport(COMPOSITE_TYPE,
+ ITEM_NAMES,
+ new Object[]
+ {
+ keyspace,
+ table,
+ dictionary.dictId().id,
+ dictionary.rawDictionary(),
+ dictionary.kind().name(),
+ dictionary.checksum(),
+ dictionary.size(),
+ });
+ }
+ catch (OpenDataException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static CompositeData from(CompressionDictionaryPojo pojo)
+ {
+ try
+ {
+ return new CompositeDataSupport(COMPOSITE_TYPE,
+ ITEM_NAMES,
+ new Object[]
+ {
+ pojo.keyspace,
+ pojo.table,
+ pojo.dictId,
+ pojo.dict,
+ pojo.kind,
+ pojo.dictChecksum,
+ pojo.dictLength
+ });
+ }
+ catch (OpenDataException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /**
+ * Deserializes data to convenience object to work further with.
+ *
+ * @param compositeData data to create pojo from
+ * @return deserialized composite data to convenience object
+ * @throws IllegalArgumentException if values in deserialized object are
invalid.
+ * @see CompressionDictionaryPojo#validate()
+ */
+ public static CompressionDictionaryPojo from(CompositeData compositeData)
+ {
+ String keyspace = (String)
compositeData.get(CompressionDictionaryDetailsTabularData.KEYSPACE_NAME);
+ String table = (String)
compositeData.get(CompressionDictionaryDetailsTabularData.TABLE_NAME);
+ long dictId = (Long)
compositeData.get(CompressionDictionaryDetailsTabularData.DICT_ID_NAME);
+ byte[] dictionaryBytes = (byte[])
compositeData.get(CompressionDictionaryDetailsTabularData.DICT_NAME);
+ String kind = (String)
compositeData.get(CompressionDictionaryDetailsTabularData.KIND_NAME);
+ int checksum = (Integer)
compositeData.get(CompressionDictionaryDetailsTabularData.CHECKSUM_NAME);
+ int size = (Integer)
compositeData.get(CompressionDictionaryDetailsTabularData.SIZE_NAME);
+
+ CompressionDictionaryPojo pojo = new CompressionDictionaryPojo();
+ pojo.keyspace = keyspace;
+ pojo.table = table;
+ pojo.dictId = dictId;
+ pojo.dict = dictionaryBytes;
+ pojo.kind = kind;
+ pojo.dictChecksum = checksum;
+ pojo.dictLength = size;
+
+ pojo.validate();
+
+ return pojo;
+ }
+
+ public static class CompressionDictionaryPojo
+ {
+ public String keyspace;
+ public String table;
+ public long dictId;
+ public byte[] dict;
+ public String kind;
+ public int dictChecksum;
+ public int dictLength;
+
+ /**
+ * Dictionary is valid if, keyspace and table are specified,
dictionary id is strictly positive integer,
+ * dictionary byte array is not nor not empty,
+ * kind corresponds to {@code Kind}, checksum and size are bigger than
0.
+ */
+ public void validate()
+ {
+ if (keyspace == null)
+ throw new IllegalArgumentException("keyspace not specified");
+ if (table == null)
+ throw new IllegalArgumentException("table not specified");
+ if (dictId <= 0)
+ throw new IllegalArgumentException("Provided dictionary id is
lower than 0, it is '" + dictId + "'.'");
+ if (dict == null || dict.length == 0)
+ throw new IllegalArgumentException("Provided dictionary byte
array is null or empty.");
Review Comment:
Can we add a new check to ensure the importing dictionary size does not
exceed certain amount? We will not want to import dictionary that is larger
than 1MiB, most likely the dictionary should be less than 100KiB.
##########
src/java/org/apache/cassandra/db/compression/ZstdCompressionDictionary.java:
##########
@@ -38,16 +39,28 @@ public class ZstdCompressionDictionary implements
CompressionDictionary, SelfRef
private final DictId dictId;
private final byte[] rawDictionary;
+ private volatile int checksum;
Review Comment:
why is it `volatile`? the value is never updated.
##########
src/java/org/apache/cassandra/schema/SystemDistributedKeyspace.java:
##########
@@ -481,6 +509,38 @@ public static CompressionDictionary
retrieveCompressionDictionary(String keyspac
}
}
+ /**
+ * Retrieves all dictionaries for a given keyspace and table.
+ *
+ * @param keyspaceName the keyspace name to retrieve the dictionary for
+ * @param tableName the table name to retrieve the dictionary for
+ * @return the compression dictionaries identified by the specified
keyspace and table,
+ * or null if no dictionary exists or if an error occurs during
retrieval
+ */
+ @Nullable
Review Comment:
Should it return an empty list? It is cheap and simplifies the call-site
handling.
##########
src/java/org/apache/cassandra/tools/nodetool/CompressionDictionary.java:
##########
@@ -18,25 +18,42 @@
package org.apache.cassandra.tools.nodetool;
import java.io.PrintStream;
+import java.util.ArrayList;
import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import javax.management.openmbean.CompositeData;
+import javax.management.openmbean.TabularData;
import com.google.common.util.concurrent.Uninterruptibles;
+import
org.apache.cassandra.db.compression.CompressionDictionaryDetailsTabularData;
+import
org.apache.cassandra.db.compression.CompressionDictionaryDetailsTabularData.CompressionDictionaryPojo;
import
org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
import org.apache.cassandra.db.compression.TrainingState;
+import org.apache.cassandra.io.util.File;
+import org.apache.cassandra.io.util.FileUtils;
import org.apache.cassandra.tools.NodeProbe;
+import org.apache.cassandra.tools.nodetool.formatter.TableBuilder;
import org.apache.cassandra.utils.Clock;
+import org.apache.cassandra.utils.JsonUtils;
import picocli.CommandLine.Command;
import picocli.CommandLine.Option;
import picocli.CommandLine.Parameters;
+import static java.nio.file.StandardOpenOption.CREATE;
+import static java.nio.file.StandardOpenOption.TRUNCATE_EXISTING;
+import static java.nio.file.StandardOpenOption.WRITE;
+
@Command(name = "compressiondictionary",
- description = "Manage compression dictionaries",
- subcommands = { CompressionDictionary.Train.class })
+description = "Manage compression dictionaries",
+subcommands = { CompressionDictionary.Train.class,
+ CompressionDictionary.List.class,
+ CompressionDictionary.Export.class,
+ CompressionDictionary.Import.class })
public class CompressionDictionary
{
@Command(name = "train",
- description = "Manually trigger compression dictionary training
for a table. If no SSTables are available, the memtable will be flushed first.")
+ description = "Manually trigger compression dictionary training for a
table. If no SSTables are available, the memtable will be flushed first.")
Review Comment:
nit: keep the alignment of the annotation elements?
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryDetailsTabularData.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Arrays;
+import javax.management.openmbean.ArrayType;
+import javax.management.openmbean.CompositeData;
+import javax.management.openmbean.CompositeDataSupport;
+import javax.management.openmbean.CompositeType;
+import javax.management.openmbean.OpenDataException;
+import javax.management.openmbean.OpenType;
+import javax.management.openmbean.SimpleType;
+import javax.management.openmbean.TabularDataSupport;
+import javax.management.openmbean.TabularType;
+
+import static java.lang.String.format;
+
+public class CompressionDictionaryDetailsTabularData
+{
+ public static final String KEYSPACE_NAME = "Keyspace";
+ public static final String TABLE_NAME = "Table";
+ public static final String DICT_ID_NAME = "DictId";
+ public static final String DICT_NAME = "Dict";
+ public static final String KIND_NAME = "Kind";
+ public static final String CHECKSUM_NAME = "Checksum";
+ public static final String SIZE_NAME = "Size";
+
+
+ private static final String[] ITEM_NAMES = new String[]{ KEYSPACE_NAME,
+ TABLE_NAME,
+ DICT_ID_NAME,
+ DICT_NAME,
+ KIND_NAME,
+ CHECKSUM_NAME,
+ SIZE_NAME };
+
+ private static final String[] ITEM_DESCS = new String[]{ "keyspace",
+ "table",
+ "dictionary_id",
+
"dictionary_bytes",
+ "kind",
+ "checksum",
+ "size" };
+
+ private static final String TYPE_NAME = "DictionaryDetails";
+ private static final String ROW_DESC = "DictionaryDetails";
+ private static final OpenType<?>[] ITEM_TYPES;
+ private static final CompositeType COMPOSITE_TYPE;
+ public static final TabularType TABULAR_TYPE;
+
+ static
+ {
+ try
+ {
+ ITEM_TYPES = new OpenType[]{ SimpleType.STRING, // keyspace
+ SimpleType.STRING, // table
+ SimpleType.LONG, // dict id
+ new
ArrayType<String[]>(SimpleType.BYTE, true), // dict bytes
+ SimpleType.STRING, // kind
+ SimpleType.INTEGER, // checksum
+ SimpleType.INTEGER }; // size of dict
bytes
+
+ COMPOSITE_TYPE = new CompositeType(TYPE_NAME, ROW_DESC,
ITEM_NAMES, ITEM_DESCS, ITEM_TYPES);
+ TABULAR_TYPE = new TabularType(TYPE_NAME, ROW_DESC,
COMPOSITE_TYPE, ITEM_NAMES);
+ }
+ catch (OpenDataException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static void from(String keyspace,
+ String table,
+ CompressionDictionary dictionary,
+ TabularDataSupport result)
+ {
+ result.put(from(keyspace, table, dictionary));
+ }
+
+ public static CompositeData from(String keyspace, String table,
CompressionDictionary dictionary)
+ {
+ try
+ {
+ return new CompositeDataSupport(COMPOSITE_TYPE,
+ ITEM_NAMES,
+ new Object[]
+ {
+ keyspace,
+ table,
+ dictionary.dictId().id,
+ dictionary.rawDictionary(),
+ dictionary.kind().name(),
+ dictionary.checksum(),
+ dictionary.size(),
+ });
+ }
+ catch (OpenDataException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static CompositeData from(CompressionDictionaryPojo pojo)
+ {
+ try
+ {
+ return new CompositeDataSupport(COMPOSITE_TYPE,
+ ITEM_NAMES,
+ new Object[]
+ {
+ pojo.keyspace,
+ pojo.table,
+ pojo.dictId,
+ pojo.dict,
+ pojo.kind,
+ pojo.dictChecksum,
+ pojo.dictLength
+ });
+ }
+ catch (OpenDataException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /**
+ * Deserializes data to convenience object to work further with.
+ *
+ * @param compositeData data to create pojo from
+ * @return deserialized composite data to convenience object
+ * @throws IllegalArgumentException if values in deserialized object are
invalid.
+ * @see CompressionDictionaryPojo#validate()
+ */
+ public static CompressionDictionaryPojo from(CompositeData compositeData)
+ {
+ String keyspace = (String)
compositeData.get(CompressionDictionaryDetailsTabularData.KEYSPACE_NAME);
+ String table = (String)
compositeData.get(CompressionDictionaryDetailsTabularData.TABLE_NAME);
+ long dictId = (Long)
compositeData.get(CompressionDictionaryDetailsTabularData.DICT_ID_NAME);
+ byte[] dictionaryBytes = (byte[])
compositeData.get(CompressionDictionaryDetailsTabularData.DICT_NAME);
+ String kind = (String)
compositeData.get(CompressionDictionaryDetailsTabularData.KIND_NAME);
+ int checksum = (Integer)
compositeData.get(CompressionDictionaryDetailsTabularData.CHECKSUM_NAME);
+ int size = (Integer)
compositeData.get(CompressionDictionaryDetailsTabularData.SIZE_NAME);
+
+ CompressionDictionaryPojo pojo = new CompressionDictionaryPojo();
+ pojo.keyspace = keyspace;
+ pojo.table = table;
+ pojo.dictId = dictId;
+ pojo.dict = dictionaryBytes;
+ pojo.kind = kind;
+ pojo.dictChecksum = checksum;
+ pojo.dictLength = size;
+
+ pojo.validate();
+
+ return pojo;
+ }
+
+ public static class CompressionDictionaryPojo
+ {
+ public String keyspace;
+ public String table;
+ public long dictId;
+ public byte[] dict;
+ public String kind;
+ public int dictChecksum;
+ public int dictLength;
+
+ /**
+ * Dictionary is valid if, keyspace and table are specified,
dictionary id is strictly positive integer,
+ * dictionary byte array is not nor not empty,
+ * kind corresponds to {@code Kind}, checksum and size are bigger than
0.
+ */
+ public void validate()
+ {
+ if (keyspace == null)
+ throw new IllegalArgumentException("keyspace not specified");
+ if (table == null)
+ throw new IllegalArgumentException("table not specified");
+ if (dictId <= 0)
+ throw new IllegalArgumentException("Provided dictionary id is
lower than 0, it is '" + dictId + "'.'");
+ if (dict == null || dict.length == 0)
+ throw new IllegalArgumentException("Provided dictionary byte
array is null or empty.");
+ if (kind == null)
+ throw new IllegalArgumentException("Provided kind is null.");
+
+ CompressionDictionary.Kind dictionarykind;
+
+ try
+ {
+ dictionarykind = CompressionDictionary.Kind.valueOf(kind);
+ }
+ catch (IllegalArgumentException ex)
+ {
+ throw new IllegalArgumentException("There is no such
dictionary kind like '" + kind + "'. Available kinds: " +
Arrays.asList(CompressionDictionary.Kind.values()));
+ }
+
+ if (dictChecksum <= 0)
+ throw new IllegalArgumentException("Checksum has to be
strictly positive number, it is '" + dictChecksum + "'.");
+ if (dictLength <= 0)
+ throw new IllegalArgumentException("Size has to be strictly
positive number, it is '" + dictLength + "'.");
+ if (dict.length != dictLength)
+ throw new IllegalArgumentException("The length of the provided
dictionary array (" + dict.length + ") does not equal to provided lenght value
(" + dictLength + ").");
+
+ int checksumOfDictionaryToImport =
CompressionDictionary.calculateChecksum((byte) dictionarykind.ordinal(),
dictId, dict);
+ if (checksumOfDictionaryToImport != dictChecksum)
+ {
+ throw new IllegalArgumentException(format("Computed checksum
of dictionary to import (%s) is different from checksum specified on input
(%s).",
Review Comment:
nit: error message revision. `Compression dictionary checksum mismatched.
Actual: %s; provided: %s."
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -244,6 +251,68 @@ public CompositeData getTrainingState()
return dictionaryTrainer.getTrainingState().toCompositeData();
}
+ @Override
+ public TabularData listDictionaries(String keyspace, String table)
+ {
+ List<LightweightCompressionDictionary> dictionaries =
SystemDistributedKeyspace.retrieveLightweightCompressionDictionaries(keyspace,
table);
+ TabularDataSupport tableData = new
TabularDataSupport(CompressionDictionaryDetailsTabularData.TABULAR_TYPE);
+
+ if (dictionaries == null)
+ {
+ return tableData;
+ }
+
+ for (LightweightCompressionDictionary dictionary : dictionaries)
+ {
+ CompressionDictionaryDetailsTabularData.from(keyspace, table,
dictionary, tableData);
Review Comment:
It looks a bit odd that no return value is there in the `from` method and
not used.
Could you use this instead and delete the unused `from` method?
```java
for (LightweightCompressionDictionary dictionary : dictionaries)
{
tableData.put(CompressionDictionaryDetailsTabularData.from(keyspace, table,
dictionary));
}
```
It leads to the question: should the
`CompressionDictionaryDetailsTabularData` be called `TabularData`? It is to
return the individual compression dictionary details. Would it make better
sense to rename to `CompressionDictionaryDetails`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
