Merge branch cassandra-2.2 into cassandra-3.0
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/97fb4d10 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/97fb4d10 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/97fb4d10 Branch: refs/heads/trunk Commit: 97fb4d1029f23b53621125493e7582b2f045fdf8 Parents: e51c85f cb6fad3 Author: Benjamin Lerer <[email protected]> Authored: Fri Jul 7 17:14:32 2017 +0200 Committer: Benjamin Lerer <[email protected]> Committed: Fri Jul 7 17:14:32 2017 +0200 ---------------------------------------------------------------------- CHANGES.txt | 3 +- .../cassandra/db/marshal/AbstractType.java | 11 +- .../apache/cassandra/db/marshal/EmptyType.java | 5 + .../apache/cassandra/db/marshal/ListType.java | 5 +- .../apache/cassandra/db/marshal/MapType.java | 7 +- .../apache/cassandra/db/marshal/TupleType.java | 3 +- .../org/apache/cassandra/cql3/CQLTester.java | 11 + .../cql3/validation/entities/JsonTest.java | 170 +++++++++- .../db/marshal/JsonConversionTest.java | 320 +++++++++++++++++++ 9 files changed, 518 insertions(+), 17 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/cassandra/blob/97fb4d10/CHANGES.txt ---------------------------------------------------------------------- diff --cc CHANGES.txt index c98ffd9,6740c9e..8095e25 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,61 -1,16 +1,62 @@@ -2.2.11 - * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592) - * Fix nested Tuples/UDTs validation (CASSANDRA-13646) - * Remove unused max_value_size_in_mb config setting from yaml (CASSANDRA-13625 +3.0.15 + * Allow different NUMACTL_ARGS to be passed in (CASSANDRA-13557) + * Allow native function calls in CQLSSTableWriter (CASSANDRA-12606) + * Fix secondary index queries on COMPACT tables (CASSANDRA-13627) + * Nodetool listsnapshots output is missing a newline, if there are no snapshots (CASSANDRA-13568) + Merged from 2.2: - * Fix nested Tuples/UDTs validation (CASSANDRA-13646) ++ * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592) ++ * Fix nested Tuples/UDTs validation (CASSANDRA-13646) - -2.2.10 +3.0.14 + * Ensure int overflow doesn't occur when calculating large partition warning size (CASSANDRA-13172) + * Ensure consistent view of partition columns between coordinator and replica in ColumnFilter (CASSANDRA-13004) + * Failed unregistering mbean during drop keyspace (CASSANDRA-13346) + * nodetool scrub/cleanup/upgradesstables exit code is wrong (CASSANDRA-13542) + * Fix the reported number of sstable data files accessed per read (CASSANDRA-13120) + * Fix schema digest mismatch during rolling upgrades from versions before 3.0.12 (CASSANDRA-13559) + * Upgrade JNA version to 4.4.0 (CASSANDRA-13072) + * Interned ColumnIdentifiers should use minimal ByteBuffers (CASSANDRA-13533) + * ReverseIndexedReader may drop rows during 2.1 to 3.0 upgrade (CASSANDRA-13525) + * Fix repair process violating start/end token limits for small ranges (CASSANDRA-13052) + * Add storage port options to sstableloader (CASSANDRA-13518) + * Properly handle quoted index names in cqlsh DESCRIBE output (CASSANDRA-12847) + * Avoid reading static row twice from old format sstables (CASSANDRA-13236) + * Fix NPE in StorageService.excise() (CASSANDRA-13163) + * Expire OutboundTcpConnection messages by a single Thread (CASSANDRA-13265) + * Fail repair if insufficient responses received (CASSANDRA-13397) + * Fix SSTableLoader fail when the loaded table contains dropped columns (CASSANDRA-13276) + * Avoid name clashes in CassandraIndexTest (CASSANDRA-13427) + * Handling partially written hint files (CASSANDRA-12728) + * Interrupt replaying hints on decommission (CASSANDRA-13308) + * Fix schema version calculation for rolling upgrades (CASSANDRA-13441) +Merged from 2.2: * Nodes started with join_ring=False should be able to serve requests when authentication is enabled (CASSANDRA-11381) * cqlsh COPY FROM: increment error count only for failures, not for attempts (CASSANDRA-13209) - * nodetool upgradesstables should upgrade system tables (CASSANDRA-13119) + +3.0.13 + * Make reading of range tombstones more reliable (CASSANDRA-12811) + * Fix startup problems due to schema tables not completely flushed (CASSANDRA-12213) + * Fix view builder bug that can filter out data on restart (CASSANDRA-13405) + * Fix 2i page size calculation when there are no regular columns (CASSANDRA-13400) + * Fix the conversion of 2.X expired rows without regular column data (CASSANDRA-13395) + * Fix hint delivery when using ext+internal IPs with prefer_local enabled (CASSANDRA-13020) + * Fix possible NPE on upgrade to 3.0/3.X in case of IO errors (CASSANDRA-13389) + * Legacy deserializer can create empty range tombstones (CASSANDRA-13341) + * Use the Kernel32 library to retrieve the PID on Windows and fix startup checks (CASSANDRA-13333) + * Fix code to not exchange schema across major versions (CASSANDRA-13274) + * Dropping column results in "corrupt" SSTable (CASSANDRA-13337) + * Bugs handling range tombstones in the sstable iterators (CASSANDRA-13340) + * Fix CONTAINS filtering for null collections (CASSANDRA-13246) + * Applying: Use a unique metric reservoir per test run when using Cassandra-wide metrics residing in MBeans (CASSANDRA-13216) + * Propagate row deletions in 2i tables on upgrade (CASSANDRA-13320) + * Slice.isEmpty() returns false for some empty slices (CASSANDRA-13305) + * Add formatted row output to assertEmpty in CQL Tester (CASSANDRA-13238) + * Legacy caching options can prevent 3.0 upgrade (CASSANDRA-13384) + * Nodetool upgradesstables/scrub/compact ignores system tables (CASSANDRA-13410) + * Fix NPE issue in StorageService (CASSANDRA-13060) +Merged from 2.2: * Avoid starting gossiper in RemoveTest (CASSANDRA-13407) * Fix weightedSize() for row-cache reported by JMX and NodeTool (CASSANDRA-13393) - * Fix JVM metric paths (CASSANDRA-13103) * Honor truststore-password parameter in cassandra-stress (CASSANDRA-12773) * Discard in-flight shadow round responses (CASSANDRA-12653) * Don't anti-compact repaired data to avoid inconsistencies (CASSANDRA-13153) http://git-wip-us.apache.org/repos/asf/cassandra/blob/97fb4d10/src/java/org/apache/cassandra/db/marshal/AbstractType.java ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/cassandra/blob/97fb4d10/src/java/org/apache/cassandra/db/marshal/EmptyType.java ---------------------------------------------------------------------- diff --cc src/java/org/apache/cassandra/db/marshal/EmptyType.java index c653084,e5abe5b..806d160 --- a/src/java/org/apache/cassandra/db/marshal/EmptyType.java +++ b/src/java/org/apache/cassandra/db/marshal/EmptyType.java @@@ -67,11 -66,11 +67,16 @@@ public class EmptyType extends Abstract } @Override + public CQL3Type asCQL3Type() + { + return CQL3Type.Native.EMPTY; + } + + public String toJSONString(ByteBuffer buffer, int protocolVersion) + { + return "\"\""; + } + public TypeSerializer<Void> getSerializer() { return EmptySerializer.instance; http://git-wip-us.apache.org/repos/asf/cassandra/blob/97fb4d10/src/java/org/apache/cassandra/db/marshal/ListType.java ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/cassandra/blob/97fb4d10/src/java/org/apache/cassandra/db/marshal/MapType.java ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/cassandra/blob/97fb4d10/src/java/org/apache/cassandra/db/marshal/TupleType.java ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/cassandra/blob/97fb4d10/test/unit/org/apache/cassandra/cql3/CQLTester.java ---------------------------------------------------------------------- diff --cc test/unit/org/apache/cassandra/cql3/CQLTester.java index 39278f0,416a4b2..11d9e19 --- a/test/unit/org/apache/cassandra/cql3/CQLTester.java +++ b/test/unit/org/apache/cassandra/cql3/CQLTester.java @@@ -38,8 -35,11 +38,9 @@@ import org.junit.* import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import static junit.framework.Assert.assertNotNull; - import com.datastax.driver.core.*; import com.datastax.driver.core.ResultSet; + import org.apache.cassandra.SchemaLoader; import org.apache.cassandra.concurrent.ScheduledExecutors; import org.apache.cassandra.config.CFMetaData; @@@ -715,14 -549,16 +716,19 @@@ public abstract class CQLTeste protected com.datastax.driver.core.ResultSet executeNet(int protocolVersion, String query, Object... values) throws Throwable { - requireNetwork(); + return sessionNet(protocolVersion).execute(formatQuery(query), values); + } - return session[protocolVersion-1].execute(formatQuery(query), values); + protected Session sessionNet() + { + return sessionNet(PROTOCOL_VERSIONS.get(PROTOCOL_VERSIONS.size() - 1)); } + protected com.datastax.driver.core.ResultSet executeNetWithPaging(String query, int pageSize) throws Throwable + { - return sessionNet(maxProtocolVersion).execute(new SimpleStatement(formatQuery(query)).setFetchSize(pageSize)); ++ return sessionNet().execute(new SimpleStatement(formatQuery(query)).setFetchSize(pageSize)); + } + protected Session sessionNet(int protocolVersion) { requireNetwork(); @@@ -849,7 -665,12 +855,12 @@@ rows.length>i ? "less" : "more", rows.length, i, protocolVersion), i == rows.length); } + protected void assertRowsNet(ResultSet result, Object[]... rows) + { - assertRowsNet(maxProtocolVersion, result, rows); ++ assertRowsNet(PROTOCOL_VERSIONS.get(PROTOCOL_VERSIONS.size() - 1), result, rows); + } + - protected void assertRows(UntypedResultSet result, Object[]... rows) + public static void assertRows(UntypedResultSet result, Object[]... rows) { if (result == null) { http://git-wip-us.apache.org/repos/asf/cassandra/blob/97fb4d10/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java ---------------------------------------------------------------------- diff --cc test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java index 823dd90,0f29928..94b2087 --- a/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java +++ b/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java @@@ -17,12 -17,13 +17,14 @@@ */ package org.apache.cassandra.cql3.validation.entities; - import org.apache.cassandra.config.DatabaseDescriptor; import org.apache.cassandra.cql3.Json; - import org.apache.cassandra.cql3.CQLTester; import org.apache.cassandra.dht.ByteOrderedPartitioner; + import org.apache.cassandra.config.DatabaseDescriptor; + import org.apache.cassandra.cql3.CQLTester; + import org.apache.cassandra.serializers.SimpleDateSerializer; import org.apache.cassandra.serializers.TimeSerializer; ++import org.apache.cassandra.service.StorageService; import org.apache.cassandra.utils.ByteBufferUtil; import org.junit.Assert; @@@ -46,7 -43,158 +44,158 @@@ public class JsonTest extends CQLTeste @BeforeClass public static void setUp() { - DatabaseDescriptor.setPartitionerUnsafe(ByteOrderedPartitioner.instance); - DatabaseDescriptor.setPartitioner(ByteOrderedPartitioner.instance); ++ StorageService.instance.setPartitionerUnsafe(ByteOrderedPartitioner.instance); + } + + @Test + public void testSelectJsonWithPagingWithFrozenTuple() throws Throwable + { + final UUID uuid = UUID.fromString("2dd2cd62-6af3-4cf6-96fc-91b9ab62eedc"); + final Object partitionKey = tuple(uuid, 2); + + createTable("CREATE TABLE %s (k1 FROZEN<TUPLE<uuid, int>>, c1 frozen<tuple<uuid, int>>, value int, PRIMARY KEY (k1, c1))"); + + // prepare data + for (int i = 1; i < 5; i++) + execute("INSERT INTO %s (k1, c1, value) VALUES (?, ?, ?)", partitionKey, tuple(uuid, i), i); + + for (int pageSize = 1; pageSize < 6; pageSize++) + { + // SELECT JSON + assertRowsNet(executeNetWithPaging("SELECT JSON * FROM %s", pageSize), + row("{\"k1\": [\"" + uuid + "\", 2], \"c1\": [\"" + uuid + "\", 1], \"value\": 1}"), + row("{\"k1\": [\"" + uuid + "\", 2], \"c1\": [\"" + uuid + "\", 2], \"value\": 2}"), + row("{\"k1\": [\"" + uuid + "\", 2], \"c1\": [\"" + uuid + "\", 3], \"value\": 3}"), + row("{\"k1\": [\"" + uuid + "\", 2], \"c1\": [\"" + uuid + "\", 4], \"value\": 4}")); + + // SELECT toJson(column) + assertRowsNet(executeNetWithPaging("SELECT toJson(k1), toJson(c1), toJson(value) FROM %s", pageSize), + row("[\"" + uuid + "\", 2]", "[\"" + uuid + "\", 1]", "1"), + row("[\"" + uuid + "\", 2]", "[\"" + uuid + "\", 2]", "2"), + row("[\"" + uuid + "\", 2]", "[\"" + uuid + "\", 3]", "3"), + row("[\"" + uuid + "\", 2]", "[\"" + uuid + "\", 4]", "4")); + } + } + + @Test + public void testSelectJsonWithPagingWithFrozenMap() throws Throwable + { + final UUID uuid = UUID.fromString("2dd2cd62-6af3-4cf6-96fc-91b9ab62eedc"); + final Object partitionKey = map(1, tuple(uuid, 1), 2, tuple(uuid, 2)); + + createTable("CREATE TABLE %s (k1 FROZEN<map<int, tuple<uuid, int>>>, c1 frozen<tuple<uuid, int>>, value int, PRIMARY KEY (k1, c1))"); + + // prepare data + for (int i = 1; i < 5; i++) + execute("INSERT INTO %s (k1, c1, value) VALUES (?, ?, ?)", partitionKey, tuple(uuid, i), i); + + for (int pageSize = 1; pageSize < 6; pageSize++) + { + // SELECT JSON + assertRowsNet(executeNetWithPaging("SELECT JSON * FROM %s", pageSize), + row("{\"k1\": {\"1\": [\"" + uuid + "\", 1], \"2\": [\"" + uuid + "\", 2]}, \"c1\": [\"" + uuid + "\", 1], \"value\": 1}"), + row("{\"k1\": {\"1\": [\"" + uuid + "\", 1], \"2\": [\"" + uuid + "\", 2]}, \"c1\": [\"" + uuid + "\", 2], \"value\": 2}"), + row("{\"k1\": {\"1\": [\"" + uuid + "\", 1], \"2\": [\"" + uuid + "\", 2]}, \"c1\": [\"" + uuid + "\", 3], \"value\": 3}"), + row("{\"k1\": {\"1\": [\"" + uuid + "\", 1], \"2\": [\"" + uuid + "\", 2]}, \"c1\": [\"" + uuid + "\", 4], \"value\": 4}")); + + // SELECT toJson(column) + assertRowsNet(executeNetWithPaging("SELECT toJson(k1), toJson(c1), toJson(value) FROM %s", pageSize), + row("{\"1\": [\"" + uuid + "\", 1], \"2\": [\"" + uuid + "\", 2]}", "[\"" + uuid + "\", 1]", "1"), + row("{\"1\": [\"" + uuid + "\", 1], \"2\": [\"" + uuid + "\", 2]}", "[\"" + uuid + "\", 2]", "2"), + row("{\"1\": [\"" + uuid + "\", 1], \"2\": [\"" + uuid + "\", 2]}", "[\"" + uuid + "\", 3]", "3"), + row("{\"1\": [\"" + uuid + "\", 1], \"2\": [\"" + uuid + "\", 2]}", "[\"" + uuid + "\", 4]", "4")); + } + } + + @Test + public void testSelectJsonWithPagingWithFrozenSet() throws Throwable + { + final UUID uuid = UUID.fromString("2dd2cd62-6af3-4cf6-96fc-91b9ab62eedc"); + final Object partitionKey = set(tuple(list(1, 2), 1), tuple(list(2, 3), 2)); + + createTable("CREATE TABLE %s (k1 frozen<set<tuple<list<int>, int>>>, c1 frozen<tuple<uuid, int>>, value int, PRIMARY KEY (k1, c1))"); + + // prepare data + for (int i = 1; i < 5; i++) + execute("INSERT INTO %s (k1, c1, value) VALUES (?, ?, ?)", partitionKey, tuple(uuid, i), i); + + for (int pageSize = 1; pageSize < 6; pageSize++) + { + // SELECT JSON + assertRowsNet(executeNetWithPaging("SELECT JSON * FROM %s", pageSize), + row("{\"k1\": [[[1, 2], 1], [[2, 3], 2]], \"c1\": [\"" + uuid + "\", 1], \"value\": 1}"), + row("{\"k1\": [[[1, 2], 1], [[2, 3], 2]], \"c1\": [\"" + uuid + "\", 2], \"value\": 2}"), + row("{\"k1\": [[[1, 2], 1], [[2, 3], 2]], \"c1\": [\"" + uuid + "\", 3], \"value\": 3}"), + row("{\"k1\": [[[1, 2], 1], [[2, 3], 2]], \"c1\": [\"" + uuid + "\", 4], \"value\": 4}")); + + // SELECT toJson(column) + assertRowsNet(executeNetWithPaging("SELECT toJson(k1), toJson(c1), toJson(value) FROM %s", pageSize), + row("[[[1, 2], 1], [[2, 3], 2]]", "[\"" + uuid + "\", 1]", "1"), + row("[[[1, 2], 1], [[2, 3], 2]]", "[\"" + uuid + "\", 2]", "2"), + row("[[[1, 2], 1], [[2, 3], 2]]", "[\"" + uuid + "\", 3]", "3"), + row("[[[1, 2], 1], [[2, 3], 2]]", "[\"" + uuid + "\", 4]", "4")); + } + } + + @Test + public void testSelectJsonWithPagingWithFrozenList() throws Throwable + { + final UUID uuid = UUID.fromString("2dd2cd62-6af3-4cf6-96fc-91b9ab62eedc"); + final Object partitionKey = list(tuple(uuid, 2), tuple(uuid, 3)); + + createTable("CREATE TABLE %s (k1 frozen<list<tuple<uuid, int>>>, c1 frozen<tuple<uuid, int>>, value int, PRIMARY KEY (k1, c1))"); + + // prepare data + for (int i = 1; i < 5; i++) + execute("INSERT INTO %s (k1, c1, value) VALUES (?, ?, ?)", partitionKey, tuple(uuid, i), i); + + for (int pageSize = 1; pageSize < 6; pageSize++) + { + // SELECT JSON + assertRowsNet(executeNetWithPaging("SELECT JSON * FROM %s", pageSize), + row("{\"k1\": [[\"" + uuid + "\", 2], [\"" + uuid + "\", 3]], \"c1\": [\"" + uuid + "\", 1], \"value\": 1}"), + row("{\"k1\": [[\"" + uuid + "\", 2], [\"" + uuid + "\", 3]], \"c1\": [\"" + uuid + "\", 2], \"value\": 2}"), + row("{\"k1\": [[\"" + uuid + "\", 2], [\"" + uuid + "\", 3]], \"c1\": [\"" + uuid + "\", 3], \"value\": 3}"), + row("{\"k1\": [[\"" + uuid + "\", 2], [\"" + uuid + "\", 3]], \"c1\": [\"" + uuid + "\", 4], \"value\": 4}")); + + // SELECT toJson(column) + assertRowsNet(executeNetWithPaging("SELECT toJson(k1), toJson(c1), toJson(value) FROM %s", pageSize), + row("[[\"" + uuid + "\", 2], [\"" + uuid + "\", 3]]", "[\"" + uuid + "\", 1]", "1"), + row("[[\"" + uuid + "\", 2], [\"" + uuid + "\", 3]]", "[\"" + uuid + "\", 2]", "2"), + row("[[\"" + uuid + "\", 2], [\"" + uuid + "\", 3]]", "[\"" + uuid + "\", 3]", "3"), + row("[[\"" + uuid + "\", 2], [\"" + uuid + "\", 3]]", "[\"" + uuid + "\", 4]", "4")); + } + } + + @Test + public void testSelectJsonWithPagingWithFrozenUDT() throws Throwable + { + final UUID uuid = UUID.fromString("2dd2cd62-6af3-4cf6-96fc-91b9ab62eedc"); + final Object partitionKey = userType(1, 2, list("1", "2")); + + String typeName = createType("CREATE TYPE %s (a int, b int, c list<text>)"); + createTable("CREATE TABLE %s (k1 frozen<" + typeName + ">, c1 frozen<tuple<uuid, int>>, value int, PRIMARY KEY (k1, c1))"); + + // prepare data + for (int i = 1; i < 5; i++) + execute("INSERT INTO %s (k1, c1, value) VALUES (?, ?, ?)", partitionKey, tuple(uuid, i), i); + + for (int pageSize = 1; pageSize < 6; pageSize++) + { + // SELECT JSON + assertRowsNet(executeNetWithPaging("SELECT JSON * FROM %s", pageSize), + row("{\"k1\": {\"a\": 1, \"b\": 2, \"c\": [\"1\", \"2\"]}, \"c1\": [\"" + uuid + "\", 1], \"value\": 1}"), + row("{\"k1\": {\"a\": 1, \"b\": 2, \"c\": [\"1\", \"2\"]}, \"c1\": [\"" + uuid + "\", 2], \"value\": 2}"), + row("{\"k1\": {\"a\": 1, \"b\": 2, \"c\": [\"1\", \"2\"]}, \"c1\": [\"" + uuid + "\", 3], \"value\": 3}"), + row("{\"k1\": {\"a\": 1, \"b\": 2, \"c\": [\"1\", \"2\"]}, \"c1\": [\"" + uuid + "\", 4], \"value\": 4}")); + + // SELECT toJson(column) + assertRowsNet(executeNetWithPaging("SELECT toJson(k1), toJson(c1), toJson(value) FROM %s", pageSize), + row("{\"a\": 1, \"b\": 2, \"c\": [\"1\", \"2\"]}", "[\"" + uuid + "\", 1]", "1"), + row("{\"a\": 1, \"b\": 2, \"c\": [\"1\", \"2\"]}", "[\"" + uuid + "\", 2]", "2"), + row("{\"a\": 1, \"b\": 2, \"c\": [\"1\", \"2\"]}", "[\"" + uuid + "\", 3]", "3"), + row("{\"a\": 1, \"b\": 2, \"c\": [\"1\", \"2\"]}", "[\"" + uuid + "\", 4]", "4")); + } } @Test --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
