[GitHub] cassandra pull request #252: Reduce heap pressure during compactions for CAS...
Github user krummas commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/252#discussion_r224459241
--- Diff: src/java/org/apache/cassandra/io/sstable/SSTableRewriter.java ---
@@ -131,16 +132,20 @@ public RowIndexEntry append(UnfilteredRowIterator
partition)
{
// we do this before appending to ensure we can resetAndTruncate()
safely if the append fails
DecoratedKey key = partition.partitionKey();
-maybeReopenEarly(key);
+if (preemptiveOpenInterval != Long.MAX_VALUE)
--- End diff --
this is checked in `maybeReopenEarly` - no need to check here as well
---
-
To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org
For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #252: Reduce heap pressure during compactions for CAS...
Github user krummas commented on a diff in the pull request: https://github.com/apache/cassandra/pull/252#discussion_r224695370 --- Diff: src/java/org/apache/cassandra/config/Config.java --- @@ -238,6 +238,7 @@ public int hints_flush_period_in_ms = 1; public int max_hints_file_size_in_mb = 128; public ParameterizedClass hints_compression; +public boolean invalidate_cache_on_compaction = true; --- End diff -- this should be volatile for hotprops --- - To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #252: Reduce heap pressure during compactions for CAS...
Github user krummas commented on a diff in the pull request: https://github.com/apache/cassandra/pull/252#discussion_r224696746 --- Diff: src/java/org/apache/cassandra/config/Config.java --- @@ -238,6 +238,7 @@ public int hints_flush_period_in_ms = 1; public int max_hints_file_size_in_mb = 128; public ParameterizedClass hints_compression; +public boolean invalidate_cache_on_compaction = true; --- End diff -- The biggest performance impact of setting this to `false` is probably not the fact that we don't invalidate the cache anymore, but that we don't warm the cache up for the new sstables - maybe we should rename the property? (warm_keycache_on_compaction or migrate_keycache_on_compaction?) Or at least add a comment about it? And it should probably say "keycache" instead of "cache" --- - To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #276: Repair job tests
Github user krummas commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/276#discussion_r222596795
--- Diff: test/unit/org/apache/cassandra/repair/RepairJobTest.java ---
@@ -0,0 +1,569 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair;
+
+import java.net.UnknownHostException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Predicate;
+
+import com.google.common.collect.Sets;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.dht.ByteOrderedPartitioner;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.streaming.PreviewKind;
+import org.apache.cassandra.utils.ByteBufferUtil;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.MerkleTree;
+import org.apache.cassandra.utils.MerkleTrees;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+public class RepairJobTest
+{
+private static final IPartitioner PARTITIONER =
ByteOrderedPartitioner.instance;
+
+static InetAddressAndPort addr1;
+static InetAddressAndPort addr2;
+static InetAddressAndPort addr3;
+static InetAddressAndPort addr4;
+static InetAddressAndPort addr5;
+
+static Range range1 = range(0, 1);
+static Range range2 = range(2, 3);
+static Range range3 = range(4, 5);
+static RepairJobDesc desc = new RepairJobDesc(UUID.randomUUID(),
UUID.randomUUID(), "ks", "cf", Arrays.asList());
+
+@AfterClass
+public static void reset()
+{
+FBUtilities.reset();
+}
+
+static
+{
+try
+{
+addr1 = InetAddressAndPort.getByName("127.0.0.1");
+addr2 = InetAddressAndPort.getByName("127.0.0.2");
+addr3 = InetAddressAndPort.getByName("127.0.0.3");
+addr4 = InetAddressAndPort.getByName("127.0.0.4");
+addr5 = InetAddressAndPort.getByName("127.0.0.5");
+DatabaseDescriptor.setBroadcastAddress(addr1.address);
+}
+catch (UnknownHostException e)
+{
+e.printStackTrace();
+}
+}
+
+@Test
+public void testCreateStandardSyncTasks()
+{
+testCreateStandardSyncTasks(false);
+}
+
+@Test
+public void testCreateStandardSyncTasksPullRepair()
+{
+testCreateStandardSyncTasks(true);
+}
+
+public static void testCreateStandardSyncTasks(boolean pullRepair)
+{
+List treeResponses =
Arrays.asList(treeResponse(addr1, range1, "same", range2, "same", range3,
"same"),
+
treeResponse(addr2, range1, "different", range2, "same", range3, "different"),
+
treeResponse(addr3, range1, "same", range2, "same", range3, "same"));
+
+Map tasks =
toMap(RepairJob.createStandardSyncTasks(desc,
+
treeResponses,
+
[GitHub] cassandra pull request #276: Repair job tests
Github user krummas commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/276#discussion_r222610027
--- Diff: test/unit/org/apache/cassandra/repair/RepairJobTest.java ---
@@ -0,0 +1,569 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair;
+
+import java.net.UnknownHostException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Predicate;
+
+import com.google.common.collect.Sets;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.dht.ByteOrderedPartitioner;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.streaming.PreviewKind;
+import org.apache.cassandra.utils.ByteBufferUtil;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.MerkleTree;
+import org.apache.cassandra.utils.MerkleTrees;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+public class RepairJobTest
+{
+private static final IPartitioner PARTITIONER =
ByteOrderedPartitioner.instance;
+
+static InetAddressAndPort addr1;
+static InetAddressAndPort addr2;
+static InetAddressAndPort addr3;
+static InetAddressAndPort addr4;
+static InetAddressAndPort addr5;
+
+static Range range1 = range(0, 1);
+static Range range2 = range(2, 3);
+static Range range3 = range(4, 5);
+static RepairJobDesc desc = new RepairJobDesc(UUID.randomUUID(),
UUID.randomUUID(), "ks", "cf", Arrays.asList());
+
+@AfterClass
+public static void reset()
+{
+FBUtilities.reset();
+}
+
+static
+{
+try
+{
+addr1 = InetAddressAndPort.getByName("127.0.0.1");
+addr2 = InetAddressAndPort.getByName("127.0.0.2");
+addr3 = InetAddressAndPort.getByName("127.0.0.3");
+addr4 = InetAddressAndPort.getByName("127.0.0.4");
+addr5 = InetAddressAndPort.getByName("127.0.0.5");
+DatabaseDescriptor.setBroadcastAddress(addr1.address);
+}
+catch (UnknownHostException e)
+{
+e.printStackTrace();
+}
+}
+
+@Test
+public void testCreateStandardSyncTasks()
+{
+testCreateStandardSyncTasks(false);
+}
+
+@Test
+public void testCreateStandardSyncTasksPullRepair()
+{
+testCreateStandardSyncTasks(true);
+}
+
+public static void testCreateStandardSyncTasks(boolean pullRepair)
+{
+List treeResponses =
Arrays.asList(treeResponse(addr1, range1, "same", range2, "same", range3,
"same"),
+
treeResponse(addr2, range1, "different", range2, "same", range3, "different"),
+
treeResponse(addr3, range1, "same", range2, "same", range3, "same"));
+
+Map tasks =
toMap(RepairJob.createStandardSyncTasks(desc,
+
treeResponses,
+
[GitHub] cassandra pull request #276: Repair job tests
Github user krummas commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/276#discussion_r222600025
--- Diff: test/unit/org/apache/cassandra/repair/RepairJobTest.java ---
@@ -0,0 +1,569 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair;
+
+import java.net.UnknownHostException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Predicate;
+
+import com.google.common.collect.Sets;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.dht.ByteOrderedPartitioner;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.streaming.PreviewKind;
+import org.apache.cassandra.utils.ByteBufferUtil;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.MerkleTree;
+import org.apache.cassandra.utils.MerkleTrees;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+public class RepairJobTest
+{
+private static final IPartitioner PARTITIONER =
ByteOrderedPartitioner.instance;
+
+static InetAddressAndPort addr1;
+static InetAddressAndPort addr2;
+static InetAddressAndPort addr3;
+static InetAddressAndPort addr4;
+static InetAddressAndPort addr5;
+
+static Range range1 = range(0, 1);
+static Range range2 = range(2, 3);
+static Range range3 = range(4, 5);
+static RepairJobDesc desc = new RepairJobDesc(UUID.randomUUID(),
UUID.randomUUID(), "ks", "cf", Arrays.asList());
+
+@AfterClass
+public static void reset()
+{
+FBUtilities.reset();
+}
+
+static
+{
+try
+{
+addr1 = InetAddressAndPort.getByName("127.0.0.1");
+addr2 = InetAddressAndPort.getByName("127.0.0.2");
+addr3 = InetAddressAndPort.getByName("127.0.0.3");
+addr4 = InetAddressAndPort.getByName("127.0.0.4");
+addr5 = InetAddressAndPort.getByName("127.0.0.5");
+DatabaseDescriptor.setBroadcastAddress(addr1.address);
+}
+catch (UnknownHostException e)
+{
+e.printStackTrace();
+}
+}
+
+@Test
+public void testCreateStandardSyncTasks()
+{
+testCreateStandardSyncTasks(false);
+}
+
+@Test
+public void testCreateStandardSyncTasksPullRepair()
+{
+testCreateStandardSyncTasks(true);
+}
+
+public static void testCreateStandardSyncTasks(boolean pullRepair)
+{
+List treeResponses =
Arrays.asList(treeResponse(addr1, range1, "same", range2, "same", range3,
"same"),
+
treeResponse(addr2, range1, "different", range2, "same", range3, "different"),
+
treeResponse(addr3, range1, "same", range2, "same", range3, "same"));
+
+Map tasks =
toMap(RepairJob.createStandardSyncTasks(desc,
+
treeResponses,
+
[GitHub] cassandra pull request #276: Repair job tests
Github user krummas commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/276#discussion_r222609899
--- Diff: test/unit/org/apache/cassandra/repair/RepairJobTest.java ---
@@ -0,0 +1,569 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair;
+
+import java.net.UnknownHostException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Predicate;
+
+import com.google.common.collect.Sets;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.dht.ByteOrderedPartitioner;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.streaming.PreviewKind;
+import org.apache.cassandra.utils.ByteBufferUtil;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.MerkleTree;
+import org.apache.cassandra.utils.MerkleTrees;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+public class RepairJobTest
+{
+private static final IPartitioner PARTITIONER =
ByteOrderedPartitioner.instance;
+
+static InetAddressAndPort addr1;
+static InetAddressAndPort addr2;
+static InetAddressAndPort addr3;
+static InetAddressAndPort addr4;
+static InetAddressAndPort addr5;
+
+static Range range1 = range(0, 1);
+static Range range2 = range(2, 3);
+static Range range3 = range(4, 5);
+static RepairJobDesc desc = new RepairJobDesc(UUID.randomUUID(),
UUID.randomUUID(), "ks", "cf", Arrays.asList());
+
+@AfterClass
+public static void reset()
+{
+FBUtilities.reset();
+}
+
+static
+{
+try
+{
+addr1 = InetAddressAndPort.getByName("127.0.0.1");
+addr2 = InetAddressAndPort.getByName("127.0.0.2");
+addr3 = InetAddressAndPort.getByName("127.0.0.3");
+addr4 = InetAddressAndPort.getByName("127.0.0.4");
+addr5 = InetAddressAndPort.getByName("127.0.0.5");
+DatabaseDescriptor.setBroadcastAddress(addr1.address);
+}
+catch (UnknownHostException e)
+{
+e.printStackTrace();
+}
+}
+
+@Test
+public void testCreateStandardSyncTasks()
+{
+testCreateStandardSyncTasks(false);
+}
+
+@Test
+public void testCreateStandardSyncTasksPullRepair()
+{
+testCreateStandardSyncTasks(true);
+}
+
+public static void testCreateStandardSyncTasks(boolean pullRepair)
+{
+List treeResponses =
Arrays.asList(treeResponse(addr1, range1, "same", range2, "same", range3,
"same"),
+
treeResponse(addr2, range1, "different", range2, "same", range3, "different"),
+
treeResponse(addr3, range1, "same", range2, "same", range3, "same"));
+
+Map tasks =
toMap(RepairJob.createStandardSyncTasks(desc,
+
treeResponses,
+
[GitHub] cassandra pull request #276: Repair job tests
Github user krummas commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/276#discussion_r222603722
--- Diff: test/unit/org/apache/cassandra/repair/RepairJobTest.java ---
@@ -0,0 +1,569 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair;
+
+import java.net.UnknownHostException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Predicate;
+
+import com.google.common.collect.Sets;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.dht.ByteOrderedPartitioner;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.streaming.PreviewKind;
+import org.apache.cassandra.utils.ByteBufferUtil;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.MerkleTree;
+import org.apache.cassandra.utils.MerkleTrees;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+public class RepairJobTest
+{
+private static final IPartitioner PARTITIONER =
ByteOrderedPartitioner.instance;
+
+static InetAddressAndPort addr1;
+static InetAddressAndPort addr2;
+static InetAddressAndPort addr3;
+static InetAddressAndPort addr4;
+static InetAddressAndPort addr5;
+
+static Range range1 = range(0, 1);
+static Range range2 = range(2, 3);
+static Range range3 = range(4, 5);
+static RepairJobDesc desc = new RepairJobDesc(UUID.randomUUID(),
UUID.randomUUID(), "ks", "cf", Arrays.asList());
+
+@AfterClass
+public static void reset()
+{
+FBUtilities.reset();
+}
+
+static
+{
+try
+{
+addr1 = InetAddressAndPort.getByName("127.0.0.1");
+addr2 = InetAddressAndPort.getByName("127.0.0.2");
+addr3 = InetAddressAndPort.getByName("127.0.0.3");
+addr4 = InetAddressAndPort.getByName("127.0.0.4");
+addr5 = InetAddressAndPort.getByName("127.0.0.5");
+DatabaseDescriptor.setBroadcastAddress(addr1.address);
+}
+catch (UnknownHostException e)
+{
+e.printStackTrace();
+}
+}
+
+@Test
+public void testCreateStandardSyncTasks()
+{
+testCreateStandardSyncTasks(false);
+}
+
+@Test
+public void testCreateStandardSyncTasksPullRepair()
+{
+testCreateStandardSyncTasks(true);
+}
+
+public static void testCreateStandardSyncTasks(boolean pullRepair)
+{
+List treeResponses =
Arrays.asList(treeResponse(addr1, range1, "same", range2, "same", range3,
"same"),
+
treeResponse(addr2, range1, "different", range2, "same", range3, "different"),
+
treeResponse(addr3, range1, "same", range2, "same", range3, "same"));
+
+Map tasks =
toMap(RepairJob.createStandardSyncTasks(desc,
+
treeResponses,
+
[GitHub] cassandra pull request #276: Repair job tests
Github user krummas commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/276#discussion_r222566547
--- Diff: test/unit/org/apache/cassandra/repair/RepairJobTest.java ---
@@ -0,0 +1,569 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair;
+
+import java.net.UnknownHostException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.Predicate;
+
+import com.google.common.collect.Sets;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.dht.ByteOrderedPartitioner;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.streaming.PreviewKind;
+import org.apache.cassandra.utils.ByteBufferUtil;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.MerkleTree;
+import org.apache.cassandra.utils.MerkleTrees;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+public class RepairJobTest
+{
+private static final IPartitioner PARTITIONER =
ByteOrderedPartitioner.instance;
+
+static InetAddressAndPort addr1;
+static InetAddressAndPort addr2;
+static InetAddressAndPort addr3;
+static InetAddressAndPort addr4;
+static InetAddressAndPort addr5;
+
+static Range range1 = range(0, 1);
+static Range range2 = range(2, 3);
+static Range range3 = range(4, 5);
+static RepairJobDesc desc = new RepairJobDesc(UUID.randomUUID(),
UUID.randomUUID(), "ks", "cf", Arrays.asList());
+
+@AfterClass
+public static void reset()
+{
+FBUtilities.reset();
+}
+
+static
+{
+try
+{
+addr1 = InetAddressAndPort.getByName("127.0.0.1");
+addr2 = InetAddressAndPort.getByName("127.0.0.2");
+addr3 = InetAddressAndPort.getByName("127.0.0.3");
+addr4 = InetAddressAndPort.getByName("127.0.0.4");
+addr5 = InetAddressAndPort.getByName("127.0.0.5");
+DatabaseDescriptor.setBroadcastAddress(addr1.address);
+}
+catch (UnknownHostException e)
+{
+e.printStackTrace();
+}
+}
+
+@Test
+public void testCreateStandardSyncTasks()
+{
+testCreateStandardSyncTasks(false);
+}
+
+@Test
+public void testCreateStandardSyncTasksPullRepair()
+{
+testCreateStandardSyncTasks(true);
+}
+
+public static void testCreateStandardSyncTasks(boolean pullRepair)
+{
+List treeResponses =
Arrays.asList(treeResponse(addr1, range1, "same", range2, "same", range3,
"same"),
+
treeResponse(addr2, range1, "different", range2, "same", range3, "different"),
+
treeResponse(addr3, range1, "same", range2, "same", range3, "same"));
+
+Map tasks =
toMap(RepairJob.createStandardSyncTasks(desc,
+
treeResponses,
+
[GitHub] cassandra pull request #276: Repair job tests
Github user krummas commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/276#discussion_r222552355
--- Diff: src/java/org/apache/cassandra/repair/LocalSyncTask.java ---
@@ -52,15 +52,9 @@
private static final Logger logger =
LoggerFactory.getLogger(LocalSyncTask.class);
private final UUID pendingRepair;
-private final boolean requestRanges;
-private final boolean transferRanges;
-public LocalSyncTask(RepairJobDesc desc, TreeResponse local,
TreeResponse remote, UUID pendingRepair,
- boolean requestRanges, boolean transferRanges,
PreviewKind previewKind)
-{
-this(desc, local.endpoint, remote.endpoint,
MerkleTrees.difference(local.trees, remote.trees),
- pendingRepair, requestRanges, transferRanges, previewKind);
-}
+protected final boolean requestRanges;
--- End diff --
these could be VisibleForTesting (and perhaps package private)
---
-
To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org
For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #255: Marcuse/14618
Github user krummas commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/255#discussion_r213572518
--- Diff: src/java/org/apache/cassandra/tools/fqltool/FQLQuery.java ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.tools.fqltool;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+import com.google.common.primitives.Longs;
+import com.google.common.util.concurrent.FluentFuture;
+import com.google.common.util.concurrent.Futures;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.MoreExecutors;
+
+import com.datastax.driver.core.BatchStatement;
+import com.datastax.driver.core.ConsistencyLevel;
+import com.datastax.driver.core.ResultSet;
+import com.datastax.driver.core.ResultSetFuture;
+import com.datastax.driver.core.Session;
+import com.datastax.driver.core.SimpleStatement;
+import org.apache.cassandra.audit.FullQueryLogger;
+import org.apache.cassandra.cql3.QueryOptions;
+import org.apache.cassandra.utils.ByteBufferUtil;
+import org.apache.cassandra.utils.binlog.BinLog;
+
+public abstract class FQLQuery implements Comparable
+{
+public final long queryTime;
+public final QueryOptions queryOptions;
+public final int protocolVersion;
+public final String keyspace;
+
+public FQLQuery(String keyspace, int protocolVersion, QueryOptions
queryOptions, long queryTime)
--- End diff --
not really, it is stored as an int in the fql files and we don't use it
(yet)
---
-
To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org
For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #255: Marcuse/14618
Github user krummas commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/255#discussion_r213570103
--- Diff: src/java/org/apache/cassandra/tools/fqltool/FQLQuery.java ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.tools.fqltool;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+import com.google.common.primitives.Longs;
+import com.google.common.util.concurrent.FluentFuture;
+import com.google.common.util.concurrent.Futures;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.MoreExecutors;
+
+import com.datastax.driver.core.BatchStatement;
+import com.datastax.driver.core.ConsistencyLevel;
+import com.datastax.driver.core.ResultSet;
+import com.datastax.driver.core.ResultSetFuture;
+import com.datastax.driver.core.Session;
+import com.datastax.driver.core.SimpleStatement;
+import org.apache.cassandra.audit.FullQueryLogger;
+import org.apache.cassandra.cql3.QueryOptions;
+import org.apache.cassandra.utils.ByteBufferUtil;
+import org.apache.cassandra.utils.binlog.BinLog;
+
+public abstract class FQLQuery implements Comparable
+{
+public final long queryTime;
+public final QueryOptions queryOptions;
+public final int protocolVersion;
+public final String keyspace;
+
+public FQLQuery(String keyspace, int protocolVersion, QueryOptions
queryOptions, long queryTime)
+{
+this.queryTime = queryTime;
+this.queryOptions = queryOptions;
+this.protocolVersion = protocolVersion;
+this.keyspace = keyspace;
+}
+
+public abstract ListenableFuture
execute(Session session);
+
+/**
+ * used when storing the queries executed
+ */
+public abstract BinLog.ReleaseableWriteMarshallable toMarshallable();
+
+/**
+ * Make sure we catch any query errors
+ *
+ * On error, this creates a failed ComparableResultSet with the
exception set to be able to store
+ * this fact in the result file and handle comparison of failed result
sets.
+ */
+ListenableFuture
handleErrors(ListenableFuture result)
+{
+FluentFuture fluentFuture =
FluentFuture.from(result)
+
.transform(DriverResultSet::new, MoreExecutors.directExecutor());
+return fluentFuture.catching(Throwable.class,
DriverResultSet::failed, MoreExecutors.directExecutor());
+}
+
+public boolean equals(Object o)
+{
+if (this == o) return true;
+if (!(o instanceof FQLQuery)) return false;
+FQLQuery fqlQuery = (FQLQuery) o;
+return queryTime == fqlQuery.queryTime &&
+ protocolVersion == fqlQuery.protocolVersion &&
+
queryOptions.getValues().equals(fqlQuery.queryOptions.getValues()) &&
+ Objects.equals(keyspace, fqlQuery.keyspace);
+}
+
+public int hashCode()
+{
+return Objects.hash(queryTime, queryOptions, protocolVersion,
keyspace);
+}
+
+public int compareTo(FQLQuery other)
+{
+return Longs.compare(queryTime, other.queryTime);
+}
+
+public static class Single extends FQLQuery
+{
+public final String query;
+public final List values;
+
+public Single(String keyspace, int protocolVersion, QueryOptions
queryOptions, long queryTime, String queryString, List values)
+{
+super(keyspace, protocolVersion, queryOptions, queryTime);
+this.query = queryString;
+this.values = values;
+}
[GitHub] cassandra pull request #255: Marcuse/14618
Github user krummas commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/255#discussion_r213569491
--- Diff: src/java/org/apache/cassandra/tools/fqltool/commands/Replay.java
---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.tools.fqltool.commands;
+
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import io.airlift.airline.Arguments;
+import io.airlift.airline.Command;
+import io.airlift.airline.Option;
+import net.openhft.chronicle.core.io.Closeable;
+import net.openhft.chronicle.queue.ChronicleQueue;
+import net.openhft.chronicle.queue.ChronicleQueueBuilder;
+
+import org.apache.cassandra.tools.fqltool.FQLQuery;
+import org.apache.cassandra.tools.fqltool.FQLQueryIterator;
+import org.apache.cassandra.tools.fqltool.QueryReplayer;
+import org.apache.cassandra.utils.AbstractIterator;
+import org.apache.cassandra.utils.MergeIterator;
+
+/**
+ * replay the contents of a list of paths containing full query logs
+ */
+@Command(name = "replay", description = "Replay full query logs")
+public class Replay implements Runnable
+{
+@Arguments(usage = " [...]", description = "Paths
containing the full query logs to replay.", required = true)
+private List arguments = new ArrayList<>();
+
+@Option(title = "target", name = {"--target"}, description = "Hosts to
replay the logs to, can be repeated to replay to more hosts.")
+private List targetHosts;
+
+@Option(title = "results", name = { "--results"}, description = "Where
to store the results of the queries, this should be a directory. Leave this
option out to avoid storing results.")
+private String resultPath;
+
+@Option(title = "keyspace", name = { "--keyspace"}, description =
"Only replay queries against this keyspace and queries without keyspace set.")
+private String keyspace;
+
+@Option(title = "debug", name = {"--debug"}, description = "Debug
mode, print all queries executed.")
+private boolean debug;
+
+@Option(title = "use", name = {"--use"}, description = "Connect to the
cluster(s) using this keyspace.")
--- End diff --
this was a left over from before we had keyspace in the full query logs,
removed
---
-
To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org
For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #255: Marcuse/14618
Github user krummas commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/255#discussion_r213565321
--- Diff: src/java/org/apache/cassandra/tools/fqltool/commands/Replay.java
---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.tools.fqltool.commands;
+
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import io.airlift.airline.Arguments;
+import io.airlift.airline.Command;
+import io.airlift.airline.Option;
+import net.openhft.chronicle.core.io.Closeable;
+import net.openhft.chronicle.queue.ChronicleQueue;
+import net.openhft.chronicle.queue.ChronicleQueueBuilder;
+
+import org.apache.cassandra.tools.fqltool.FQLQuery;
+import org.apache.cassandra.tools.fqltool.FQLQueryIterator;
+import org.apache.cassandra.tools.fqltool.QueryReplayer;
+import org.apache.cassandra.utils.AbstractIterator;
+import org.apache.cassandra.utils.MergeIterator;
+
+/**
+ * replay the contents of a list of paths containing full query logs
+ */
+@Command(name = "replay", description = "Replay full query logs")
+public class Replay implements Runnable
+{
+@Arguments(usage = " [...]", description = "Paths
containing the full query logs to replay.", required = true)
+private List arguments = new ArrayList<>();
+
+@Option(title = "target", name = {"--target"}, description = "Hosts to
replay the logs to, can be repeated to replay to more hosts.")
+private List targetHosts;
+
+@Option(title = "results", name = { "--results"}, description = "Where
to store the results of the queries, this should be a directory. Leave this
option out to avoid storing results.")
+private String resultPath;
+
+@Option(title = "keyspace", name = { "--keyspace"}, description =
"Only replay queries against this keyspace and queries without keyspace set.")
+private String keyspace;
+
+@Option(title = "debug", name = {"--debug"}, description = "Debug
mode, print all queries executed.")
+private boolean debug;
+
+@Option(title = "use", name = {"--use"}, description = "Connect to the
cluster(s) using this keyspace.")
+private String useKeyspace;
+
+@Option(title = "legacy", name = {"--legacyfiles"}, description = "If
the FQL files don't contain keyspace information.")
--- End diff --
no we don't need this here since full query logs have not really been
released yet, I'll remove it
---
-
To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org
For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #255: Marcuse/14618
Github user krummas commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/255#discussion_r213565376
--- Diff: src/java/org/apache/cassandra/tools/fqltool/commands/Replay.java
---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.tools.fqltool.commands;
+
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import io.airlift.airline.Arguments;
+import io.airlift.airline.Command;
+import io.airlift.airline.Option;
+import net.openhft.chronicle.core.io.Closeable;
+import net.openhft.chronicle.queue.ChronicleQueue;
+import net.openhft.chronicle.queue.ChronicleQueueBuilder;
+
+import org.apache.cassandra.tools.fqltool.FQLQuery;
+import org.apache.cassandra.tools.fqltool.FQLQueryIterator;
+import org.apache.cassandra.tools.fqltool.QueryReplayer;
+import org.apache.cassandra.utils.AbstractIterator;
+import org.apache.cassandra.utils.MergeIterator;
+
+/**
+ * replay the contents of a list of paths containing full query logs
+ */
+@Command(name = "replay", description = "Replay full query logs")
+public class Replay implements Runnable
+{
+@Arguments(usage = " [...]", description = "Paths
containing the full query logs to replay.", required = true)
+private List arguments = new ArrayList<>();
+
+@Option(title = "target", name = {"--target"}, description = "Hosts to
replay the logs to, can be repeated to replay to more hosts.")
+private List targetHosts;
+
+@Option(title = "results", name = { "--results"}, description = "Where
to store the results of the queries, this should be a directory. Leave this
option out to avoid storing results.")
+private String resultPath;
+
+@Option(title = "keyspace", name = { "--keyspace"}, description =
"Only replay queries against this keyspace and queries without keyspace set.")
+private String keyspace;
+
+@Option(title = "debug", name = {"--debug"}, description = "Debug
mode, print all queries executed.")
+private boolean debug;
+
+@Option(title = "use", name = {"--use"}, description = "Connect to the
cluster(s) using this keyspace.")
+private String useKeyspace;
+
+@Option(title = "legacy", name = {"--legacyfiles"}, description = "If
the FQL files don't contain keyspace information.")
--- End diff --
and yes, at some point we should add versioning to the full query logs
---
-
To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org
For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra-dtest pull request #32: compact storage when testing thrift
Github user krummas closed the pull request at: https://github.com/apache/cassandra-dtest/pull/32 --- - To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra-dtest pull request #32: compact storage when testing thrift
GitHub user krummas opened a pull request: https://github.com/apache/cassandra-dtest/pull/32 compact storage when testing thrift Patch by marcuse; reviewed by Jason Brown for CASSANDRA-14583 You can merge this pull request into a Git repository by running: $ git pull https://github.com/krummas/cassandra-dtest marcuse/14583 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cassandra-dtest/pull/32.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #32 commit 194ad8a22315f0410155ab5eb2283d006a4fdd37 Author: Marcus Eriksson Date: 2018-07-23T08:03:17Z compact storage when testing thrift Patch by marcuse; reviewed by Jason Brown for CASSANDRA-14583 --- - To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra-dtest pull request #30: CASSANDRA-14467: always enable tombstone v...
GitHub user krummas opened a pull request: https://github.com/apache/cassandra-dtest/pull/30 CASSANDRA-14467: always enable tombstone validation exceptions during tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/krummas/cassandra-dtest marcuse/14467 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cassandra-dtest/pull/30.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #30 commit 3dc02646948120e9847347f951d1539c16cef2a9 Author: Marcus Eriksson Date: 2018-05-31T06:41:11Z always enable tombstone validation exceptions during tests --- - To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #:
Github user krummas commented on the pull request: https://github.com/apache/cassandra/commit/74a0728106e8d61340a4d3cd2a34d0c87118529b#commitcomment-29253214 In src/java/org/apache/cassandra/config/DatabaseDescriptor.java: In src/java/org/apache/cassandra/config/DatabaseDescriptor.java on line 137: haha, leftover, will remove --- - To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org For additional commands, e-mail: pr-h...@cassandra.apache.org
