[jira] [Created] (CASSANDRA-13557) allow different NUMACTL_ARGS to be passed in

2017-05-26 Thread Matt Byrd (JIRA) 
Matt Byrd created CASSANDRA-13557:
-

 Summary: allow different NUMACTL_ARGS to be passed in
 Key: CASSANDRA-13557
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13557
 Project: Cassandra
  Issue Type: Improvement
  Components: Configuration
Reporter: Matt Byrd
Assignee: Matt Byrd
Priority: Minor
 Fix For: 4.x


Currently in bin/cassandra the following is hardcoded:
NUMACTL_ARGS="--interleave=all"
Ideally users of cassandra/bin could pass in a different set of NUMACTL_ARGS if 
they wanted to say bind the process to a socket for cpu/memory reasons, rather 
than having to comment out/modify this line in the deployed cassandra/bin. e.g 
as described in:
https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html

This could be done by just having the default be set to "--interleave=all" but 
pickup any value which has already been set for the variable NUMACTL_ARGS.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-11381) Node running with join_ring=false and authentication can not serve requests

2017-05-26 Thread Joel Knighton (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026830#comment-16026830
 ] 

Joel Knighton commented on CASSANDRA-11381:
---

Thanks - this looks just about good. A few comments:
* The 3.0 branch looks like it is an older version of the patch than the 2.2, 
3.11, and trunk patches - it's missing the atomic guard ensuring we only run 
the set up one. Is this just an oversight?
* The new exception looks good, but the condition is too restrictive. We'll 
only hit an error when there are no tokens in the ring; the attached patch 
fails whenever the local node has no tokens, even if there are other tokens in 
the ring. Changing the condition to something like 
{{StorageService.instance.getTokenMetadata().sortedTokens().isEmpty()}} should 
suffice.

It would be great it we could cut down the time the new tests take to run. I 
have a few suggestions that I'll post on the dtest PR once this is ready to go.

> Node running with join_ring=false and authentication can not serve requests
> ---
>
> Key: CASSANDRA-11381
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11381
> Project: Cassandra
>  Issue Type: Bug
>Reporter: mck
>Assignee: mck
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
>
> Starting up a node with {{-Dcassandra.join_ring=false}} in a cluster that has 
> authentication configured, eg PasswordAuthenticator, won't be able to serve 
> requests. This is because {{Auth.setup()}} never gets called during the 
> startup.
> Without {{Auth.setup()}} having been called in {{StorageService}} clients 
> connecting to the node fail with the node throwing
> {noformat}
> java.lang.NullPointerException
> at 
> org.apache.cassandra.auth.PasswordAuthenticator.authenticate(PasswordAuthenticator.java:119)
> at 
> org.apache.cassandra.thrift.CassandraServer.login(CassandraServer.java:1471)
> at 
> org.apache.cassandra.thrift.Cassandra$Processor$login.getResult(Cassandra.java:3505)
> at 
> org.apache.cassandra.thrift.Cassandra$Processor$login.getResult(Cassandra.java:3489)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at com.thinkaurelius.thrift.Message.invoke(Message.java:314)
> at 
> com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90)
> at 
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
> at 
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
> at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The exception thrown from the 
> [code|https://github.com/apache/cassandra/blob/cassandra-2.0.16/src/java/org/apache/cassandra/auth/PasswordAuthenticator.java#L119]
> {code}
> ResultMessage.Rows rows = 
> authenticateStatement.execute(QueryState.forInternalCalls(), new 
> QueryOptions(consistencyForUser(username),
>   
>Lists.newArrayList(ByteBufferUtil.bytes(username;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13533) ColumnIdentifier object size wrong when tables are not flushed

2017-05-26 Thread Joel Knighton (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Knighton updated CASSANDRA-13533:
--
Status: Ready to Commit  (was: Patch Available)

> ColumnIdentifier object size wrong when tables are not flushed
> --
>
> Key: CASSANDRA-13533
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13533
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Eduard Tudenhoefner
>Assignee: Eduard Tudenhoefner
> Fix For: 3.0.14, 3.11.0, 4.0
>
> Attachments: columnidentifier.png
>
>
> It turns out that the object size of {{ColumnIdentifier}} is wrong when 
> *cassandra.test.flush_local_schema_changes: false*. This looks like stuff is 
> being wrongly reused when no flush is happening.
> We only noticed this because we were using the prepared stmt cache and 
> noticed that prepared statements would account for *1-6mb* when 
> *cassandra.test.flush_local_schema_changes: false*. With 
> *cassandra.test.flush_local_schema_changes: true* (which is the default) 
> those would be around *5000 bytes*.
> Attached is a test that reproduces the problem and also a fix.
> Also after talking to [~jkni] / [~blerer] we shouldn't probably take 
> {{ColumnDefinition}} into account when measuring object sizes with 
> {{MemoryMeter}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13533) ColumnIdentifier object size wrong when tables are not flushed

2017-05-26 Thread Joel Knighton (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Knighton updated CASSANDRA-13533:
--
Resolution: Fixed
Status: Resolved  (was: Ready to Commit)

> ColumnIdentifier object size wrong when tables are not flushed
> --
>
> Key: CASSANDRA-13533
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13533
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Eduard Tudenhoefner
>Assignee: Eduard Tudenhoefner
> Fix For: 3.0.14, 3.11.0, 4.0
>
> Attachments: columnidentifier.png
>
>
> It turns out that the object size of {{ColumnIdentifier}} is wrong when 
> *cassandra.test.flush_local_schema_changes: false*. This looks like stuff is 
> being wrongly reused when no flush is happening.
> We only noticed this because we were using the prepared stmt cache and 
> noticed that prepared statements would account for *1-6mb* when 
> *cassandra.test.flush_local_schema_changes: false*. With 
> *cassandra.test.flush_local_schema_changes: true* (which is the default) 
> those would be around *5000 bytes*.
> Attached is a test that reproduces the problem and also a fix.
> Also after talking to [~jkni] / [~blerer] we shouldn't probably take 
> {{ColumnDefinition}} into account when measuring object sizes with 
> {{MemoryMeter}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13533) ColumnIdentifier object size wrong when tables are not flushed

2017-05-26 Thread Joel Knighton (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026719#comment-16026719
 ] 

Joel Knighton commented on CASSANDRA-13533:
---

+1 - thanks for the patch. I ran tests for all relevant branches; no unit tests 
failed on 3.0/3.11/trunk, and dtests looked the same as the present state of 
those branches.

Committed to 3.0 as {{8ffdd26cbee33c5dc1205c0f7292628e1a2c69e3}} and merged 
forward.

> ColumnIdentifier object size wrong when tables are not flushed
> --
>
> Key: CASSANDRA-13533
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13533
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Eduard Tudenhoefner
>Assignee: Eduard Tudenhoefner
> Fix For: 3.0.14, 3.11.0, 4.0
>
> Attachments: columnidentifier.png
>
>
> It turns out that the object size of {{ColumnIdentifier}} is wrong when 
> *cassandra.test.flush_local_schema_changes: false*. This looks like stuff is 
> being wrongly reused when no flush is happening.
> We only noticed this because we were using the prepared stmt cache and 
> noticed that prepared statements would account for *1-6mb* when 
> *cassandra.test.flush_local_schema_changes: false*. With 
> *cassandra.test.flush_local_schema_changes: true* (which is the default) 
> those would be around *5000 bytes*.
> Attached is a test that reproduces the problem and also a fix.
> Also after talking to [~jkni] / [~blerer] we shouldn't probably take 
> {{ColumnDefinition}} into account when measuring object sizes with 
> {{MemoryMeter}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2017-05-26 Thread jkni 
Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/710720d9
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/710720d9
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/710720d9

Branch: refs/heads/trunk
Commit: 710720d98214c635aacdf01b3177bcdf39e6660b
Parents: ef40405 60c2004
Author: Joel Knighton 
Authored: Fri May 26 14:00:17 2017 -0500
Committer: Joel Knighton 
Committed: Fri May 26 14:00:17 2017 -0500

--
 CHANGES.txt|  1 +
 .../org/apache/cassandra/cql3/ColumnIdentifier.java|  2 ++
 .../org/apache/cassandra/schema/ColumnMetadata.java|  2 ++
 .../apache/cassandra/cql3/ColumnIdentifierTest.java| 13 +
 4 files changed, 18 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/710720d9/CHANGES.txt
--
diff --cc CHANGES.txt
index 87cb817,a8e172e..ebc819a
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -107,6 -35,8 +107,7 @@@
   * Fix cqlsh automatic protocol downgrade regression (CASSANDRA-13307)
   * Tracing payload not passed from QueryMessage to tracing session 
(CASSANDRA-12835)
  Merged from 3.0:
+  * Interned ColumnIdentifiers should use minimal ByteBuffers (CASSANDRA-13533)
 - * ReverseIndexedReader may drop rows during 2.1 to 3.0 upgrade 
(CASSANDRA-13525)
   * Fix repair process violating start/end token limits for small ranges 
(CASSANDRA-13052)
   * Add storage port options to sstableloader (CASSANDRA-13518)
   * Properly handle quoted index names in cqlsh DESCRIBE output 
(CASSANDRA-12847)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/710720d9/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/710720d9/src/java/org/apache/cassandra/schema/ColumnMetadata.java
--
diff --cc src/java/org/apache/cassandra/schema/ColumnMetadata.java
index 788351e,000..6943fa4
mode 100644,00..100644
--- a/src/java/org/apache/cassandra/schema/ColumnMetadata.java
+++ b/src/java/org/apache/cassandra/schema/ColumnMetadata.java
@@@ -1,617 -1,0 +1,619 @@@
 +/*
 + * Licensed to the Apache Software Foundation (ASF) under one
 + * or more contributor license agreements.  See the NOTICE file
 + * distributed with this work for additional information
 + * regarding copyright ownership.  The ASF licenses this file
 + * to you under the Apache License, Version 2.0 (the
 + * "License"); you may not use this file except in compliance
 + * with the License.  You may obtain a copy of the License at
 + *
 + * http://www.apache.org/licenses/LICENSE-2.0
 + *
 + * Unless required by applicable law or agreed to in writing, software
 + * distributed under the License is distributed on an "AS IS" BASIS,
 + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 + * See the License for the specific language governing permissions and
 + * limitations under the License.
 + */
 +package org.apache.cassandra.schema;
 +
 +import java.nio.ByteBuffer;
 +import java.util.*;
 +
 +import com.google.common.annotations.VisibleForTesting;
 +import com.google.common.base.MoreObjects;
 +import com.google.common.base.Objects;
 +import com.google.common.collect.Collections2;
 +
 +import org.apache.cassandra.cql3.*;
 +import org.apache.cassandra.cql3.selection.Selectable;
 +import org.apache.cassandra.cql3.selection.Selector;
 +import org.apache.cassandra.cql3.selection.SimpleSelector;
 +import org.apache.cassandra.db.rows.*;
 +import org.apache.cassandra.db.marshal.*;
 +import org.apache.cassandra.exceptions.InvalidRequestException;
 +import org.apache.cassandra.serializers.MarshalException;
 +import org.apache.cassandra.utils.ByteBufferUtil;
++import org.github.jamm.Unmetered;
 +
++@Unmetered
 +public final class ColumnMetadata extends ColumnSpecification implements 
Selectable, Comparable
 +{
 +public static final Comparator asymmetricColumnDataComparator =
 +(a, b) -> ((ColumnData) a).column().compareTo((ColumnMetadata) b);
 +
 +public static final int NO_POSITION = -1;
 +
 +public enum ClusteringOrder
 +{
 +ASC, DESC, NONE
 +}
 +
 +/**
 + * The type of CQL3 column this definition represents.
 + * There is 4 main type of CQL3 columns: those parts of the partition key,
 + * those parts of the clustering columns and amongst the others, regular 
and
 + * static ones.
 + *
 + * IMPORTANT: this enum is serialized as toString() and deserialized by 
calling
 + * Kind.valueOf(), so do not override toString() or

[2/6] cassandra git commit: Interned ColumnIdentifiers should use minimal ByteBuffers. MemoryMeter shouldn't take ColumnDefinition into account when measuring object sizes.

2017-05-26 Thread jkni 
Interned ColumnIdentifiers should use minimal ByteBuffers. MemoryMeter 
shouldn't take ColumnDefinition into account when measuring object sizes.

Patch by Eduard Tudenhoefner; reviewed by Joel Knighton for CASSANDRA-13533


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8ffdd26c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8ffdd26c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8ffdd26c

Branch: refs/heads/cassandra-3.11
Commit: 8ffdd26cbee33c5dc1205c0f7292628e1a2c69e3
Parents: 62092e4
Author: Eduard Tudenhoefner 
Authored: Wed May 10 18:04:33 2017 -0700
Committer: Joel Knighton 
Committed: Fri May 26 13:48:54 2017 -0500

--
 CHANGES.txt|  1 +
 .../org/apache/cassandra/config/ColumnDefinition.java  |  2 ++
 .../org/apache/cassandra/cql3/ColumnIdentifier.java|  2 ++
 .../apache/cassandra/cql3/ColumnIdentifierTest.java| 13 +
 4 files changed, 18 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/8ffdd26c/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index fe6ab09..18dca09 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.14
+ * Interned ColumnIdentifiers should use minimal ByteBuffers (CASSANDRA-13533)
  * ReverseIndexedReader may drop rows during 2.1 to 3.0 upgrade 
(CASSANDRA-13525)
  * Fix repair process violating start/end token limits for small ranges 
(CASSANDRA-13052)
  * Add storage port options to sstableloader (CASSANDRA-13518)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/8ffdd26c/src/java/org/apache/cassandra/config/ColumnDefinition.java
--
diff --git a/src/java/org/apache/cassandra/config/ColumnDefinition.java 
b/src/java/org/apache/cassandra/config/ColumnDefinition.java
index 6bcc2e0..34840e3 100644
--- a/src/java/org/apache/cassandra/config/ColumnDefinition.java
+++ b/src/java/org/apache/cassandra/config/ColumnDefinition.java
@@ -29,7 +29,9 @@ import org.apache.cassandra.cql3.*;
 import org.apache.cassandra.db.rows.*;
 import org.apache.cassandra.db.marshal.*;
 import org.apache.cassandra.serializers.MarshalException;
+import org.github.jamm.Unmetered;
 
+@Unmetered
 public class ColumnDefinition extends ColumnSpecification implements 
Comparable
 {
 public static final Comparator asymmetricColumnDataComparator =

http://git-wip-us.apache.org/repos/asf/cassandra/blob/8ffdd26c/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
--
diff --git a/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java 
b/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
index ceb81be..9b9ec4a 100644
--- a/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
+++ b/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
@@ -146,6 +146,8 @@ public class ColumnIdentifier extends Selectable implements 
IMeasurableMemory, C
 
 public static ColumnIdentifier getInterned(AbstractType type, 
ByteBuffer bytes, String text)
 {
+bytes = ByteBufferUtil.minimalBufferFor(bytes);
+
 InternedKey key = new InternedKey(type, bytes);
 ColumnIdentifier id = internedInstances.get(key);
 if (id != null)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/8ffdd26c/test/unit/org/apache/cassandra/cql3/ColumnIdentifierTest.java
--
diff --git a/test/unit/org/apache/cassandra/cql3/ColumnIdentifierTest.java 
b/test/unit/org/apache/cassandra/cql3/ColumnIdentifierTest.java
index c4b43b8..3a34ad5 100644
--- a/test/unit/org/apache/cassandra/cql3/ColumnIdentifierTest.java
+++ b/test/unit/org/apache/cassandra/cql3/ColumnIdentifierTest.java
@@ -82,4 +82,17 @@ public class ColumnIdentifierTest
 Assert.assertEquals(text, c3.toString());
 }
 
+@Test
+public void testInterningUsesMinimalByteBuffer()
+{
+byte[] bytes = new byte[2];
+bytes[0] = 0x63;
+ByteBuffer byteBuffer = ByteBuffer.wrap(bytes);
+byteBuffer.limit(1);
+
+ColumnIdentifier c1 = ColumnIdentifier.getInterned(byteBuffer, 
UTF8Type.instance);
+
+Assert.assertEquals(2, byteBuffer.capacity());
+Assert.assertEquals(1, c1.bytes.capacity());
+}
 }


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-05-26 Thread jkni 
Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/60c20049
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/60c20049
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/60c20049

Branch: refs/heads/cassandra-3.11
Commit: 60c20049631a2ed5565989c85dd62c2353ac6445
Parents: 6214b1e 8ffdd26
Author: Joel Knighton 
Authored: Fri May 26 13:53:06 2017 -0500
Committer: Joel Knighton 
Committed: Fri May 26 13:53:06 2017 -0500

--
 CHANGES.txt|  1 +
 .../org/apache/cassandra/config/ColumnDefinition.java  |  2 ++
 .../org/apache/cassandra/cql3/ColumnIdentifier.java|  2 ++
 .../apache/cassandra/cql3/ColumnIdentifierTest.java| 13 +
 4 files changed, 18 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/60c20049/CHANGES.txt
--
diff --cc CHANGES.txt
index 48f17c2,18dca09..a8e172e
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,40 -1,5 +1,41 @@@
 -3.0.14
 +3.11.0
 + * Fix the problem with duplicated rows when using paging with SASI 
(CASSANDRA-13302)
 + * Allow CONTAINS statements filtering on the partition key and it’s parts 
(CASSANDRA-13275)
 + * Fall back to even ranges calculation in clusters with vnodes when tokens 
are distributed unevenly (CASSANDRA-13229)
 + * Fix duration type validation to prevent overflow (CASSANDRA-13218)
 + * Forbid unsupported creation of SASI indexes over partition key columns 
(CASSANDRA-13228)
 + * Reject multiple values for a key in CQL grammar. (CASSANDRA-13369)
 + * UDA fails without input rows (CASSANDRA-13399)
 + * Fix compaction-stress by using daemonInitialization (CASSANDRA-13188)
 + * V5 protocol flags decoding broken (CASSANDRA-13443)
 + * Use write lock not read lock for removing sstables from compaction 
strategies. (CASSANDRA-13422)
 + * Use corePoolSize equal to maxPoolSize in JMXEnabledThreadPoolExecutors 
(CASSANDRA-13329)
 + * Avoid rebuilding SASI indexes containing no values (CASSANDRA-12962)
 + * Add charset to Analyser input stream (CASSANDRA-13151)
 + * Fix testLimitSSTables flake caused by concurrent flush (CASSANDRA-12820)
 + * cdc column addition strikes again (CASSANDRA-13382)
 + * Fix static column indexes (CASSANDRA-13277)
 + * DataOutputBuffer.asNewBuffer broken (CASSANDRA-13298)
 + * unittest CipherFactoryTest failed on MacOS (CASSANDRA-13370)
 + * Forbid SELECT restrictions and CREATE INDEX over non-frozen UDT columns 
(CASSANDRA-13247)
 + * Default logging we ship will incorrectly print "?:?" for "%F:%L" pattern 
(CASSANDRA-13317)
 + * Possible AssertionError in UnfilteredRowIteratorWithLowerBound 
(CASSANDRA-13366)
 + * Support unaligned memory access for AArch64 (CASSANDRA-13326)
 + * Improve SASI range iterator efficiency on intersection with an empty range 
(CASSANDRA-12915).
 + * Fix equality comparisons of columns using the duration type 
(CASSANDRA-13174)
 + * Obfuscate password in stress-graphs (CASSANDRA-12233)
 + * Move to FastThreadLocalThread and FastThreadLocal (CASSANDRA-13034)
 + * nodetool stopdaemon errors out (CASSANDRA-13030)
 + * Tables in system_distributed should not use gcgs of 0 (CASSANDRA-12954)
 + * Fix primary index calculation for SASI (CASSANDRA-12910)
 + * More fixes to the TokenAllocator (CASSANDRA-12990)
 + * NoReplicationTokenAllocator should work with zero replication factor 
(CASSANDRA-12983)
 + * Address message coalescing regression (CASSANDRA-12676)
 + * Delete illegal character from StandardTokenizerImpl.jflex (CASSANDRA-13417)
 + * Fix cqlsh automatic protocol downgrade regression (CASSANDRA-13307)
 + * Tracing payload not passed from QueryMessage to tracing session 
(CASSANDRA-12835)
 +Merged from 3.0:
+  * Interned ColumnIdentifiers should use minimal ByteBuffers (CASSANDRA-13533)
   * ReverseIndexedReader may drop rows during 2.1 to 3.0 upgrade 
(CASSANDRA-13525)
   * Fix repair process violating start/end token limits for small ranges 
(CASSANDRA-13052)
   * Add storage port options to sstableloader (CASSANDRA-13518)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/60c20049/src/java/org/apache/cassandra/config/ColumnDefinition.java
--
diff --cc src/java/org/apache/cassandra/config/ColumnDefinition.java
index 61739a3,34840e3..ea508d2
--- a/src/java/org/apache/cassandra/config/ColumnDefinition.java
+++ b/src/java/org/apache/cassandra/config/ColumnDefinition.java
@@@ -27,16 -26,13 +27,18 @@@ import com.google.common.base.Objects
  import com.google.common.collect.Collections2;
  
  import org.apache.cassandra.cql3.*;
 +import org.apache.cassandra.cql3.selection.Selectable;
 +import

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-05-26 Thread jkni 
Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/60c20049
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/60c20049
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/60c20049

Branch: refs/heads/trunk
Commit: 60c20049631a2ed5565989c85dd62c2353ac6445
Parents: 6214b1e 8ffdd26
Author: Joel Knighton 
Authored: Fri May 26 13:53:06 2017 -0500
Committer: Joel Knighton 
Committed: Fri May 26 13:53:06 2017 -0500

--
 CHANGES.txt|  1 +
 .../org/apache/cassandra/config/ColumnDefinition.java  |  2 ++
 .../org/apache/cassandra/cql3/ColumnIdentifier.java|  2 ++
 .../apache/cassandra/cql3/ColumnIdentifierTest.java| 13 +
 4 files changed, 18 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/60c20049/CHANGES.txt
--
diff --cc CHANGES.txt
index 48f17c2,18dca09..a8e172e
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,40 -1,5 +1,41 @@@
 -3.0.14
 +3.11.0
 + * Fix the problem with duplicated rows when using paging with SASI 
(CASSANDRA-13302)
 + * Allow CONTAINS statements filtering on the partition key and it’s parts 
(CASSANDRA-13275)
 + * Fall back to even ranges calculation in clusters with vnodes when tokens 
are distributed unevenly (CASSANDRA-13229)
 + * Fix duration type validation to prevent overflow (CASSANDRA-13218)
 + * Forbid unsupported creation of SASI indexes over partition key columns 
(CASSANDRA-13228)
 + * Reject multiple values for a key in CQL grammar. (CASSANDRA-13369)
 + * UDA fails without input rows (CASSANDRA-13399)
 + * Fix compaction-stress by using daemonInitialization (CASSANDRA-13188)
 + * V5 protocol flags decoding broken (CASSANDRA-13443)
 + * Use write lock not read lock for removing sstables from compaction 
strategies. (CASSANDRA-13422)
 + * Use corePoolSize equal to maxPoolSize in JMXEnabledThreadPoolExecutors 
(CASSANDRA-13329)
 + * Avoid rebuilding SASI indexes containing no values (CASSANDRA-12962)
 + * Add charset to Analyser input stream (CASSANDRA-13151)
 + * Fix testLimitSSTables flake caused by concurrent flush (CASSANDRA-12820)
 + * cdc column addition strikes again (CASSANDRA-13382)
 + * Fix static column indexes (CASSANDRA-13277)
 + * DataOutputBuffer.asNewBuffer broken (CASSANDRA-13298)
 + * unittest CipherFactoryTest failed on MacOS (CASSANDRA-13370)
 + * Forbid SELECT restrictions and CREATE INDEX over non-frozen UDT columns 
(CASSANDRA-13247)
 + * Default logging we ship will incorrectly print "?:?" for "%F:%L" pattern 
(CASSANDRA-13317)
 + * Possible AssertionError in UnfilteredRowIteratorWithLowerBound 
(CASSANDRA-13366)
 + * Support unaligned memory access for AArch64 (CASSANDRA-13326)
 + * Improve SASI range iterator efficiency on intersection with an empty range 
(CASSANDRA-12915).
 + * Fix equality comparisons of columns using the duration type 
(CASSANDRA-13174)
 + * Obfuscate password in stress-graphs (CASSANDRA-12233)
 + * Move to FastThreadLocalThread and FastThreadLocal (CASSANDRA-13034)
 + * nodetool stopdaemon errors out (CASSANDRA-13030)
 + * Tables in system_distributed should not use gcgs of 0 (CASSANDRA-12954)
 + * Fix primary index calculation for SASI (CASSANDRA-12910)
 + * More fixes to the TokenAllocator (CASSANDRA-12990)
 + * NoReplicationTokenAllocator should work with zero replication factor 
(CASSANDRA-12983)
 + * Address message coalescing regression (CASSANDRA-12676)
 + * Delete illegal character from StandardTokenizerImpl.jflex (CASSANDRA-13417)
 + * Fix cqlsh automatic protocol downgrade regression (CASSANDRA-13307)
 + * Tracing payload not passed from QueryMessage to tracing session 
(CASSANDRA-12835)
 +Merged from 3.0:
+  * Interned ColumnIdentifiers should use minimal ByteBuffers (CASSANDRA-13533)
   * ReverseIndexedReader may drop rows during 2.1 to 3.0 upgrade 
(CASSANDRA-13525)
   * Fix repair process violating start/end token limits for small ranges 
(CASSANDRA-13052)
   * Add storage port options to sstableloader (CASSANDRA-13518)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/60c20049/src/java/org/apache/cassandra/config/ColumnDefinition.java
--
diff --cc src/java/org/apache/cassandra/config/ColumnDefinition.java
index 61739a3,34840e3..ea508d2
--- a/src/java/org/apache/cassandra/config/ColumnDefinition.java
+++ b/src/java/org/apache/cassandra/config/ColumnDefinition.java
@@@ -27,16 -26,13 +27,18 @@@ import com.google.common.base.Objects
  import com.google.common.collect.Collections2;
  
  import org.apache.cassandra.cql3.*;
 +import org.apache.cassandra.cql3.selection.Selectable;
 +import

[3/6] cassandra git commit: Interned ColumnIdentifiers should use minimal ByteBuffers. MemoryMeter shouldn't take ColumnDefinition into account when measuring object sizes.

2017-05-26 Thread jkni 
Interned ColumnIdentifiers should use minimal ByteBuffers. MemoryMeter 
shouldn't take ColumnDefinition into account when measuring object sizes.

Patch by Eduard Tudenhoefner; reviewed by Joel Knighton for CASSANDRA-13533


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8ffdd26c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8ffdd26c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8ffdd26c

Branch: refs/heads/trunk
Commit: 8ffdd26cbee33c5dc1205c0f7292628e1a2c69e3
Parents: 62092e4
Author: Eduard Tudenhoefner 
Authored: Wed May 10 18:04:33 2017 -0700
Committer: Joel Knighton 
Committed: Fri May 26 13:48:54 2017 -0500

--
 CHANGES.txt|  1 +
 .../org/apache/cassandra/config/ColumnDefinition.java  |  2 ++
 .../org/apache/cassandra/cql3/ColumnIdentifier.java|  2 ++
 .../apache/cassandra/cql3/ColumnIdentifierTest.java| 13 +
 4 files changed, 18 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/8ffdd26c/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index fe6ab09..18dca09 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.14
+ * Interned ColumnIdentifiers should use minimal ByteBuffers (CASSANDRA-13533)
  * ReverseIndexedReader may drop rows during 2.1 to 3.0 upgrade 
(CASSANDRA-13525)
  * Fix repair process violating start/end token limits for small ranges 
(CASSANDRA-13052)
  * Add storage port options to sstableloader (CASSANDRA-13518)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/8ffdd26c/src/java/org/apache/cassandra/config/ColumnDefinition.java
--
diff --git a/src/java/org/apache/cassandra/config/ColumnDefinition.java 
b/src/java/org/apache/cassandra/config/ColumnDefinition.java
index 6bcc2e0..34840e3 100644
--- a/src/java/org/apache/cassandra/config/ColumnDefinition.java
+++ b/src/java/org/apache/cassandra/config/ColumnDefinition.java
@@ -29,7 +29,9 @@ import org.apache.cassandra.cql3.*;
 import org.apache.cassandra.db.rows.*;
 import org.apache.cassandra.db.marshal.*;
 import org.apache.cassandra.serializers.MarshalException;
+import org.github.jamm.Unmetered;
 
+@Unmetered
 public class ColumnDefinition extends ColumnSpecification implements 
Comparable
 {
 public static final Comparator asymmetricColumnDataComparator =

http://git-wip-us.apache.org/repos/asf/cassandra/blob/8ffdd26c/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
--
diff --git a/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java 
b/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
index ceb81be..9b9ec4a 100644
--- a/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
+++ b/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
@@ -146,6 +146,8 @@ public class ColumnIdentifier extends Selectable implements 
IMeasurableMemory, C
 
 public static ColumnIdentifier getInterned(AbstractType type, 
ByteBuffer bytes, String text)
 {
+bytes = ByteBufferUtil.minimalBufferFor(bytes);
+
 InternedKey key = new InternedKey(type, bytes);
 ColumnIdentifier id = internedInstances.get(key);
 if (id != null)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/8ffdd26c/test/unit/org/apache/cassandra/cql3/ColumnIdentifierTest.java
--
diff --git a/test/unit/org/apache/cassandra/cql3/ColumnIdentifierTest.java 
b/test/unit/org/apache/cassandra/cql3/ColumnIdentifierTest.java
index c4b43b8..3a34ad5 100644
--- a/test/unit/org/apache/cassandra/cql3/ColumnIdentifierTest.java
+++ b/test/unit/org/apache/cassandra/cql3/ColumnIdentifierTest.java
@@ -82,4 +82,17 @@ public class ColumnIdentifierTest
 Assert.assertEquals(text, c3.toString());
 }
 
+@Test
+public void testInterningUsesMinimalByteBuffer()
+{
+byte[] bytes = new byte[2];
+bytes[0] = 0x63;
+ByteBuffer byteBuffer = ByteBuffer.wrap(bytes);
+byteBuffer.limit(1);
+
+ColumnIdentifier c1 = ColumnIdentifier.getInterned(byteBuffer, 
UTF8Type.instance);
+
+Assert.assertEquals(2, byteBuffer.capacity());
+Assert.assertEquals(1, c1.bytes.capacity());
+}
 }


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[1/6] cassandra git commit: Interned ColumnIdentifiers should use minimal ByteBuffers. MemoryMeter shouldn't take ColumnDefinition into account when measuring object sizes.

2017-05-26 Thread jkni 
Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 62092e45c -> 8ffdd26cb
  refs/heads/cassandra-3.11 6214b1ec8 -> 60c200496
  refs/heads/trunk ef404053b -> 710720d98


Interned ColumnIdentifiers should use minimal ByteBuffers. MemoryMeter 
shouldn't take ColumnDefinition into account when measuring object sizes.

Patch by Eduard Tudenhoefner; reviewed by Joel Knighton for CASSANDRA-13533


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8ffdd26c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8ffdd26c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8ffdd26c

Branch: refs/heads/cassandra-3.0
Commit: 8ffdd26cbee33c5dc1205c0f7292628e1a2c69e3
Parents: 62092e4
Author: Eduard Tudenhoefner 
Authored: Wed May 10 18:04:33 2017 -0700
Committer: Joel Knighton 
Committed: Fri May 26 13:48:54 2017 -0500

--
 CHANGES.txt|  1 +
 .../org/apache/cassandra/config/ColumnDefinition.java  |  2 ++
 .../org/apache/cassandra/cql3/ColumnIdentifier.java|  2 ++
 .../apache/cassandra/cql3/ColumnIdentifierTest.java| 13 +
 4 files changed, 18 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/8ffdd26c/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index fe6ab09..18dca09 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.14
+ * Interned ColumnIdentifiers should use minimal ByteBuffers (CASSANDRA-13533)
  * ReverseIndexedReader may drop rows during 2.1 to 3.0 upgrade 
(CASSANDRA-13525)
  * Fix repair process violating start/end token limits for small ranges 
(CASSANDRA-13052)
  * Add storage port options to sstableloader (CASSANDRA-13518)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/8ffdd26c/src/java/org/apache/cassandra/config/ColumnDefinition.java
--
diff --git a/src/java/org/apache/cassandra/config/ColumnDefinition.java 
b/src/java/org/apache/cassandra/config/ColumnDefinition.java
index 6bcc2e0..34840e3 100644
--- a/src/java/org/apache/cassandra/config/ColumnDefinition.java
+++ b/src/java/org/apache/cassandra/config/ColumnDefinition.java
@@ -29,7 +29,9 @@ import org.apache.cassandra.cql3.*;
 import org.apache.cassandra.db.rows.*;
 import org.apache.cassandra.db.marshal.*;
 import org.apache.cassandra.serializers.MarshalException;
+import org.github.jamm.Unmetered;
 
+@Unmetered
 public class ColumnDefinition extends ColumnSpecification implements 
Comparable
 {
 public static final Comparator asymmetricColumnDataComparator =

http://git-wip-us.apache.org/repos/asf/cassandra/blob/8ffdd26c/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
--
diff --git a/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java 
b/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
index ceb81be..9b9ec4a 100644
--- a/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
+++ b/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
@@ -146,6 +146,8 @@ public class ColumnIdentifier extends Selectable implements 
IMeasurableMemory, C
 
 public static ColumnIdentifier getInterned(AbstractType type, 
ByteBuffer bytes, String text)
 {
+bytes = ByteBufferUtil.minimalBufferFor(bytes);
+
 InternedKey key = new InternedKey(type, bytes);
 ColumnIdentifier id = internedInstances.get(key);
 if (id != null)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/8ffdd26c/test/unit/org/apache/cassandra/cql3/ColumnIdentifierTest.java
--
diff --git a/test/unit/org/apache/cassandra/cql3/ColumnIdentifierTest.java 
b/test/unit/org/apache/cassandra/cql3/ColumnIdentifierTest.java
index c4b43b8..3a34ad5 100644
--- a/test/unit/org/apache/cassandra/cql3/ColumnIdentifierTest.java
+++ b/test/unit/org/apache/cassandra/cql3/ColumnIdentifierTest.java
@@ -82,4 +82,17 @@ public class ColumnIdentifierTest
 Assert.assertEquals(text, c3.toString());
 }
 
+@Test
+public void testInterningUsesMinimalByteBuffer()
+{
+byte[] bytes = new byte[2];
+bytes[0] = 0x63;
+ByteBuffer byteBuffer = ByteBuffer.wrap(bytes);
+byteBuffer.limit(1);
+
+ColumnIdentifier c1 = ColumnIdentifier.getInterned(byteBuffer, 
UTF8Type.instance);
+
+Assert.assertEquals(2, byteBuffer.capacity());
+Assert.assertEquals(1, c1.bytes.capacity());
+}
 }


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional

[jira] [Updated] (CASSANDRA-13556) Corrupted SSTables

2017-05-26 Thread Jeff Jirsa (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13556:
---
Priority: Critical  (was: Major)

> Corrupted SSTables
> --
>
> Key: CASSANDRA-13556
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13556
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: CentOS Linux release 7.3.1611 (Core)
> openjdk version "1.8.0_121"
> OpenJDK Runtime Environment (build 1.8.0_121-b13)
> OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)
> Python cassandra (DataStax) driver v3.6.0
>Reporter: Ihor Prokopov
>Priority: Critical
> Fix For: 3.9
>
>
> After 3 month of working, we noticed that number of compaction tasks were 
> growing (~600 pending tasks). SStables verification shows that some of them 
> were corrupted. Repairing didn't help (it was crashing with error). 
> Also some of requests (f.e. select * from fetcher where 
> domain=8289511971670945261 and uri=-5417197141545933706; ) fails with next 
> error:
> {color:red}
> Traceback (most recent call last):
>   File "/var/cassandra/apache-cassandra-3.9/bin/cqlsh.py", line 1264, in 
> perform_simple_statement
> result = future.result()
>   File 
> "/var/cassandra/apache-cassandra-3.9/bin/../lib/cassandra-driver-internal-only-3.5.0.post0-d8d0456.zip/cassandra-driver-3.5.0.post0-d8d0456/cassandra/cluster.py",
>  line 3650, in result
> raise self._final_exception
> error: unpack requires a string argument of length 4
> {color}
> Table chema:
> {quote}
> CREATE TABLE fetcher (
> domain bigint,
> uri bigint,
> date date,
> content_length int,
> elapsed float,
> encoding text,
> fetched_time bigint,
> flinks frozen,
> flinks_count int,
> html_fingerprint bigint,
> indexed boolean,
> adult boolean,
> kws_count int,
> lang_id int,
> last_updated bigint,
> redirect_url tuple,
> revisit_date date,
> revisit_interval int,
> status_code int,
> tokens_fingerprint bigint,
> uris frozen url text,
> PRIMARY KEY (domain, uri, date)
> ) WITH CLUSTERING ORDER BY (uri ASC, date DESC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = \{'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = 'fetcher history'
> AND compaction = \{'class': 
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy',
>   'sstable_size_in_mb': '256',
>   'tombstone_threshold': '.2'}
> AND compression = \{'chunk_length_in_kb': '64',
>'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.5
> AND speculative_retry = '99PERCENTILE';
> {quote}
> Corrupted 
> [SSTable|https://drive.google.com/file/d/0B4ZaUOv0G9oMcHpERTdlb3ozSVk/view?usp=sharing].



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13556) Corrupted SSTables

2017-05-26 Thread Jeff Jirsa (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026428#comment-16026428
 ] 

Jeff Jirsa commented on CASSANDRA-13556:


There should be a stack trace on the server when you run the {{SELECT}} in 
cqlsh. Please include that as well.

> Corrupted SSTables
> --
>
> Key: CASSANDRA-13556
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13556
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: CentOS Linux release 7.3.1611 (Core)
> openjdk version "1.8.0_121"
> OpenJDK Runtime Environment (build 1.8.0_121-b13)
> OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)
> Python cassandra (DataStax) driver v3.6.0
>Reporter: Ihor Prokopov
> Fix For: 3.9
>
>
> After 3 month of working, we noticed that number of compaction tasks were 
> growing (~600 pending tasks). SStables verification shows that some of them 
> were corrupted. Repairing didn't help (it was crashing with error). 
> Also some of requests (f.e. select * from fetcher where 
> domain=8289511971670945261 and uri=-5417197141545933706; ) fails with next 
> error:
> {color:red}
> Traceback (most recent call last):
>   File "/var/cassandra/apache-cassandra-3.9/bin/cqlsh.py", line 1264, in 
> perform_simple_statement
> result = future.result()
>   File 
> "/var/cassandra/apache-cassandra-3.9/bin/../lib/cassandra-driver-internal-only-3.5.0.post0-d8d0456.zip/cassandra-driver-3.5.0.post0-d8d0456/cassandra/cluster.py",
>  line 3650, in result
> raise self._final_exception
> error: unpack requires a string argument of length 4
> {color}
> Table chema:
> {quote}
> CREATE TABLE fetcher (
> domain bigint,
> uri bigint,
> date date,
> content_length int,
> elapsed float,
> encoding text,
> fetched_time bigint,
> flinks frozen,
> flinks_count int,
> html_fingerprint bigint,
> indexed boolean,
> adult boolean,
> kws_count int,
> lang_id int,
> last_updated bigint,
> redirect_url tuple,
> revisit_date date,
> revisit_interval int,
> status_code int,
> tokens_fingerprint bigint,
> uris frozen url text,
> PRIMARY KEY (domain, uri, date)
> ) WITH CLUSTERING ORDER BY (uri ASC, date DESC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = \{'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = 'fetcher history'
> AND compaction = \{'class': 
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy',
>   'sstable_size_in_mb': '256',
>   'tombstone_threshold': '.2'}
> AND compression = \{'chunk_length_in_kb': '64',
>'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.5
> AND speculative_retry = '99PERCENTILE';
> {quote}
> Corrupted 
> [SSTable|https://drive.google.com/file/d/0B4ZaUOv0G9oMcHpERTdlb3ozSVk/view?usp=sharing].



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13444) Fast and garbage-free Streaming Histogram

2017-05-26 Thread Fuud (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025999#comment-16025999
 ] 

Fuud edited comment on CASSANDRA-13444 at 5/26/17 8:42 AM:
---

Sorry. It was mistake to amend commit during the review. 
Bug was fixed + equals method fixed + small refactoring to keep size() 
calculation clear.
Diff:
{code}
diff --git 
a/src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java
 
b/src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java
index 8affb4e..21b3bef 100755
--- 
a/src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java
+++ 
b/src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java
@@ -404,6 +404,13 @@ public class StreamingTombstoneHistogramBuilder
 }
 }
 
+public int size()
+{
+int[] accumulator = new int[1];
+forEach((point, value) -> accumulator[0]++);
+return accumulator[0];
+}
+
 public double sum(int b)
 {
 double sum = 0;
@@ -480,7 +487,21 @@ public class StreamingTombstoneHistogramBuilder
 @Override
 public boolean equals(Object o)
 {
-return o instanceof DataHolder && Arrays.equals(data, 
((DataHolder) o).data);
+if (!(o instanceof DataHolder))
+return false;
+
+
+final DataHolder other = ((DataHolder) o);
+
+if (this.size()!=other.size())
+return false;
+
+for (int i=0; i accumulator[0]++);
-return accumulator[0];
+return this.bin.size();
 }
 
 public  void forEach(HistogramDataConsumer 
histogramDataConsumer) throws E
@@ -90,7 +88,10 @@ public class TombstoneHistogram
 in.readInt(); // max bin size
 int size = in.readInt();
 DataHolder dataHolder = new DataHolder(size, 1);
-dataHolder.addValue((int)in.readDouble(), (int)in.readLong());
+for (int i = 0; i < size; i++)
+{
+dataHolder.addValue((int)in.readDouble(), (int)in.readLong());
+}
 
 return new TombstoneHistogram(dataHolder);
 }
{code}


was (Author: fuud):
Sorry. It was mistake to amend commit during the review. 
Bug was fixed + equals method fixed + small refactoring to keep size() 
calculation clear.
Diff:
```
diff --git 
a/src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java
 
b/src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java
index 8affb4e..21b3bef 100755
--- 
a/src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java
+++ 
b/src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java
@@ -404,6 +404,13 @@ public class StreamingTombstoneHistogramBuilder
 }
 }
 
+public int size()
+{
+int[] accumulator = new int[1];
+forEach((point, value) -> accumulator[0]++);
+return accumulator[0];
+}
+
 public double sum(int b)
 {
 double sum = 0;
@@ -480,7 +487,21 @@ public class StreamingTombstoneHistogramBuilder
 @Override
 public boolean equals(Object o)
 {
-return o instanceof DataHolder && Arrays.equals(data, 
((DataHolder) o).data);
+if (!(o instanceof DataHolder))
+return false;
+
+
+final DataHolder other = ((DataHolder) o);
+
+if (this.size()!=other.size())
+return false;
+
+for (int i=0; i

[jira] [Commented] (CASSANDRA-13444) Fast and garbage-free Streaming Histogram

2017-05-26 Thread Fuud (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025999#comment-16025999
 ] 

Fuud commented on CASSANDRA-13444:
--

Sorry. It was mistake to amend commit during the review. 
Bug was fixed + equals method fixed + small refactoring to keep size() 
calculation clear.
Diff:
```
diff --git 
a/src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java
 
b/src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java
index 8affb4e..21b3bef 100755
--- 
a/src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java
+++ 
b/src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java
@@ -404,6 +404,13 @@ public class StreamingTombstoneHistogramBuilder
 }
 }
 
+public int size()
+{
+int[] accumulator = new int[1];
+forEach((point, value) -> accumulator[0]++);
+return accumulator[0];
+}
+
 public double sum(int b)
 {
 double sum = 0;
@@ -480,7 +487,21 @@ public class StreamingTombstoneHistogramBuilder
 @Override
 public boolean equals(Object o)
 {
-return o instanceof DataHolder && Arrays.equals(data, 
((DataHolder) o).data);
+if (!(o instanceof DataHolder))
+return false;
+
+
+final DataHolder other = ((DataHolder) o);
+
+if (this.size()!=other.size())
+return false;
+
+for (int i=0; i accumulator[0]++);
-return accumulator[0];
+return this.bin.size();
 }
 
 public  void forEach(HistogramDataConsumer 
histogramDataConsumer) throws E
@@ -90,7 +88,10 @@ public class TombstoneHistogram
 in.readInt(); // max bin size
 int size = in.readInt();
 DataHolder dataHolder = new DataHolder(size, 1);
-dataHolder.addValue((int)in.readDouble(), (int)in.readLong());
+for (int i = 0; i < size; i++)
+{
+dataHolder.addValue((int)in.readDouble(), (int)in.readLong());
+}
 
 return new TombstoneHistogram(dataHolder);
 }
```

> Fast and garbage-free Streaming Histogram
> -
>
> Key: CASSANDRA-13444
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13444
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Fuud
>Assignee: Fuud
> Fix For: 4.x
>
> Attachments: results.csv, results.xlsx
>
>
> StreamingHistogram is cause of high cpu usage and GC pressure.
> It was improved at CASSANDRA-13038 by introducing intermediate buffer to try 
> accumulate different values into the big map before merging them into smaller 
> one.
> But there was not enought for TTL's distributed within large time. Rounding 
> (also introduced at 13038) can help but it reduce histogram precision 
> specially in case where TTL's does not distributed uniformly.
> There are several improvements that can help to reduce cpu and gc usage. Them 
> all included in the pool-request as separate revisions thus you can test them 
> independently.
> Improvements list:
> # Use Map.computeIfAbsent instead of get->checkIfNull->put chain. In this way 
> "add-or-accumulate" operation takes one map operation instead of two. But 
> this method (default-defined in interface Map) is overriden in HashMap but 
> not in TreeMap. Thus I changed spool type to HashMap.
> # As we round incoming values to _roundSeconds_ we can also round value on 
> merge. It will enlarge hit rate for bin operations.
> # Because we inserted only integers into Histogram and rounding values to 
> integers we can use *int* type everywhere.
> # Histogram takes huge amount of time merging values. In merge method largest 
> amount of time taken by finding nearest points. It can be eliminated by 
> holding additional TreeSet with differences, sorted from smalest to greatest.
> # Because we know max size of _bin_ and _differences_ maps we can replace 
> them with sorted arrays. Search can be done

[jira] [Created] (CASSANDRA-13556) Corrupted SSTables

2017-05-26 Thread Ihor Prokopov (JIRA) 
Ihor Prokopov created CASSANDRA-13556:
-

 Summary: Corrupted SSTables
 Key: CASSANDRA-13556
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13556
 Project: Cassandra
  Issue Type: Bug
  Components: Compaction
 Environment: CentOS Linux release 7.3.1611 (Core)
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)
Python cassandra (DataStax) driver v3.6.0
Reporter: Ihor Prokopov
 Fix For: 3.9


After 3 month of working, we noticed that number of compaction tasks were 
growing (~600 pending tasks). SStables verification shows that some of them 
were corrupted. Repairing didn't help (it was crashing with error). 
Also some of requests (f.e. select * from fetcher where 
domain=8289511971670945261 and uri=-5417197141545933706; ) fails with next 
error:
{color:red}
Traceback (most recent call last):
  File "/var/cassandra/apache-cassandra-3.9/bin/cqlsh.py", line 1264, in 
perform_simple_statement
result = future.result()
  File 
"/var/cassandra/apache-cassandra-3.9/bin/../lib/cassandra-driver-internal-only-3.5.0.post0-d8d0456.zip/cassandra-driver-3.5.0.post0-d8d0456/cassandra/cluster.py",
 line 3650, in result
raise self._final_exception
error: unpack requires a string argument of length 4
{color}

Table chema:
{quote}
CREATE TABLE fetcher (
domain bigint,
uri bigint,
date date,
content_length int,
elapsed float,
encoding text,
fetched_time bigint,
flinks frozen,
flinks_count int,
html_fingerprint bigint,
indexed boolean,
adult boolean,
kws_count int,
lang_id int,
last_updated bigint,
redirect_url tuple,
revisit_date date,
revisit_interval int,
status_code int,
tokens_fingerprint bigint,
uris frozen