[jira] [Updated] (CASSANDRA-14057) The size of a byte is not 2

2017-11-18 Thread Sylvain Lebresne (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-14057:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

No problem, thanks for the quick review, committed.

> The size of a byte is not 2
> ---
>
> Key: CASSANDRA-14057
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14057
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Sylvain Lebresne
>Assignee: Sylvain Lebresne
>Priority: Trivial
> Fix For: 3.0.16, 3.11.2
>
>
> {{DataLimits}} serializer uses 
> {{TypeSizes.sizeof((byte)limits.kind().ordinal())}} in it's 
> {{serializedSize}} method, but {{TypeSizes}} does not (on 3.0/3.11) have a 
> {{sizeof(byte)}} override, so {{sizeof(short)}} is picked and it returns 2, 
> which is wrong.
> This is actually fixed on trunk, [~jasobrown] committed the fix as part of 
> CASSANDRA-8457, but it wasn't committed in 3.0/3.11 and that still feel dodgy 
> to leave as is.
> To be clear, I don't think it's a problem as of now: it does break the size 
> computed for read commands, and thus the payload size of such message, but 
> said payload size is not use (for read requests that is).
> Still, not reason to leave something wrong like this and risk a future bug 
> when the fix is trivial.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[2/6] cassandra git commit: Fix serialized size of DataLimits

2017-11-18 Thread slebresne 
Fix serialized size of DataLimits

patch by Sylvain Lebresne; reviewed by Jason Brown for CASSANDRA-14057


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/eb05025c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/eb05025c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/eb05025c

Branch: refs/heads/cassandra-3.11
Commit: eb05025c0a768241ea61fd86db9a88cfd8f6e93e
Parents: f1e850a
Author: Sylvain Lebresne 
Authored: Fri Nov 17 15:46:58 2017 +0100
Committer: Sylvain Lebresne 
Committed: Sat Nov 18 14:16:42 2017 +0100

--
 CHANGES.txt | 1 +
 src/java/org/apache/cassandra/db/TypeSizes.java | 6 ++
 2 files changed, 7 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb05025c/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index a3c43fd..6c50a3f 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * Fix serialized size of DataLimits (CASSANDRA-14057)
  * Add flag to allow dropping oversized read repair mutations (CASSANDRA-13975)
  * Fix SSTableLoader logger message (CASSANDRA-14003)
  * Fix repair race that caused gossip to block (CASSANDRA-13849)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb05025c/src/java/org/apache/cassandra/db/TypeSizes.java
--
diff --git a/src/java/org/apache/cassandra/db/TypeSizes.java 
b/src/java/org/apache/cassandra/db/TypeSizes.java
index 6543025..b47e300 100644
--- a/src/java/org/apache/cassandra/db/TypeSizes.java
+++ b/src/java/org/apache/cassandra/db/TypeSizes.java
@@ -28,6 +28,7 @@ public final class TypeSizes
 private TypeSizes(){}
 
 private static final int BOOL_SIZE = 1;
+private static final int BYTE_SIZE = 1;
 private static final int SHORT_SIZE = 2;
 private static final int INT_SIZE = 4;
 private static final int LONG_SIZE = 8;
@@ -78,6 +79,11 @@ public final class TypeSizes
 return BOOL_SIZE;
 }
 
+public static int sizeof(byte value)
+{
+return BYTE_SIZE;
+}
+
 public static int sizeof(short value)
 {
 return SHORT_SIZE;


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-11-18 Thread slebresne 
Merge branch 'cassandra-3.0' into cassandra-3.11

* cassandra-3.0:
  Fix serialized size of DataLimits


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d2bc37f4
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d2bc37f4
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d2bc37f4

Branch: refs/heads/trunk
Commit: d2bc37f43db14661a159406e421985a2484d5387
Parents: 9ee44db eb05025
Author: Sylvain Lebresne 
Authored: Sat Nov 18 14:17:43 2017 +0100
Committer: Sylvain Lebresne 
Committed: Sat Nov 18 14:17:43 2017 +0100

--
 CHANGES.txt | 1 +
 src/java/org/apache/cassandra/db/TypeSizes.java | 6 ++
 2 files changed, 7 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d2bc37f4/CHANGES.txt
--
diff --cc CHANGES.txt
index a1a1a37,6c50a3f..3fc934e
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,9 -1,5 +1,10 @@@
 -3.0.16
 +3.11.2
 + * Add asm jar to build.xml for maven builds (CASSANDRA-11193)
 + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
 + * Update jackson JSON jars (CASSANDRA-13949)
 + * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
 +Merged from 3.0:
+  * Fix serialized size of DataLimits (CASSANDRA-14057)
   * Add flag to allow dropping oversized read repair mutations 
(CASSANDRA-13975)
   * Fix SSTableLoader logger message (CASSANDRA-14003)
   * Fix repair race that caused gossip to block (CASSANDRA-13849)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2017-11-18 Thread slebresne 
Merge branch 'cassandra-3.11' into trunk

* cassandra-3.11:
  Fix serialized size of DataLimits


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/da410153
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/da410153
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/da410153

Branch: refs/heads/trunk
Commit: da410153ed250e03080fd48cd0f57baee481412d
Parents: f1f6ed6 d2bc37f
Author: Sylvain Lebresne 
Authored: Sat Nov 18 14:17:58 2017 +0100
Committer: Sylvain Lebresne 
Committed: Sat Nov 18 14:17:58 2017 +0100

--
 CHANGES.txt | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/da410153/CHANGES.txt
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[3/6] cassandra git commit: Fix serialized size of DataLimits

2017-11-18 Thread slebresne 
Fix serialized size of DataLimits

patch by Sylvain Lebresne; reviewed by Jason Brown for CASSANDRA-14057


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/eb05025c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/eb05025c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/eb05025c

Branch: refs/heads/trunk
Commit: eb05025c0a768241ea61fd86db9a88cfd8f6e93e
Parents: f1e850a
Author: Sylvain Lebresne 
Authored: Fri Nov 17 15:46:58 2017 +0100
Committer: Sylvain Lebresne 
Committed: Sat Nov 18 14:16:42 2017 +0100

--
 CHANGES.txt | 1 +
 src/java/org/apache/cassandra/db/TypeSizes.java | 6 ++
 2 files changed, 7 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb05025c/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index a3c43fd..6c50a3f 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * Fix serialized size of DataLimits (CASSANDRA-14057)
  * Add flag to allow dropping oversized read repair mutations (CASSANDRA-13975)
  * Fix SSTableLoader logger message (CASSANDRA-14003)
  * Fix repair race that caused gossip to block (CASSANDRA-13849)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb05025c/src/java/org/apache/cassandra/db/TypeSizes.java
--
diff --git a/src/java/org/apache/cassandra/db/TypeSizes.java 
b/src/java/org/apache/cassandra/db/TypeSizes.java
index 6543025..b47e300 100644
--- a/src/java/org/apache/cassandra/db/TypeSizes.java
+++ b/src/java/org/apache/cassandra/db/TypeSizes.java
@@ -28,6 +28,7 @@ public final class TypeSizes
 private TypeSizes(){}
 
 private static final int BOOL_SIZE = 1;
+private static final int BYTE_SIZE = 1;
 private static final int SHORT_SIZE = 2;
 private static final int INT_SIZE = 4;
 private static final int LONG_SIZE = 8;
@@ -78,6 +79,11 @@ public final class TypeSizes
 return BOOL_SIZE;
 }
 
+public static int sizeof(byte value)
+{
+return BYTE_SIZE;
+}
+
 public static int sizeof(short value)
 {
 return SHORT_SIZE;


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-11-18 Thread slebresne 
Merge branch 'cassandra-3.0' into cassandra-3.11

* cassandra-3.0:
  Fix serialized size of DataLimits


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d2bc37f4
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d2bc37f4
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d2bc37f4

Branch: refs/heads/cassandra-3.11
Commit: d2bc37f43db14661a159406e421985a2484d5387
Parents: 9ee44db eb05025
Author: Sylvain Lebresne 
Authored: Sat Nov 18 14:17:43 2017 +0100
Committer: Sylvain Lebresne 
Committed: Sat Nov 18 14:17:43 2017 +0100

--
 CHANGES.txt | 1 +
 src/java/org/apache/cassandra/db/TypeSizes.java | 6 ++
 2 files changed, 7 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d2bc37f4/CHANGES.txt
--
diff --cc CHANGES.txt
index a1a1a37,6c50a3f..3fc934e
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,9 -1,5 +1,10 @@@
 -3.0.16
 +3.11.2
 + * Add asm jar to build.xml for maven builds (CASSANDRA-11193)
 + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
 + * Update jackson JSON jars (CASSANDRA-13949)
 + * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
 +Merged from 3.0:
+  * Fix serialized size of DataLimits (CASSANDRA-14057)
   * Add flag to allow dropping oversized read repair mutations 
(CASSANDRA-13975)
   * Fix SSTableLoader logger message (CASSANDRA-14003)
   * Fix repair race that caused gossip to block (CASSANDRA-13849)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[1/6] cassandra git commit: Fix serialized size of DataLimits

2017-11-18 Thread slebresne 
Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 f1e850a49 -> eb05025c0
  refs/heads/cassandra-3.11 9ee44db49 -> d2bc37f43
  refs/heads/trunk f1f6ed609 -> da410153e


Fix serialized size of DataLimits

patch by Sylvain Lebresne; reviewed by Jason Brown for CASSANDRA-14057


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/eb05025c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/eb05025c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/eb05025c

Branch: refs/heads/cassandra-3.0
Commit: eb05025c0a768241ea61fd86db9a88cfd8f6e93e
Parents: f1e850a
Author: Sylvain Lebresne 
Authored: Fri Nov 17 15:46:58 2017 +0100
Committer: Sylvain Lebresne 
Committed: Sat Nov 18 14:16:42 2017 +0100

--
 CHANGES.txt | 1 +
 src/java/org/apache/cassandra/db/TypeSizes.java | 6 ++
 2 files changed, 7 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb05025c/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index a3c43fd..6c50a3f 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * Fix serialized size of DataLimits (CASSANDRA-14057)
  * Add flag to allow dropping oversized read repair mutations (CASSANDRA-13975)
  * Fix SSTableLoader logger message (CASSANDRA-14003)
  * Fix repair race that caused gossip to block (CASSANDRA-13849)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb05025c/src/java/org/apache/cassandra/db/TypeSizes.java
--
diff --git a/src/java/org/apache/cassandra/db/TypeSizes.java 
b/src/java/org/apache/cassandra/db/TypeSizes.java
index 6543025..b47e300 100644
--- a/src/java/org/apache/cassandra/db/TypeSizes.java
+++ b/src/java/org/apache/cassandra/db/TypeSizes.java
@@ -28,6 +28,7 @@ public final class TypeSizes
 private TypeSizes(){}
 
 private static final int BOOL_SIZE = 1;
+private static final int BYTE_SIZE = 1;
 private static final int SHORT_SIZE = 2;
 private static final int INT_SIZE = 4;
 private static final int LONG_SIZE = 8;
@@ -78,6 +79,11 @@ public final class TypeSizes
 return BOOL_SIZE;
 }
 
+public static int sizeof(byte value)
+{
+return BYTE_SIZE;
+}
+
 public static int sizeof(short value)
 {
 return SHORT_SIZE;


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12245) initial view build can be parallel

2017-11-18 Thread JIRA 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258054#comment-16258054
 ] 

Andrés de la Peña commented on CASSANDRA-12245:
---

Thanks for the comments, this is almost finished :)

[Here|https://github.com/apache/cassandra/compare/trunk...adelapena:12245-trunk]
 is the new version of the patch, rebased and squashed. The udpated dtests can 
be found 
[here|https://github.com/apache/cassandra-dtest/compare/master...adelapena:12245].

bq. One minor thing is that we should probably only split the view build tasks 
at all if the base table is larger than a given size (let's say 500MB or so?), 
to avoid 4 * num_processor flushes for base tables with negligible size, WDYT?

As discussed, I have moved the base table flush from {{ViewBuilderTask}} to 
{{ViewBuilder}} 
[here|https://github.com/adelapena/cassandra/commit/478ed88b490378caf4f8ddc82c8e3aa3f90e5264],
 to do a single flush at the begining of the view build. The following writes 
will be writen to the MV through the regular path so it seems that they won't 
need any further flushes. I think that with this we don't need to check the 
table size and give a special treatment to small ones, what do you think?

bq. I noticed we don't stop in-progess view builds when a view is removed, 
would you mind adding that?

Right, good catch. Done 
[here|https://github.com/adelapena/cassandra/commit/e1ace2f47be71d48ab1987d0e2c7a07cc9486e97].
 I have also added [this 
dtest|https://github.com/adelapena/cassandra-dtest/blob/12245/materialized_views_test.py#L1025-L1067]
 to verify that the build is properly stopped.

bq. ViewBuildExecutor is being constructed with minThreads=1 and 
maxPoolSize=concurrent_materialized_view_builders, but according to the 
{{DebuggableThreadPoolExecutor}}'s' 
[javadoc|https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/concurrent/DebuggableThreadPoolExecutor.java#L33],
 this will actually make the executor with size 1 since maxPoolSize is not 
supported by {{DebuggableThreadPoolExecutor}} - and even if it were, new 
threads would only be created after the queue of the initial threads were full 
(which is quite unintuitive), but we actually want the pool to have 
concurrent_materialized_view_builders concurrent threads at most, so we should 
use the {{threadCount}} constructor instead - at some point we should actually 
remove the maximumPoolSize

Done 
[here|https://github.com/adelapena/cassandra/commit/fc14b034bb5d36c23435f313541445dc5adb0078].

bq. I think we could take a {{buildAllViews}} parameter on reload, and set that 
to false during Keyspace initialization, since views will be build during 
daemon initialization and keyspace changes anyway, WDYT?

Makes sense, done 
[here|https://github.com/adelapena/cassandra/commit/c4f19a5461434c0d5ca5e1301d92da26cca5083e].

bq. One last thing, can you please add the new yaml option 
{{concurrent_materialized_view_builders}} to the configuration section of the 
doc?

It seems that [the configuration 
section|https://github.com/apache/cassandra/blob/trunk/doc/source/configuration/index.rst]
 of the doc is currently empty. I think that writting this section (structure, 
introduction, etc.) is probably out of the scope of this ticket and it might be 
done in a separate, dedicated ticket. Instead, I have 
[updated|https://github.com/adelapena/cassandra/commit/30293f852584189a5b46c2dce5ae4042ae62d3e4]
 the NEWS.txt file with more detailed info and I have added [a 
note|https://github.com/adelapena/cassandra/commit/82c446398d0b6b4b1b13b35b3502489fc71fe703]
 to the doc about {{CREATE MATERIALIZED VIEW}} statement. WDYT?

I have updated the dtest {{interrupt_build_process_test}} to make sure that the 
build is really interrupted also in 3.x through [new byteman 
scripts|https://github.com/adelapena/cassandra-dtest/blob/f7aac39ee5d0c661b2f7f5b1db2a7347635f85c5/materialized_views_test.py#L962-L963].
 Without that, the build could finish before the cluster stop.

The CI results look good, at least for MVs:
||[utest|http://jenkins-cassandra.datastax.lan/view/Dev/view/adelapena/job/adelapena-12245-trunk-testall/]||[dtest|http://jenkins-cassandra.datastax.lan/view/Dev/view/adelapena/job/adelapena-12245-trunk-dtest/]||

> initial view build can be parallel
> --
>
> Key: CASSANDRA-12245
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12245
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Materialized Views
>Reporter: Tom van der Woerdt
>Assignee: Andrés de la Peña
> Fix For: 4.x
>
>
> On a node with lots of data (~3TB) building a materialized view takes several 
> weeks, which is not ideal. It's doing this in a single thread.
> There are several potential ways this can be