[jira] [Commented] (CASSANDRA-16127) NullPointerException when calling nodetool enablethrift
[
https://issues.apache.org/jira/browse/CASSANDRA-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198618#comment-17198618
]
David Capwell commented on CASSANDRA-16127:
---
Ok trunk is finally stable; patch is ready to review.
> NullPointerException when calling nodetool enablethrift
> ---
>
> Key: CASSANDRA-16127
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16127
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Thrift
>Reporter: Tibor Repasi
>Assignee: David Capwell
>Priority: Normal
> Fix For: 2.2.x, 3.0.x, 3.11.x
>
>
> Having thrift disabled, it's impossible to enable it again without restarting
> the node:
> {code}
> $ nodetool statusthrift
> not running
> $ nodetool enablethrift
> error: null
> -- StackTrace --
> java.lang.NullPointerException
> at
> org.apache.cassandra.service.StorageService.startRPCServer(StorageService.java:392)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16102) Build target for shaded in-JVM dtest jar
[ https://issues.apache.org/jira/browse/CASSANDRA-16102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198605#comment-17198605 ] David Capwell commented on CASSANDRA-16102: --- tested reverting the snakeyaml changes and the build passed again. > Build target for shaded in-JVM dtest jar > > > Key: CASSANDRA-16102 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16102 > Project: Cassandra > Issue Type: Task > Components: Build >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: High > Fix For: 4.0-beta > > > Several small changes that are required as a prerequisite for releasing > [Harry|https://issues.apache.org/jira/browse/CASSANDRA-15348]: > 1. Update snakeYaml in Cassandra > 2. Add a shade maven target and packaging script for shaded dtest artifacts > This patch is only a bridge for everyone to test out Harry with trunk/4.0, > before we're ready to build in-JVM dtest jars for every version, and stop > depending on java-driver (and, subsequently, on Netty and Guava) in Harry. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16102) Build target for shaded in-JVM dtest jar
[
https://issues.apache.org/jira/browse/CASSANDRA-16102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198603#comment-17198603
]
David Capwell edited comment on CASSANDRA-16102 at 9/19/20, 12:37 AM:
--
FYI org.apache.cassandra.db.commitlog.CommitLogInitWithExceptionTest now fails
{code}
java.lang.ClassCastException: org.apache.cassandra.config.Config cannot be cast
to org.apache.cassandra.config.Config
at
org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:170)
at
org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:126)
at
org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:103)
at
org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:299)
at
org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:175)
at
org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:160)
at
org.apache.cassandra.db.commitlog.CommitLogInitWithExceptionTest.setUp(CommitLogInitWithExceptionTest.java:39)
{code}
this fails the same way consistently with this commit, and doesn't fail with
previous
build:
https://app.circleci.com/pipelines/github/dcapwell/cassandra/545/workflows/9ac34efd-4c0f-4f10-ad8e-d1a1e181ac58/jobs/3004
was (Author: dcapwell):
FYI org.apache.cassandra.db.commitlog.CommitLogInitWithExceptionTest now fails
{code}
java.lang.ClassCastException: org.apache.cassandra.config.Config cannot be cast
to org.apache.cassandra.config.Config
at
org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:170)
at
org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:126)
at
org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:103)
at
org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:299)
at
org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:175)
at
org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:160)
at
org.apache.cassandra.db.commitlog.CommitLogInitWithExceptionTest.setUp(CommitLogInitWithExceptionTest.java:39)
{code}
this fails the same way consistently with this commit, and doesn't fail with
previous
> Build target for shaded in-JVM dtest jar
>
>
> Key: CASSANDRA-16102
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16102
> Project: Cassandra
> Issue Type: Task
> Components: Build
>Reporter: Alex Petrov
>Assignee: Alex Petrov
>Priority: High
> Fix For: 4.0-beta
>
>
> Several small changes that are required as a prerequisite for releasing
> [Harry|https://issues.apache.org/jira/browse/CASSANDRA-15348]:
> 1. Update snakeYaml in Cassandra
> 2. Add a shade maven target and packaging script for shaded dtest artifacts
> This patch is only a bridge for everyone to test out Harry with trunk/4.0,
> before we're ready to build in-JVM dtest jars for every version, and stop
> depending on java-driver (and, subsequently, on Netty and Guava) in Harry.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16102) Build target for shaded in-JVM dtest jar
[
https://issues.apache.org/jira/browse/CASSANDRA-16102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198603#comment-17198603
]
David Capwell commented on CASSANDRA-16102:
---
FYI org.apache.cassandra.db.commitlog.CommitLogInitWithExceptionTest now fails
{code}
java.lang.ClassCastException: org.apache.cassandra.config.Config cannot be cast
to org.apache.cassandra.config.Config
at
org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:170)
at
org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:126)
at
org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:103)
at
org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:299)
at
org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:175)
at
org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:160)
at
org.apache.cassandra.db.commitlog.CommitLogInitWithExceptionTest.setUp(CommitLogInitWithExceptionTest.java:39)
{code}
this fails the same way consistently with this commit, and doesn't fail with
previous
> Build target for shaded in-JVM dtest jar
>
>
> Key: CASSANDRA-16102
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16102
> Project: Cassandra
> Issue Type: Task
> Components: Build
>Reporter: Alex Petrov
>Assignee: Alex Petrov
>Priority: High
> Fix For: 4.0-beta
>
>
> Several small changes that are required as a prerequisite for releasing
> [Harry|https://issues.apache.org/jira/browse/CASSANDRA-15348]:
> 1. Update snakeYaml in Cassandra
> 2. Add a shade maven target and packaging script for shaded dtest artifacts
> This patch is only a bridge for everyone to test out Harry with trunk/4.0,
> before we're ready to build in-JVM dtest jars for every version, and stop
> depending on java-driver (and, subsequently, on Netty and Guava) in Harry.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Issue Comment Deleted] (CASSANDRA-15965) Fix flaky test StreamingInboundHandlerTest channelRead_Normal
[
https://issues.apache.org/jira/browse/CASSANDRA-15965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Capwell updated CASSANDRA-15965:
--
Comment: was deleted
(was: FYI still failing:
https://app.circleci.com/pipelines/github/dcapwell/cassandra/544/workflows/3349686a-a375-474d-8a85-e0c0b0fa9fd2/jobs/2996)
> Fix flaky test StreamingInboundHandlerTest channelRead_Normal
> -
>
> Key: CASSANDRA-15965
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15965
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: David Capwell
>Assignee: Adam Holmberg
>Priority: Normal
> Fix For: 4.0-beta3
>
>
> channelRead_Normal -
> org.apache.cassandra.streaming.async.StreamingInboundHandlerTest
> {code}
> junit.framework.AssertionFailedError: expected:<8> but was:<0>
> at
> org.apache.cassandra.streaming.async.StreamingInboundHandlerTest.channelRead_Normal(StreamingInboundHandlerTest.java:98)
> {code}
> Failed build:
> https://app.circleci.com/pipelines/github/dcapwell/cassandra/298/workflows/e3296f33-2289-401c-8fc8-a7f786e3692a/jobs/1445
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15214) OOMs caught and not rethrown
[
https://issues.apache.org/jira/browse/CASSANDRA-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198585#comment-17198585
]
Yifan Cai commented on CASSANDRA-15214:
---
> where would that exception end up if it were rethrown
{{JVMStabilityInspector}} only re-throws {{OutOfMemoryError}}. Depending on the
presence of those OOM-related JVM options, {{OnOutOfMemoryError}},
{{ExitOnOutOfMemoryError}} or {{HeapDumpOnOutOfMemoryError}}, the JVM exits and
trigger a heap dump if it is a heap space OOM error.
However, the call-sites indicate to not re-throw OOM error (e.g.
[here|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/InboundMessageHandler.java#L647-L659]),
which I'd like to learn why we do not let the JVM to exit.
Netty by default just logs the exception, when {{exceptionCaught()}} is _not_
implemented in any of the handler in the inbound direction. For the outbound,
client code handles exception by adding listener to {{ChannelFuture}} or
{{ChannelPromise}}. We have the handling in both directions.
Besides the inbound/outbound pathes, it looks like that Netty does do a lot of
catch-{{Throwable}}-and-swallow things in its code base. So it is possible that
errors from Netty internal are not bubbled up. For example, this
[issue|https://github.com/netty/netty/issues/6096].
> OOMs caught and not rethrown
>
>
> Key: CASSANDRA-15214
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15214
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Client, Messaging/Internode
>Reporter: Benedict Elliott Smith
>Assignee: Yifan Cai
>Priority: Normal
> Fix For: 4.0, 4.0-rc
>
> Attachments: oom-experiments.zip
>
>
> Netty (at least, and perhaps elsewhere in Executors) catches all exceptions,
> so presently there is no way to ensure that an OOM reaches the JVM handler to
> trigger a crash/heapdump.
> It may be that the simplest most consistent way to do this would be to have a
> single thread spawned at startup that waits for any exceptions we must
> propagate to the Runtime.
> We could probably submit a patch upstream to Netty, but for a guaranteed
> future proof approach, it may be worth paying the cost of a single thread.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15965) Fix flaky test StreamingInboundHandlerTest channelRead_Normal
[
https://issues.apache.org/jira/browse/CASSANDRA-15965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198578#comment-17198578
]
David Capwell commented on CASSANDRA-15965:
---
FYI still failing:
https://app.circleci.com/pipelines/github/dcapwell/cassandra/544/workflows/3349686a-a375-474d-8a85-e0c0b0fa9fd2/jobs/2996
> Fix flaky test StreamingInboundHandlerTest channelRead_Normal
> -
>
> Key: CASSANDRA-15965
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15965
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: David Capwell
>Assignee: Adam Holmberg
>Priority: Normal
> Fix For: 4.0-beta3
>
>
> channelRead_Normal -
> org.apache.cassandra.streaming.async.StreamingInboundHandlerTest
> {code}
> junit.framework.AssertionFailedError: expected:<8> but was:<0>
> at
> org.apache.cassandra.streaming.async.StreamingInboundHandlerTest.channelRead_Normal(StreamingInboundHandlerTest.java:98)
> {code}
> Failed build:
> https://app.circleci.com/pipelines/github/dcapwell/cassandra/298/workflows/e3296f33-2289-401c-8fc8-a7f786e3692a/jobs/1445
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16057) Should update in-jvm dtest to expose stdout and stderr for nodetool
[ https://issues.apache.org/jira/browse/CASSANDRA-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198563#comment-17198563 ] David Capwell commented on CASSANDRA-16057: --- committed jvm-dtest https://github.com/apache/cassandra-in-jvm-dtest-api/commit/5b26885187d0aa8bc09cd1b1f4b6be81aebb40c7 > Should update in-jvm dtest to expose stdout and stderr for nodetool > --- > > Key: CASSANDRA-16057 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16057 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest/java >Reporter: David Capwell >Assignee: Yifan Cai >Priority: Normal > Time Spent: 2h 20m > Remaining Estimate: 0h > > Many nodetool commands output to stdout or stderr so running nodetool using > in-jvm dtest should expose that to tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-in-jvm-dtest-api] branch master updated: Update in-jvm dtest to expose stdout and stderr for nodetool
This is an automated email from the ASF dual-hosted git repository.
dcapwell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-in-jvm-dtest-api.git
The following commit(s) were added to refs/heads/master by this push:
new 5b26885 Update in-jvm dtest to expose stdout and stderr for nodetool
5b26885 is described below
commit 5b26885187d0aa8bc09cd1b1f4b6be81aebb40c7
Author: Yifan Cai
AuthorDate: Fri Sep 18 15:09:02 2020 -0700
Update in-jvm dtest to expose stdout and stderr for nodetool
Patch by Yifan Cai; reviewed by Alex Petrov, David Capwell for
CASSANDRA-16057
---
.../cassandra/distributed/api/NodeToolResult.java | 64 +++
.../distributed/api/NodeToolOutputTest.java| 74 ++
2 files changed, 138 insertions(+)
diff --git
a/src/main/java/org/apache/cassandra/distributed/api/NodeToolResult.java
b/src/main/java/org/apache/cassandra/distributed/api/NodeToolResult.java
index bdd75b5..6cfe4e5 100644
--- a/src/main/java/org/apache/cassandra/distributed/api/NodeToolResult.java
+++ b/src/main/java/org/apache/cassandra/distributed/api/NodeToolResult.java
@@ -36,13 +36,22 @@ public class NodeToolResult
private final int rc;
private final List notifications;
private final Throwable error;
+private final String stdout;
+private final String stderr;
public NodeToolResult(String[] commandAndArgs, int rc, List
notifications, Throwable error)
{
+this(commandAndArgs, rc, notifications, error, null, null);
+}
+
+public NodeToolResult(String[] commandAndArgs, int rc, List
notifications, Throwable error, String stdout, String stderr)
+{
this.commandAndArgs = commandAndArgs;
this.rc = rc;
this.notifications = notifications;
this.error = error;
+this.stdout = stdout;
+this.stderr = stderr;
}
public String[] getCommandAndArgs()
@@ -65,6 +74,16 @@ public class NodeToolResult
return error;
}
+public String getStdout()
+{
+return stdout;
+}
+
+public String getStderr()
+{
+return stderr;
+}
+
public Asserts asserts()
{
return new Asserts();
@@ -134,6 +153,51 @@ public class NodeToolResult
return this; // unreachable
}
+public Asserts stdoutContains(String substring)
+{
+return assertConsoleOutput(substring, true, true);
+}
+
+public Asserts stderrContains(String substring)
+{
+return assertConsoleOutput(substring, false, true);
+}
+
+public Asserts stdoutNotContains(String substring)
+{
+return assertConsoleOutput(substring, true, false);
+}
+
+public Asserts stderrNotContains(String substring)
+{
+return assertConsoleOutput(substring, false, false);
+}
+
+private Asserts assertConsoleOutput(String substring, boolean
isStdout, boolean assertContains)
+{
+String name = isStdout ? "stdout" : "stderr";
+String output = isStdout ? stdout : stderr;
+AssertUtils.assertNotNull(name + " not defined", output);
+if (assertContains)
+{
+if (output.contains(substring))
+{
+return this;
+}
+fail("Unable to locate substring '" + substring + "' in " +
name + ": " + output);
+}
+else
+{
+if (!output.contains(substring))
+{
+return this;
+}
+fail("Found unexpected substring '" + substring + "' in " +
name + ": " + output);
+}
+
+return this; // unreachable
+}
+
private void fail(String message)
{
StringBuilder sb = new StringBuilder();
diff --git
a/src/test/java/org/apache/cassandra/distributed/api/NodeToolOutputTest.java
b/src/test/java/org/apache/cassandra/distributed/api/NodeToolOutputTest.java
new file mode 100644
index 000..4420485
--- /dev/null
+++ b/src/test/java/org/apache/cassandra/distributed/api/NodeToolOutputTest.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
[jira] [Updated] (CASSANDRA-16057) Should update in-jvm dtest to expose stdout and stderr for nodetool
[ https://issues.apache.org/jira/browse/CASSANDRA-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Capwell updated CASSANDRA-16057: -- Reviewers: Alex Petrov, David Capwell (was: David Capwell) > Should update in-jvm dtest to expose stdout and stderr for nodetool > --- > > Key: CASSANDRA-16057 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16057 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest/java >Reporter: David Capwell >Assignee: Yifan Cai >Priority: Normal > Time Spent: 2h 20m > Remaining Estimate: 0h > > Many nodetool commands output to stdout or stderr so running nodetool using > in-jvm dtest should expose that to tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16057) Should update in-jvm dtest to expose stdout and stderr for nodetool
[ https://issues.apache.org/jira/browse/CASSANDRA-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Capwell updated CASSANDRA-16057: -- Status: Ready to Commit (was: Review In Progress) > Should update in-jvm dtest to expose stdout and stderr for nodetool > --- > > Key: CASSANDRA-16057 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16057 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest/java >Reporter: David Capwell >Assignee: Yifan Cai >Priority: Normal > Time Spent: 2h 20m > Remaining Estimate: 0h > > Many nodetool commands output to stdout or stderr so running nodetool using > in-jvm dtest should expose that to tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16057) Should update in-jvm dtest to expose stdout and stderr for nodetool
[ https://issues.apache.org/jira/browse/CASSANDRA-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198562#comment-17198562 ] David Capwell commented on CASSANDRA-16057: --- +1 the new changes (only skimmed trunk for the latest changes) Will commit the jvm-dtest patch. > Should update in-jvm dtest to expose stdout and stderr for nodetool > --- > > Key: CASSANDRA-16057 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16057 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest/java >Reporter: David Capwell >Assignee: Yifan Cai >Priority: Normal > Time Spent: 2h 20m > Remaining Estimate: 0h > > Many nodetool commands output to stdout or stderr so running nodetool using > in-jvm dtest should expose that to tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16120) Add ability for jvm-dtest to grep instance logs
[
https://issues.apache.org/jira/browse/CASSANDRA-16120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198557#comment-17198557
]
David Capwell commented on CASSANDRA-16120:
---
merged jvm-dtest api.
> Add ability for jvm-dtest to grep instance logs
> ---
>
> Key: CASSANDRA-16120
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16120
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/dtest/java
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-beta
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> One of the main gaps between python dtest and jvm dtest is python dtest
> supports the ability to grep the logs of an instance; we need this capability
> as some tests require validating logs were triggered.
> Pydocs for common log methods
> {code}
> | grep_log(self, expr, filename='system.log', from_mark=None)
> | Returns a list of lines matching the regular expression in parameter
> | in the Cassandra log of this node
> |
> | grep_log_for_errors(self, filename='system.log')
> | Returns a list of errors with stack traces
> | in the Cassandra log of this node
> |
> | grep_log_for_errors_from(self, filename='system.log', seek_start=0)
> {code}
> {code}
> | watch_log_for(self, exprs, from_mark=None, timeout=600, process=None,
> verbose=False, filename='system.log')
> | Watch the log until one or more (regular) expression are found.
> | This methods when all the expressions have been found or the method
> | timeouts (a TimeoutError is then raised). On successful completion,
> | a list of pair (line matched, match object) is returned.
> {code}
> Below is a POC showing a way to do such logic
> {code}
> package org.apache.cassandra.distributed.test;
> import java.io.BufferedReader;
> import java.io.FileInputStream;
> import java.io.IOException;
> import java.io.InputStreamReader;
> import java.io.UncheckedIOException;
> import java.nio.charset.StandardCharsets;
> import java.util.Iterator;
> import java.util.Spliterator;
> import java.util.Spliterators;
> import java.util.regex.Matcher;
> import java.util.regex.Pattern;
> import java.util.stream.Stream;
> import java.util.stream.StreamSupport;
> import com.google.common.io.Closeables;
> import org.junit.Test;
> import org.apache.cassandra.distributed.Cluster;
> import org.apache.cassandra.utils.AbstractIterator;
> public class AllTheLogs extends TestBaseImpl
> {
>@Test
>public void test() throws IOException
>{
>try (final Cluster cluster = init(Cluster.build(1).start()))
>{
>String tag = System.getProperty("cassandra.testtag",
> "cassandra.testtag_IS_UNDEFINED");
>String suite = System.getProperty("suitename",
> "suitename_IS_UNDEFINED");
>String log = String.format("build/test/logs/%s/TEST-%s.log", tag,
> suite);
>grep(log, "Enqueuing flush of tables").forEach(l ->
> System.out.println("I found the thing: " + l));
>}
>}
>private static Stream grep(String file, String regex) throws
> IOException
>{
>return grep(file, Pattern.compile(regex));
>}
>private static Stream grep(String file, Pattern regex) throws
> IOException
>{
>BufferedReader reader = new BufferedReader(new InputStreamReader(new
> FileInputStream(file), StandardCharsets.UTF_8));
>Iterator it = new AbstractIterator()
>{
>protected String computeNext()
>{
>try
>{
>String s;
>while ((s = reader.readLine()) != null)
>{
>Matcher m = regex.matcher(s);
>if (m.find())
>return s;
>}
>reader.close();
>return endOfData();
>}
>catch (IOException e)
>{
>Closeables.closeQuietly(reader);
>throw new UncheckedIOException(e);
>}
>}
>};
>return StreamSupport.stream(Spliterators.spliteratorUnknownSize(it,
> Spliterator.ORDERED), false);
>}
> }
> {code}
> And
> {code}
> @Test
>public void test() throws IOException
>{
>try (final Cluster cluster = init(Cluster.build(1).start()))
>{
>String tag = System.getProperty("cassandra.testtag",
> "cassandra.testtag_IS_UNDEFINED");
>String suite = System.getProperty("suitename",
> "suitename_IS_UNDEFINED");
>//TODO missing way to get node id
> //cluster.get(1);
>String log =
>
[cassandra-in-jvm-dtest-api] branch master updated: Add ability for jvm-dtest to grep instance logs
This is an automated email from the ASF dual-hosted git repository.
dcapwell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-in-jvm-dtest-api.git
The following commit(s) were added to refs/heads/master by this push:
new 672af9b Add ability for jvm-dtest to grep instance logs
672af9b is described below
commit 672af9b56f1729c6511a2279923eb435df4b7b9b
Author: David Capwell
AuthorDate: Fri Sep 18 14:53:07 2020 -0700
Add ability for jvm-dtest to grep instance logs
Patch by David Capwell; reviewed by Alex Petrov, Yifan Cai for
CASSANDRA-16120
---
pom.xml| 2 +-
.../cassandra/distributed/api/IInstance.java | 19 +
.../cassandra/distributed/api/LineIterator.java| 37 ++
.../cassandra/distributed/api/LogAction.java | 416 +
.../cassandra/distributed/api/LogResult.java | 25 ++
.../cassandra/distributed/api/LogActionTest.java | 242
6 files changed, 740 insertions(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index 49113d7..020b404 100644
--- a/pom.xml
+++ b/pom.xml
@@ -70,7 +70,7 @@
test
- org.mockito
+org.mockito
mockito-core
3.5.10
test
diff --git a/src/main/java/org/apache/cassandra/distributed/api/IInstance.java
b/src/main/java/org/apache/cassandra/distributed/api/IInstance.java
index 496d33d..4ffc36a 100644
--- a/src/main/java/org/apache/cassandra/distributed/api/IInstance.java
+++ b/src/main/java/org/apache/cassandra/distributed/api/IInstance.java
@@ -90,4 +90,23 @@ public interface IInstance extends IIsolatedExecutor
void forceCompact(String keyspace, String table);
+List getUncaughtExceptions();
+
+default boolean getLogsEnabled()
+{
+try
+{
+logs();
+return true;
+}
+catch (UnsupportedOperationException e)
+{
+return false;
+}
+}
+
+default LogAction logs()
+{
+throw new UnsupportedOperationException();
+}
}
diff --git
a/src/main/java/org/apache/cassandra/distributed/api/LineIterator.java
b/src/main/java/org/apache/cassandra/distributed/api/LineIterator.java
new file mode 100644
index 000..8971232
--- /dev/null
+++ b/src/main/java/org/apache/cassandra/distributed/api/LineIterator.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.distributed.api;
+
+import java.util.Iterator;
+
+public interface LineIterator extends Iterator, AutoCloseable
+{
+/**
+ * @return current position of the iterator
+ * @see LogAction#grep(long, String)
+ * @see LogAction#watchFor(long, String)
+ */
+long mark();
+
+@Override
+default void close()
+{
+
+}
+}
diff --git a/src/main/java/org/apache/cassandra/distributed/api/LogAction.java
b/src/main/java/org/apache/cassandra/distributed/api/LogAction.java
new file mode 100644
index 000..8f9e693
--- /dev/null
+++ b/src/main/java/org/apache/cassandra/distributed/api/LogAction.java
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.distributed.api;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import
[jira] [Commented] (CASSANDRA-16057) Should update in-jvm dtest to expose stdout and stderr for nodetool
[
https://issues.apache.org/jira/browse/CASSANDRA-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198556#comment-17198556
]
Yifan Cai commented on CASSANDRA-16057:
---
Rebased onto trunk. 2 files conflicted and resolved.
* {{src/java/org/apache/cassandra/tools/nodetool/NetStats.java}}
* {{test/distributed/org/apache/cassandra/distributed/impl/Instance.java}}
> Should update in-jvm dtest to expose stdout and stderr for nodetool
> ---
>
> Key: CASSANDRA-16057
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16057
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/dtest/java
>Reporter: David Capwell
>Assignee: Yifan Cai
>Priority: Normal
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Many nodetool commands output to stdout or stderr so running nodetool using
> in-jvm dtest should expose that to tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16057) Should update in-jvm dtest to expose stdout and stderr for nodetool
[ https://issues.apache.org/jira/browse/CASSANDRA-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198522#comment-17198522 ] Alex Petrov commented on CASSANDRA-16057: - +1 with new changes. Thank you for the patch! > Should update in-jvm dtest to expose stdout and stderr for nodetool > --- > > Key: CASSANDRA-16057 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16057 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest/java >Reporter: David Capwell >Assignee: Yifan Cai >Priority: Normal > Time Spent: 2h 20m > Remaining Estimate: 0h > > Many nodetool commands output to stdout or stderr so running nodetool using > in-jvm dtest should expose that to tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16102) Build target for shaded in-JVM dtest jar
[ https://issues.apache.org/jira/browse/CASSANDRA-16102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Petrov updated CASSANDRA-16102: Fix Version/s: 4.0-beta Source Control Link: https://github.com/apache/cassandra/commit/0dc5bd51f5ff36434bf7b5244242977ecbb47e39 Resolution: Fixed Status: Resolved (was: Ready to Commit) > Build target for shaded in-JVM dtest jar > > > Key: CASSANDRA-16102 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16102 > Project: Cassandra > Issue Type: Task > Components: Build >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: High > Fix For: 4.0-beta > > > Several small changes that are required as a prerequisite for releasing > [Harry|https://issues.apache.org/jira/browse/CASSANDRA-15348]: > 1. Update snakeYaml in Cassandra > 2. Add a shade maven target and packaging script for shaded dtest artifacts > This patch is only a bridge for everyone to test out Harry with trunk/4.0, > before we're ready to build in-JVM dtest jars for every version, and stop > depending on java-driver (and, subsequently, on Netty and Guava) in Harry. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Changes required for Harry build
This is an automated email from the ASF dual-hosted git repository.
ifesdjeen pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new 0dc5bd5 Changes required for Harry build
0dc5bd5 is described below
commit 0dc5bd51f5ff36434bf7b5244242977ecbb47e39
Author: Alex Petrov
AuthorDate: Thu Sep 3 16:00:31 2020 +0200
Changes required for Harry build
Patch by Alex Petrov; reviewed by Marcus Eriksson for CASSANDRA-16102
---
build-shaded-dtest-jar.sh | 38 ++
build.xml | 2 +-
lib/snakeyaml-1.11.jar | Bin 270552 -> 0 bytes
lib/snakeyaml-1.23.jar | Bin 0 -> 301298 bytes
relocate-dependencies.pom | 132 +
.../cassandra/config/YamlConfigurationLoader.java | 23 ++--
6 files changed, 186 insertions(+), 9 deletions(-)
diff --git a/build-shaded-dtest-jar.sh b/build-shaded-dtest-jar.sh
new file mode 100755
index 000..efedcaa
--- /dev/null
+++ b/build-shaded-dtest-jar.sh
@@ -0,0 +1,38 @@
+#!/bin/bash
+
+set -xe
+
+# Cassandra version that is specified in build.xml
+CASSANDRA_VERSION=$1
+# In-JVM dtest version that is specified in relocate-dependencies.pom
+DTEST_VERSION=$2
+ARTIFACT_NAME=cassandra-dtest
+REPO_DIR=~/.m2/repository
+
+ant clean
+ant dtest-jar
+
+# Install the version that will be shaded
+mvn install:install-file \
+ -Dfile=./build/dtest-${CASSANDRA_VERSION}.jar \
+ -DgroupId=org.apache.cassandra \
+ -DartifactId=${ARTIFACT_NAME}-local \
+ -Dversion=${DTEST_VERSION} \
+ -Dpackaging=jar \
+ -DgeneratePom=true \
+ -DlocalRepositoryPath=${REPO_DIR}
+
+# Create shaded artifact
+mvn -f relocate-dependencies.pom package -DskipTests -nsu
+
+# Deploy shaded artifact
+mvn install:install-file \
+ -Dfile=./target/${ARTIFACT_NAME}-shaded-${DTEST_VERSION}.jar \
+ -DgroupId=org.apache.cassandra\
+ -DartifactId=${ARTIFACT_NAME}-shaded \
+ -Dversion=${DTEST_VERSION}\
+ -Dpackaging=jar \
+ -DgeneratePom=true\
+ -DlocalRepositoryPath=${REPO_DIR}
+
+set +xe
diff --git a/build.xml b/build.xml
index e77dd0b..8c9c49d 100644
--- a/build.xml
+++ b/build.xml
@@ -583,7 +583,7 @@
-
+
diff --git a/lib/snakeyaml-1.11.jar b/lib/snakeyaml-1.11.jar
deleted file mode 100644
index 3e237cd..000
Binary files a/lib/snakeyaml-1.11.jar and /dev/null differ
diff --git a/lib/snakeyaml-1.23.jar b/lib/snakeyaml-1.23.jar
new file mode 100644
index 000..adcef4f
Binary files /dev/null and b/lib/snakeyaml-1.23.jar differ
diff --git a/relocate-dependencies.pom b/relocate-dependencies.pom
new file mode 100644
index 000..0e5eabc
--- /dev/null
+++ b/relocate-dependencies.pom
@@ -0,0 +1,132 @@
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
+4.0.0
+
+org.apache.cassandra
+cassandra-dtest-shaded
+4.0.1-SNAPSHOT
+jar
+
+Cassandra in-jvm dtests shaded jar
+
+
+library
+1.8
+src/test/unit/java
+UTF-8
+
UTF-8
+
+4.12
+${java.version}
+${java.version}
+
+
+
+
+
+org.apache.cassandra
+cassandra-dtest-local
+4.0.0-SNAPSHOT
+
+
+
+junit
+junit
+${junit.version}
+test
+
+
+
+
+
+${test.source.directory}
+
+
+
+
+org.apache.maven.plugins
+maven-shade-plugin
+3.2.1
+
+
+false
+
+
+
+
org.apache.cassandra:cassandra-dtest-local
+
+
+
+
+
+io.netty
+
relocated.shaded.io.netty
+
+
+
+com.google
+
relocated.shaded.com.google
+
+
+
+com.datastax
+
relocated.shaded.com.datastax
+
+
+
+
+
+*:*
+
+**/Log4j2Plugins.dat
+
+
+
+
+
[jira] [Updated] (CASSANDRA-15406) Show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Capwell updated CASSANDRA-15406: -- Fix Version/s: (was: 4.0-beta) 4.0-beta3 Source Control Link: https://github.com/apache/cassandra/commit/b936d3e95db3180a2f1f32ed7c80543e81971fde Resolution: Fixed Status: Resolved (was: Ready to Commit) Spoke with Stefan in slack and merged as this patch doesn't conflict with the nodetool stdout patch, so no need to wait on it. CI results: https://app.circleci.com/pipelines/github/dcapwell/cassandra/537/workflows/57dac3da-9a5d-43f2-a9ae-7bd1a329 https://ci-cassandra.apache.org/job/Cassandra-devbranch/24/ > Show the progress of data streaming and index build > > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0-beta3 > > Time Spent: 3.5h > Remaining Estimate: 0h > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . > > PR [https://github.com/apache/cassandra/pull/711] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Show the progress of data streaming and index build
This is an automated email from the ASF dual-hosted git repository.
dcapwell pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new b936d3e Show the progress of data streaming and index build
b936d3e is described below
commit b936d3e95db3180a2f1f32ed7c80543e81971fde
Author: Stefan Miklosovic
AuthorDate: Fri Sep 18 11:40:19 2020 -0700
Show the progress of data streaming and index build
patch by Stefan Miklosovic; reviewed by Benjamin Lerer, Berenguer Blasi,
David Capwell for CASSANDRA-15406
---
CHANGES.txt| 1 +
.../streaming/CassandraCompressedStreamReader.java | 2 +-
.../db/streaming/CassandraStreamReader.java| 2 +-
.../apache/cassandra/streaming/ProgressInfo.java | 2 +-
.../cassandra/streaming/StreamReceiveTask.java | 4 +-
.../apache/cassandra/tools/nodetool/NetStats.java | 64 ++-
.../cassandra/distributed/impl/Instance.java | 14 +-
.../shared/NodeToolResultWithOutput.java | 48 ++
.../test/AbstractNetstatsBootstrapStreaming.java | 85
.../test/AbstractNetstatsStreaming.java| 548 +
...WithEntireSSTablesCompressionStreamingTest.java | 36 ++
...houtEntireSSTablesCompressionStreamingTest.java | 36 ++
.../test/NetstatsRepairStreamingTest.java | 88
.../cassandra/distributed/util/NodetoolUtils.java | 68 +++
14 files changed, 978 insertions(+), 20 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 4916faa..42b7aa7 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -12,6 +12,7 @@
* Make Table/Keyspace Metric Names Consistent With Each Other
(CASSANDRA-15909)
* Mutating sstable component may race with entire-sstable-streaming(ZCS)
causing checksum validation failure (CASSANDRA-15861)
* NPE thrown while updating speculative execution time if keyspace is removed
during task execution (CASSANDRA-15949)
+ * Show the progress of data streaming and index build (CASSANDRA-15406)
Merged from 3.11:
* Use IF NOT EXISTS for index and UDT create statements in snapshot schema
files (CASSANDRA-13935)
* Make sure LCS handles duplicate sstable added/removed notifications
correctly (CASSANDRA-14103)
diff --git
a/src/java/org/apache/cassandra/db/streaming/CassandraCompressedStreamReader.java
b/src/java/org/apache/cassandra/db/streaming/CassandraCompressedStreamReader.java
index 2491fe1..ff9e6f7 100644
---
a/src/java/org/apache/cassandra/db/streaming/CassandraCompressedStreamReader.java
+++
b/src/java/org/apache/cassandra/db/streaming/CassandraCompressedStreamReader.java
@@ -97,7 +97,7 @@ public class CassandraCompressedStreamReader extends
CassandraStreamReader
{
writePartition(deserializer, writer);
// when compressed, report total bytes of compressed
chunks read since remoteFile.size is the sum of chunks transferred
-session.progress(filename, ProgressInfo.Direction.IN,
cis.chunkBytesRead(), totalSize);
+session.progress(filename + '-' + fileSeqNum,
ProgressInfo.Direction.IN, cis.chunkBytesRead(), totalSize);
}
assert in.getBytesRead() == sectionLength;
}
diff --git
a/src/java/org/apache/cassandra/db/streaming/CassandraStreamReader.java
b/src/java/org/apache/cassandra/db/streaming/CassandraStreamReader.java
index 686d874..6835fad 100644
--- a/src/java/org/apache/cassandra/db/streaming/CassandraStreamReader.java
+++ b/src/java/org/apache/cassandra/db/streaming/CassandraStreamReader.java
@@ -124,7 +124,7 @@ public class CassandraStreamReader implements IStreamReader
{
writePartition(deserializer, writer);
// TODO move this to BytesReadTracker
-session.progress(writer.getFilename(),
ProgressInfo.Direction.IN, in.getBytesRead(), totalSize);
+session.progress(writer.getFilename() + '-' + fileSeqNum,
ProgressInfo.Direction.IN, in.getBytesRead(), totalSize);
}
logger.debug("[Stream #{}] Finished receiving file #{} from {}
readBytes = {}, totalSize = {}",
session.planId(), fileSeqNum, session.peer,
FBUtilities.prettyPrintMemory(in.getBytesRead()),
FBUtilities.prettyPrintMemory(totalSize));
diff --git a/src/java/org/apache/cassandra/streaming/ProgressInfo.java
b/src/java/org/apache/cassandra/streaming/ProgressInfo.java
index ac91855..2b306f8 100644
--- a/src/java/org/apache/cassandra/streaming/ProgressInfo.java
+++ b/src/java/org/apache/cassandra/streaming/ProgressInfo.java
@@ -110,7 +110,7 @@ public class ProgressInfo implements Serializable
{
StringBuilder sb = new StringBuilder(fileName);
sb.append(" ").append(currentBytes);
-sb.append("/").append(totalBytes).append(" bytes");
+
[jira] [Commented] (CASSANDRA-16101) Make sure we don't throw any uncaught exceptions during in-jvm dtests
[ https://issues.apache.org/jira/browse/CASSANDRA-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198504#comment-17198504 ] David Capwell commented on CASSANDRA-16101: --- +1 > Make sure we don't throw any uncaught exceptions during in-jvm dtests > - > > Key: CASSANDRA-16101 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16101 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest/java >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Labels: pull-request-available > > We should assert that we don't throw any uncaught exceptions when running > in-jvm dtests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16057) Should update in-jvm dtest to expose stdout and stderr for nodetool
[
https://issues.apache.org/jira/browse/CASSANDRA-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198496#comment-17198496
]
Yifan Cai commented on CASSANDRA-16057:
---
Updated the dtest-api patch with {{notContains}} asserts and added test cases.
> Should update in-jvm dtest to expose stdout and stderr for nodetool
> ---
>
> Key: CASSANDRA-16057
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16057
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/dtest/java
>Reporter: David Capwell
>Assignee: Yifan Cai
>Priority: Normal
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Many nodetool commands output to stdout or stderr so running nodetool using
> in-jvm dtest should expose that to tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16057) Should update in-jvm dtest to expose stdout and stderr for nodetool
[ https://issues.apache.org/jira/browse/CASSANDRA-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198489#comment-17198489 ] Yifan Cai commented on CASSANDRA-16057: --- Thanks for all the feedbacks. [~stefan.miklosovic], as David mentioned, this patch does what you described. Capture the nodetool output per instance. All nodetools are updated to write to a designated output. Additionally, it captures *only* the nodetool output. Meanwhile, redirecting the system out/err captures the server logs, which could make assertion a bit more complicated. [~ifesdjeen][~dcapwell], I have added 1 commit to address the comments. * rename class and variable * use public final fields instead of methods to avoid resource warnings > Should update in-jvm dtest to expose stdout and stderr for nodetool > --- > > Key: CASSANDRA-16057 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16057 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest/java >Reporter: David Capwell >Assignee: Yifan Cai >Priority: Normal > Time Spent: 2h 20m > Remaining Estimate: 0h > > Many nodetool commands output to stdout or stderr so running nodetool using > in-jvm dtest should expose that to tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16036) Add flag to disable chunk cache and disable by default
[
https://issues.apache.org/jira/browse/CASSANDRA-16036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198466#comment-17198466
]
David Capwell commented on CASSANDRA-16036:
---
Awesome, thanks for the feedback.
> Add flag to disable chunk cache and disable by default
> --
>
> Key: CASSANDRA-16036
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16036
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: 15229_128mb.png, 16036_128mb.png,
> async-profile.collapsed.svg,
> clustering-in-clause_latency_selects_baseline.png,
> clustering-in-clause_latency_selects_baseline_attempt3.png,
> clustering-in-clause_latency_under90_selects_baseline.png,
> clustering-in-clause_latency_under90_selects_baseline_attempt3.png,
> clustering-slice_latency_selects_baseline.png,
> clustering-slice_latency_under90_selects_baseline.png,
> medium-blobs_latency_selects_baseline.png,
> medium-blobs_latency_under90_selects_baseline.png,
> partition-single-row-read_latency_selects_baseline.png,
> partition-single-row-read_latency_under90_selects_baseline.png
>
>
> Chunk cache is enabled by default and doesn’t have a flag to disable without
> impacting networking. In performance testing 4.0 against 3.0 I found that
> reads were slower in 4.0 and after profiling found that the ChunkCache was
> partially to blame; after disabling the chunk cache, read performance had
> improved.
> {code}
> 40_w_cc-selects.hdr
> #[Mean= 11.50063, StdDeviation = 13.44014]
> #[Max =482.41254, Total count= 316477]
> #[Buckets = 25, SubBuckets = 262144]
> 40_wo_cc-selects.hdr
> #[Mean= 9.82115, StdDeviation = 10.14270]
> #[Max =522.36493, Total count= 317444]
> #[Buckets = 25, SubBuckets = 262144]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15299) CASSANDRA-13304 follow-up: improve checksumming and compression in protocol v5-beta
[
https://issues.apache.org/jira/browse/CASSANDRA-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198465#comment-17198465
]
Sam Tunnicliffe commented on CASSANDRA-15299:
-
Sorry it's been a while without any visible movement here, but I've just pushed
some more commits to address the latest comments from [~ifesdjeen] and
[~omichallat]. I've added some tests for protocol negotiation, correct handling
of corrupt messages and resource management.
{quote} * In CQLMessageHandler#processOneContainedMessage, when we can't
acquire capacity and, subsequently, we're not passing the frame further down
the line. Shouold we release the frame in this case, since usually we're
releasing the source frame after flush.
{quote}
Done, though we only need to do this when {{throwOnOverload == true}} as
otherwise we process the inflight request before applying backpressure.
{quote} * ReusableBuffer is unused. {quote}
Ah yes, removed
{quote} * Server has a few unused imports and eventExecutorGroup which is
unused.{quote}
Cleaned up the imports and removed eventExecutorGroup
{quote} * I'm not sure if we currently handle releasing corrupted frames.{quote}
For self-contained frames, there's nothing to do here as no resources have been
acquired before the corruption is detected, hence {{CorruptFrame::release}} is
a no-op. For frames which are part of a large message, there may be some
resource allocated before we discover corruption. This is ok though, as we
consume the frame, supplying it to the large message state machine, which
handles releasing the bufffers of the previous frames (if any). I've added a
test for this scenario which includes a check that everything allocated has
been freed.
{quote} Shouold we maybe make FrameSet auto-closeable and make sure we always
release buffers in finally? I've also made a similar change to processItem
which would add item to flushed to make sure it's released. That makes flushed
variable name not quite right though. {quote}
I've pulled in some of your change to {{processItem}} as it removes some
duplication around polling the queue. I've removed the condition in the
{{finally}} of {{LegacyFlusher}} though, since if we throw from
{{processQueue}} then {{doneWork}} will be false anyway, but there may have
been some items processed and waiting to flush. The trade off is calling
{{flushWrittenChannels}} even if there's no work to do, but that seems both
cheap and unlikely, what do you think?
As far as making {{FrameSet}} autoclosable, I don't think that's feasible,
given how they are created and accessed. Instead, I've moved the release of
source frame buffers out of {{finish}} and added a {{try..finally}} to ensure
we release them in the case that {{finish}} fails. I've also tried to address
one of your comment re: the memory management here by adding some comments.
They're probably not yet enough, but let me know if they are helpful at all.
{quote} We can (and probably should) open a separate ticket that could aim at
performance improvements around native protocol. {quote}
Agreed, I'd like to do some further perf testing, but the results from your
initial tests makes a follow-up ticket seem a reasonable option.
{quote} I've noticed an issue when the client starts protocol negotiation with
an unsupported version. {quote}
Fixed, thanks.
[~ifesdjeen], I haven't pulled in your burn test or changes to {{SimpleClient}}
yet, I'll try to do that next week. I also haven't done any automated renaming
yet, I'll hold off on that so as not to add to the cognitive burden until we're
pretty much done with review.
||branch||CI||
|[15299-trunk|https://github.com/beobal/cassandra/tree/15299-trunk]|[circle|https://app.circleci.com/pipelines/github/beobal/cassandra?branch=15299-trunk]|
> CASSANDRA-13304 follow-up: improve checksumming and compression in protocol
> v5-beta
> ---
>
> Key: CASSANDRA-15299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15299
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Aleksey Yeschenko
>Assignee: Alex Petrov
>Priority: Normal
> Labels: protocolv5
> Fix For: 4.0-alpha
>
>
> CASSANDRA-13304 made an important improvement to our native protocol: it
> introduced checksumming/CRC32 to request and response bodies. It’s an
> important step forward, but it doesn’t cover the entire stream. In
> particular, the message header is not covered by a checksum or a crc, which
> poses a correctness issue if, for example, {{streamId}} gets corrupted.
> Additionally, we aren’t quite using CRC32 correctly, in two ways:
> 1. We are calculating the CRC32 of the *decompressed* value instead of
> computing the
[jira] [Commented] (CASSANDRA-16126) Blog post on Cassandra Usage Report 2020
[ https://issues.apache.org/jira/browse/CASSANDRA-16126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198464#comment-17198464 ] Michael Semb Wever commented on CASSANDRA-16126: Pushed to website after testing from Melissa on cassandra.staged.apache.org > Blog post on Cassandra Usage Report 2020 > > > Key: CASSANDRA-16126 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16126 > Project: Cassandra > Issue Type: Task > Components: Documentation/Blog >Reporter: Melissa Logan >Assignee: Melissa Logan >Priority: Normal > Fix For: 4.0-beta3 > > > Blog post on the 2020 Cassandra Usage Report. > ML: > [https://lists.apache.org/thread.html/r8a91b1d63079739c8dfbdab1833c66d58326de91a659afa3fb2a41a7%40%3Cdev.cassandra.apache.org%3E] > > PR: [https://github.com/apache/cassandra-website/pull/21] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-website] branch asf-site updated (458bdb9 -> cb9bb3e)
This is an automated email from the ASF dual-hosted git repository. mck pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/cassandra-website.git. discard 458bdb9 generate docs for f211daf6 add 9e024b2 Blog Cassandra Usage Report 2020 add 06395ed ninja-fix to 'Blog Cassandra Usage Report 2020' after staging check: fixing blockquotes and images, from Melissa Logan add cb9bb3e generate docs for 06395edd This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (458bdb9) \ N -- N -- N refs/heads/asf-site (cb9bb3e) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../2020/09/17/cassandra-usage-report-2020.html| 251 + content/blog/index.html| 18 +- content/blog/page2/index.html | 9 + content/feed.xml | 100 +++- content/img/blog-post-usage-report-2020/image1.jpg | Bin 0 -> 73535 bytes content/img/blog-post-usage-report-2020/image2.jpg | Bin 0 -> 77037 bytes content/img/blog-post-usage-report-2020/image3.jpg | Bin 0 -> 92250 bytes content/img/blog-post-usage-report-2020/image4.jpg | Bin 0 -> 100861 bytes content/img/blog-post-usage-report-2020/image5.jpg | Bin 0 -> 172291 bytes ...2020-09-17-cassandra-usage-report-2020.markdown | 97 src/img/blog-post-usage-report-2020/image1.jpg | Bin 0 -> 73535 bytes src/img/blog-post-usage-report-2020/image2.jpg | Bin 0 -> 77037 bytes src/img/blog-post-usage-report-2020/image3.jpg | Bin 0 -> 92250 bytes src/img/blog-post-usage-report-2020/image4.jpg | Bin 0 -> 100861 bytes src/img/blog-post-usage-report-2020/image5.jpg | Bin 0 -> 172291 bytes 15 files changed, 464 insertions(+), 11 deletions(-) create mode 100644 content/blog/2020/09/17/cassandra-usage-report-2020.html create mode 100755 content/img/blog-post-usage-report-2020/image1.jpg create mode 100755 content/img/blog-post-usage-report-2020/image2.jpg create mode 100755 content/img/blog-post-usage-report-2020/image3.jpg create mode 100755 content/img/blog-post-usage-report-2020/image4.jpg create mode 100755 content/img/blog-post-usage-report-2020/image5.jpg create mode 100755 src/_posts/2020-09-17-cassandra-usage-report-2020.markdown create mode 100755 src/img/blog-post-usage-report-2020/image1.jpg create mode 100755 src/img/blog-post-usage-report-2020/image2.jpg create mode 100755 src/img/blog-post-usage-report-2020/image3.jpg create mode 100755 src/img/blog-post-usage-report-2020/image4.jpg create mode 100755 src/img/blog-post-usage-report-2020/image5.jpg - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16057) Should update in-jvm dtest to expose stdout and stderr for nodetool
[ https://issues.apache.org/jira/browse/CASSANDRA-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198461#comment-17198461 ] David Capwell commented on CASSANDRA-16057: --- bq. Ultimately, formatted output is good, but structured reporting would be even better if we ever have chance to tacle this. Maybe it's a good point to continue the conversation and create a separate issue for this? +1 to separate issue. bq. We are just exposing there stdout and stderr. Please keep in mind that I need to do this against an instance, meaning I need to be able to do it such way that it will be executed in the "scope" of that node because I can spin up more clusters and I need to have "stdout" per each "jvm / classloader". Yep, this is part of the nodetool interface, which is scoped to the single instance. The implementation also deals with this by updating all nodetool commands to update an interface rather than stdout and stderr directly; this avoids concurrent node tools clobbering the capture. bq. I would love to merge 15406 very soon because it was there literally for months and it was blocked by few other tickets so I am personally little impatient here I don't mind us moving forward with 15406 without this patch; I thought you wanted to hold off given your comment in there. Because both touch the same classes, if 15406 goes first then this patch needs to update the pointer from stdout to the new interface, if we hold off 15406 then that would need to change the pointer. > Should update in-jvm dtest to expose stdout and stderr for nodetool > --- > > Key: CASSANDRA-16057 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16057 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest/java >Reporter: David Capwell >Assignee: Yifan Cai >Priority: Normal > Time Spent: 2h 10m > Remaining Estimate: 0h > > Many nodetool commands output to stdout or stderr so running nodetool using > in-jvm dtest should expose that to tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16127) NullPointerException when calling nodetool enablethrift
[
https://issues.apache.org/jira/browse/CASSANDRA-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198442#comment-17198442
]
David Capwell commented on CASSANDRA-16127:
---
Ok I think the code is almost ready to review. python dtest passes locally and
skips previous versions, all branches have the new tests added and refactored
to have the same lifecycle methods, and thrift/binary operations work in
isolation without affecting each other.
I have updated the CI for each branch to run with the latest python dtest
branch, so will ask for review once green.
> NullPointerException when calling nodetool enablethrift
> ---
>
> Key: CASSANDRA-16127
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16127
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Thrift
>Reporter: Tibor Repasi
>Assignee: David Capwell
>Priority: Normal
> Fix For: 2.2.x, 3.0.x, 3.11.x
>
>
> Having thrift disabled, it's impossible to enable it again without restarting
> the node:
> {code}
> $ nodetool statusthrift
> not running
> $ nodetool enablethrift
> error: null
> -- StackTrace --
> java.lang.NullPointerException
> at
> org.apache.cassandra.service.StorageService.startRPCServer(StorageService.java:392)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15406) Show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Capwell updated CASSANDRA-15406: -- Fix Version/s: (was: 4.x) 4.0-beta > Show the progress of data streaming and index build > > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0-beta > > Time Spent: 3.5h > Remaining Estimate: 0h > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . > > PR [https://github.com/apache/cassandra/pull/711] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15406) Show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Capwell updated CASSANDRA-15406: -- Fix Version/s: (was: 4.0) > Show the progress of data streaming and index build > > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.x > > Time Spent: 3.5h > Remaining Estimate: 0h > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . > > PR [https://github.com/apache/cassandra/pull/711] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15406) Show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198439#comment-17198439 ] David Capwell commented on CASSANDRA-15406: --- patch +1 > Show the progress of data streaming and index build > > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0, 4.x > > Time Spent: 3.5h > Remaining Estimate: 0h > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . > > PR [https://github.com/apache/cassandra/pull/711] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15406) Show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Capwell updated CASSANDRA-15406: -- Status: Ready to Commit (was: Review In Progress) > Show the progress of data streaming and index build > > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0, 4.x > > Time Spent: 3.5h > Remaining Estimate: 0h > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . > > PR [https://github.com/apache/cassandra/pull/711] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15406) Show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198438#comment-17198438 ] David Capwell commented on CASSANDRA-15406: --- up to you but it likely will be a while. There are a few patches pending for jvm-dtest, then we need to release, then update all branches. > Show the progress of data streaming and index build > > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0, 4.x > > Time Spent: 3.5h > Remaining Estimate: 0h > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . > > PR [https://github.com/apache/cassandra/pull/711] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16130) Nodetool support show the progress rate of index rebuild
[ https://issues.apache.org/jira/browse/CASSANDRA-16130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-16130: - Resolution: Fixed Status: Resolved (was: Open) > Nodetool support show the progress rate of index rebuild > > > Key: CASSANDRA-16130 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16130 > Project: Cassandra > Issue Type: New Feature > Components: Feature/2i Index, Tool/nodetool >Reporter: maxwellguo >Priority: Normal > > After Doing 2i index rebuild we may want to seed the progress of the task , > so Adding a nodetool feature to show the progress may be usefull -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16102) Build target for shaded in-JVM dtest jar
[ https://issues.apache.org/jira/browse/CASSANDRA-16102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198336#comment-17198336 ] Marcus Eriksson commented on CASSANDRA-16102: - +1 > Build target for shaded in-JVM dtest jar > > > Key: CASSANDRA-16102 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16102 > Project: Cassandra > Issue Type: Task > Components: Build >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: High > > Several small changes that are required as a prerequisite for releasing > [Harry|https://issues.apache.org/jira/browse/CASSANDRA-15348]: > 1. Update snakeYaml in Cassandra > 2. Add a shade maven target and packaging script for shaded dtest artifacts > This patch is only a bridge for everyone to test out Harry with trunk/4.0, > before we're ready to build in-JVM dtest jars for every version, and stop > depending on java-driver (and, subsequently, on Netty and Guava) in Harry. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15406) Show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198330#comment-17198330 ] Benjamin Lerer commented on CASSANDRA-15406: (y) > Show the progress of data streaming and index build > > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0, 4.x > > Time Spent: 3.5h > Remaining Estimate: 0h > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . > > PR [https://github.com/apache/cassandra/pull/711] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16128) Jenkins: dsl for website build, logging repo SHAs, and using nightlies.a.o instead of archiving
[
https://issues.apache.org/jira/browse/CASSANDRA-16128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198313#comment-17198313
]
Michael Semb Wever commented on CASSANDRA-16128:
bq. Could you doublecheck in your local run in the dtests SHA is somewhere to
be seen? If it is some complicated thing bc of Docker maybe an echo of the SHA
even if it's outside them would be great.
Done.
Example, from a dtest build where SHAs from inside the dtest docker container…
{code}
dtest.sh (dtest 1/1024) cassandra: 68aca10 Adam Holmberg Tue Sep 15 15:56:39
2020 -0500 Remove flaky channelRead_Normal test that wasn't doing much
dtest.sh (dtest 1/1024) cassandra-dtest: 5db813c Yifan Cai Tue Sep 8 14:41:12
2020 -0700 When compaction gets interrupted, the exception should include the
compactionId
dtest.sh (dtest 1/1024) cassandra-builds: 13f0f39 Mick Semb Wever Fri Sep 18
12:33:12 2020 +0200 squash{code}
This output will appear in the dtest runs and in the pipeline run (for each
dtest split).
This required an addition to the patch, see the additional "squash" commit on
the existing {{mck/jenkins--print-git-shas-and-website-build}} branch.
> Jenkins: dsl for website build, logging repo SHAs, and using nightlies.a.o
> instead of archiving
> ---
>
> Key: CASSANDRA-16128
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16128
> Project: Cassandra
> Issue Type: Task
> Components: CI
>Reporter: Michael Semb Wever
>Assignee: Michael Semb Wever
>Priority: Normal
> Fix For: 4.0-beta
>
>
> Jenkins improvements
> 1. Add the cassandra-website job into cassandra_job_dsl.seed.groovy (so we
> don't lose it next time the Jenkins master is corrupted)
> 2. Print the SHAs of the different git repos used during the build process.
> Also store them in the .head files (so the pipeline can print them out too).
> 3. Instead of archiving artefacts, ssh them to
> https://nightlies.apache.org/cassandra/
> (Disk usage on agents is largely under control, but disk usage on master was
> the new problem. The suspicion here is the Cassandra-*-artifact's artefacts
> was the disk usage culprit, though we have to evidence to support it.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15584) 4.0 quality testing: Tooling - External Ecosystem
[
https://issues.apache.org/jira/browse/CASSANDRA-15584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Lerer updated CASSANDRA-15584:
---
Change Category: Quality Assurance
Complexity: Normal
Component/s: Tool/external
Status: Open (was: Triage Needed)
> 4.0 quality testing: Tooling - External Ecosystem
> -
>
> Key: CASSANDRA-15584
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15584
> Project: Cassandra
> Issue Type: Task
> Components: Tool/external
>Reporter: Josh McKenzie
>Assignee: Benjamin Lerer
>Priority: Normal
> Fix For: 4.0-beta
>
>
> Reference [doc from
> NGCC|https://docs.google.com/document/d/1uhUOp7wpE9ZXNDgxoCZHejHt5SO4Qw1dArZqqsJccyQ/edit#]
> for context.
> *Shepherd: Benjamin Lerer*
> Many users of Apache Cassandra employ open source tooling to automate
> Cassandra configuration, runtime management, and repair scheduling. Prior to
> release, we need to confirm that popular third-party tools such function
> properly.
> Current list of tools:
> || Name || Status || Contact ||
> | [Priam|http://netflix.github.io/Priam/] | NOT STARTED |
> [~sumanth.pasupuleti]|
> Columns descriptions:
> * *Name*: Name and link to the tool official page
> * *Status*: {{NOT STARTED}}, {{IN PROGRESS}}, {{BLOCKED}} if you hit any
> issue and have to wait for it to be solved, {{DONE}}, {{AUTOMATIC}} if
> testing 4.0 is part of your CI process.
> * *Contact*: The person acting as the contact point for that tool.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15584) 4.0 quality testing: Tooling - External Ecosystem
[
https://issues.apache.org/jira/browse/CASSANDRA-15584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198289#comment-17198289
]
Benjamin Lerer commented on CASSANDRA-15584:
[~adejanovski], [~stefan.miklosovic], [~tjake], [~jshook] , [~jal06]
Hi,
If I am not mistaken, you or your organisations are involved in maintening
open-source tools that are part of the Cassandra ecosystem. We woud love if you
could ensure that those tools are tested with the 4.0 version before it is
released and report any issue that you might hit. Our goal being to ensure that
all users can have a smooth upgrade path to 4.0.
It will be great if you could add to the ticket description the tools you plan
to tests and their testing status.
If you are not the one directly working on those tool feel free to forward that
request to the appropriate persons.
Thanks in avance for your help
> 4.0 quality testing: Tooling - External Ecosystem
> -
>
> Key: CASSANDRA-15584
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15584
> Project: Cassandra
> Issue Type: Task
>Reporter: Josh McKenzie
>Assignee: Benjamin Lerer
>Priority: Normal
> Fix For: 4.0-beta
>
>
> Reference [doc from
> NGCC|https://docs.google.com/document/d/1uhUOp7wpE9ZXNDgxoCZHejHt5SO4Qw1dArZqqsJccyQ/edit#]
> for context.
> *Shepherd: Benjamin Lerer*
> Many users of Apache Cassandra employ open source tooling to automate
> Cassandra configuration, runtime management, and repair scheduling. Prior to
> release, we need to confirm that popular third-party tools such function
> properly.
> Current list of tools:
> || Name || Status || Contact ||
> | [Priam|http://netflix.github.io/Priam/] | NOT STARTED |
> [~sumanth.pasupuleti]|
> Columns descriptions:
> * *Name*: Name and link to the tool official page
> * *Status*: {{NOT STARTED}}, {{IN PROGRESS}}, {{BLOCKED}} if you hit any
> issue and have to wait for it to be solved, {{DONE}}, {{AUTOMATIC}} if
> testing 4.0 is part of your CI process.
> * *Contact*: The person acting as the contact point for that tool.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15584) 4.0 quality testing: Tooling - External Ecosystem
[
https://issues.apache.org/jira/browse/CASSANDRA-15584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Lerer updated CASSANDRA-15584:
---
Description:
Reference [doc from
NGCC|https://docs.google.com/document/d/1uhUOp7wpE9ZXNDgxoCZHejHt5SO4Qw1dArZqqsJccyQ/edit#]
for context.
*Shepherd: Benjamin Lerer*
Many users of Apache Cassandra employ open source tooling to automate Cassandra
configuration, runtime management, and repair scheduling. Prior to release, we
need to confirm that popular third-party tools such function properly.
Current list of tools:
|| Name || Status || Contact ||
| [Priam|http://netflix.github.io/Priam/] | NOT STARTED |
[~sumanth.pasupuleti]|
Columns descriptions:
* *Name*: Name and link to the tool official page
* *Status*: {{NOT STARTED}}, {{IN PROGRESS}}, {{BLOCKED}} if you hit any issue
and have to wait for it to be solved, {{DONE}}, {{AUTOMATIC}} if testing 4.0 is
part of your CI process.
* *Contact*: The person acting as the contact point for that tool.
was:
Reference [doc from
NGCC|https://docs.google.com/document/d/1uhUOp7wpE9ZXNDgxoCZHejHt5SO4Qw1dArZqqsJccyQ/edit#]
for context.
*Shepherd: Benjamin Lerer*
Many users of Apache Cassandra employ open source tooling to automate Cassandra
configuration, runtime management, and repair scheduling. Prior to release, we
need to confirm that popular third-party tools such function properly.
Current list of tools:
|| Name || Status || Contact ||
| Priam | NOT STARTED | | | |
Columns descriptions:
* *Name*: Name and link to the tool official page
* *Status*: {{NOT STARTED}}, {{IN PROGRESS}}, {{BLOCKED}} if you hit any issue
and have to wait for it to be solved, {{DONE}}, {{AUTOMATIC}} if testing 4.0 is
part of your CI process.
* *Contact*: The person acting as the contact point for that tool.
> 4.0 quality testing: Tooling - External Ecosystem
> -
>
> Key: CASSANDRA-15584
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15584
> Project: Cassandra
> Issue Type: Task
>Reporter: Josh McKenzie
>Assignee: Benjamin Lerer
>Priority: Normal
> Fix For: 4.0-beta
>
>
> Reference [doc from
> NGCC|https://docs.google.com/document/d/1uhUOp7wpE9ZXNDgxoCZHejHt5SO4Qw1dArZqqsJccyQ/edit#]
> for context.
> *Shepherd: Benjamin Lerer*
> Many users of Apache Cassandra employ open source tooling to automate
> Cassandra configuration, runtime management, and repair scheduling. Prior to
> release, we need to confirm that popular third-party tools such function
> properly.
> Current list of tools:
> || Name || Status || Contact ||
> | [Priam|http://netflix.github.io/Priam/] | NOT STARTED |
> [~sumanth.pasupuleti]|
> Columns descriptions:
> * *Name*: Name and link to the tool official page
> * *Status*: {{NOT STARTED}}, {{IN PROGRESS}}, {{BLOCKED}} if you hit any
> issue and have to wait for it to be solved, {{DONE}}, {{AUTOMATIC}} if
> testing 4.0 is part of your CI process.
> * *Contact*: The person acting as the contact point for that tool.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15584) 4.0 quality testing: Tooling - External Ecosystem
[
https://issues.apache.org/jira/browse/CASSANDRA-15584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Lerer updated CASSANDRA-15584:
---
Description:
Reference [doc from
NGCC|https://docs.google.com/document/d/1uhUOp7wpE9ZXNDgxoCZHejHt5SO4Qw1dArZqqsJccyQ/edit#]
for context.
*Shepherd: Benjamin Lerer*
Many users of Apache Cassandra employ open source tooling to automate Cassandra
configuration, runtime management, and repair scheduling. Prior to release, we
need to confirm that popular third-party tools such function properly.
Current list of tools:
|| Name || Status || Contact ||
| Priam | NOT STARTED | | | |
Columns descriptions:
* *Name*: Name and link to the tool official page
* *Status*: {{NOT STARTED}}, {{IN PROGRESS}}, {{BLOCKED}} if you hit any issue
and have to wait for it to be solved, {{DONE}}, {{AUTOMATIC}} if testing 4.0 is
part of your CI process.
* *Contact*: The person acting as the contact point for that tool.
was:
Reference [doc from
NGCC|https://docs.google.com/document/d/1uhUOp7wpE9ZXNDgxoCZHejHt5SO4Qw1dArZqqsJccyQ/edit#]
for context.
*Shepherd: Sam Tunnicliffe*
Many users of Apache Cassandra employ open source tooling to automate Cassandra
configuration, runtime management, and repair scheduling. Prior to release, we
need to confirm that popular third-party tools such function properly.
Current list of tools:
|| Name || Tested || Test date || Tested via CI || Contact ||
| Priam |
> 4.0 quality testing: Tooling - External Ecosystem
> -
>
> Key: CASSANDRA-15584
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15584
> Project: Cassandra
> Issue Type: Task
>Reporter: Josh McKenzie
>Assignee: Benjamin Lerer
>Priority: Normal
> Fix For: 4.0-beta
>
>
> Reference [doc from
> NGCC|https://docs.google.com/document/d/1uhUOp7wpE9ZXNDgxoCZHejHt5SO4Qw1dArZqqsJccyQ/edit#]
> for context.
> *Shepherd: Benjamin Lerer*
> Many users of Apache Cassandra employ open source tooling to automate
> Cassandra configuration, runtime management, and repair scheduling. Prior to
> release, we need to confirm that popular third-party tools such function
> properly.
> Current list of tools:
> || Name || Status || Contact ||
> | Priam | NOT STARTED | | | |
> Columns descriptions:
> * *Name*: Name and link to the tool official page
> * *Status*: {{NOT STARTED}}, {{IN PROGRESS}}, {{BLOCKED}} if you hit any
> issue and have to wait for it to be solved, {{DONE}}, {{AUTOMATIC}} if
> testing 4.0 is part of your CI process.
> * *Contact*: The person acting as the contact point for that tool.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15584) 4.0 quality testing: Tooling - External Ecosystem
[ https://issues.apache.org/jira/browse/CASSANDRA-15584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Lerer updated CASSANDRA-15584: --- Description: Reference [doc from NGCC|https://docs.google.com/document/d/1uhUOp7wpE9ZXNDgxoCZHejHt5SO4Qw1dArZqqsJccyQ/edit#] for context. *Shepherd: Sam Tunnicliffe* Many users of Apache Cassandra employ open source tooling to automate Cassandra configuration, runtime management, and repair scheduling. Prior to release, we need to confirm that popular third-party tools such function properly. Current list of tools: || Name || Tested || Test date || Tested via CI || Contact || | Priam | was: Reference [doc from NGCC|https://docs.google.com/document/d/1uhUOp7wpE9ZXNDgxoCZHejHt5SO4Qw1dArZqqsJccyQ/edit#] for context. *Shepherd: Sam Tunnicliffe* Many users of Apache Cassandra employ open source tooling to automate Cassandra configuration, runtime management, and repair scheduling. Prior to release, we need to confirm that popular third-party tools such as Reaper, Priam, etc. function properly. > 4.0 quality testing: Tooling - External Ecosystem > - > > Key: CASSANDRA-15584 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15584 > Project: Cassandra > Issue Type: Task >Reporter: Josh McKenzie >Assignee: Benjamin Lerer >Priority: Normal > Fix For: 4.0-beta > > > Reference [doc from > NGCC|https://docs.google.com/document/d/1uhUOp7wpE9ZXNDgxoCZHejHt5SO4Qw1dArZqqsJccyQ/edit#] > for context. > *Shepherd: Sam Tunnicliffe* > Many users of Apache Cassandra employ open source tooling to automate > Cassandra configuration, runtime management, and repair scheduling. Prior to > release, we need to confirm that popular third-party tools such function > properly. > Current list of tools: > || Name || Tested || Test date || Tested via CI || Contact || > | Priam | -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15584) 4.0 quality testing: Tooling - External Ecosystem
[ https://issues.apache.org/jira/browse/CASSANDRA-15584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198260#comment-17198260 ] Benjamin Lerer commented on CASSANDRA-15584: My plan here is to coordinate with the different groups that have developed open source tooling arround Cassandra to ensure that they had the chance to test their tool with C* 4.0 before we go GA and report any issues that they might hit. > 4.0 quality testing: Tooling - External Ecosystem > - > > Key: CASSANDRA-15584 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15584 > Project: Cassandra > Issue Type: Task >Reporter: Josh McKenzie >Assignee: Benjamin Lerer >Priority: Normal > Fix For: 4.0-beta > > > Reference [doc from > NGCC|https://docs.google.com/document/d/1uhUOp7wpE9ZXNDgxoCZHejHt5SO4Qw1dArZqqsJccyQ/edit#] > for context. > *Shepherd: Sam Tunnicliffe* > Many users of Apache Cassandra employ open source tooling to automate > Cassandra configuration, runtime management, and repair scheduling. Prior to > release, we need to confirm that popular third-party tools such as Reaper, > Priam, etc. function properly. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15584) 4.0 quality testing: Tooling - External Ecosystem
[ https://issues.apache.org/jira/browse/CASSANDRA-15584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Lerer reassigned CASSANDRA-15584: -- Assignee: Benjamin Lerer > 4.0 quality testing: Tooling - External Ecosystem > - > > Key: CASSANDRA-15584 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15584 > Project: Cassandra > Issue Type: Task >Reporter: Josh McKenzie >Assignee: Benjamin Lerer >Priority: Normal > Fix For: 4.0-beta > > > Reference [doc from > NGCC|https://docs.google.com/document/d/1uhUOp7wpE9ZXNDgxoCZHejHt5SO4Qw1dArZqqsJccyQ/edit#] > for context. > *Shepherd: Sam Tunnicliffe* > Many users of Apache Cassandra employ open source tooling to automate > Cassandra configuration, runtime management, and repair scheduling. Prior to > release, we need to confirm that popular third-party tools such as Reaper, > Priam, etc. function properly. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15406) Show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198248#comment-17198248 ] Stefan Miklosovic commented on CASSANDRA-15406: --- Thanks [~blerer], I think we need to wait a bit before this hits trunk - there are some bits here which are covering what I do more properly - https://issues.apache.org/jira/browse/CASSANDRA-16057 > Show the progress of data streaming and index build > > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0, 4.x > > Time Spent: 3.5h > Remaining Estimate: 0h > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . > > PR [https://github.com/apache/cassandra/pull/711] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15406) Show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198246#comment-17198246 ] Benjamin Lerer commented on CASSANDRA-15406: The patch looks good to me. > Show the progress of data streaming and index build > > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0, 4.x > > Time Spent: 3.5h > Remaining Estimate: 0h > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . > > PR [https://github.com/apache/cassandra/pull/711] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15214) OOMs caught and not rethrown
[ https://issues.apache.org/jira/browse/CASSANDRA-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198233#comment-17198233 ] Benedict Elliott Smith commented on CASSANDRA-15214: Zoom out a bit - where would that exception end up if it were rethrown? I can't remember precisely, but it is caught by Netty's default exception handling and iirc simply logged. > OOMs caught and not rethrown > > > Key: CASSANDRA-15214 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15214 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client, Messaging/Internode >Reporter: Benedict Elliott Smith >Assignee: Yifan Cai >Priority: Normal > Fix For: 4.0, 4.0-rc > > Attachments: oom-experiments.zip > > > Netty (at least, and perhaps elsewhere in Executors) catches all exceptions, > so presently there is no way to ensure that an OOM reaches the JVM handler to > trigger a crash/heapdump. > It may be that the simplest most consistent way to do this would be to have a > single thread spawned at startup that waits for any exceptions we must > propagate to the Runtime. > We could probably submit a patch upstream to Netty, but for a guaranteed > future proof approach, it may be worth paying the cost of a single thread. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-website] branch asf-staging updated (70eaf55 -> cb9bb3e)
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a change to branch asf-staging in repository https://gitbox.apache.org/repos/asf/cassandra-website.git. discard 70eaf55 generate docs for 9e024b24 add 06395ed ninja-fix to 'Blog Cassandra Usage Report 2020' after staging check: fixing blockquotes and images, from Melissa Logan new cb9bb3e generate docs for 06395edd This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (70eaf55) \ N -- N -- N refs/heads/asf-staging (cb9bb3e) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../2020/09/17/cassandra-usage-report-2020.html| 36 +-- content/feed.xml | 38 ++--- content/img/blog-post-usage-report-2020/image1.jpg | Bin 77307 -> 73535 bytes content/img/blog-post-usage-report-2020/image4.jpg | Bin 105206 -> 100861 bytes content/img/blog-post-usage-report-2020/image5.jpg | Bin 176257 -> 172291 bytes ...2020-09-17-cassandra-usage-report-2020.markdown | 13 --- src/img/blog-post-usage-report-2020/image1.jpg | Bin 77307 -> 73535 bytes src/img/blog-post-usage-report-2020/image4.jpg | Bin 105206 -> 100861 bytes src/img/blog-post-usage-report-2020/image5.jpg | Bin 176257 -> 172291 bytes 9 files changed, 43 insertions(+), 44 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-builds] branch master updated: Add root volume space minimum requirement for jenkins agents
This is an automated email from the ASF dual-hosted git repository. mck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/cassandra-builds.git The following commit(s) were added to refs/heads/master by this push: new ac53371 Add root volume space minimum requirement for jenkins agents ac53371 is described below commit ac533710606db332389abf08b5a55d327d319c7f Author: mck AuthorDate: Fri Sep 18 10:56:18 2020 +0200 Add root volume space minimum requirement for jenkins agents --- ASF-jenkins-agents.txt | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/ASF-jenkins-agents.txt b/ASF-jenkins-agents.txt index 1092297..8d91bb1 100644 --- a/ASF-jenkins-agents.txt +++ b/ASF-jenkins-agents.txt @@ -23,7 +23,9 @@ Edit sudoers to allow INFRA to sudo without password, or provide them with the p For example, set: %sudo ALL=(ALL:ALL) NOPASSWD:ALL $ sudo visudo -Ensure the additional (500GB) data volume is mounted (and added to `/etc/fstab` at `/home/jenkins` +Ensure the additional (500GB) data volume is mounted (and added to `/etc/fstab` at `/home/jenkins`. +Ensure the root volume is 100GB+. +Ensure `hostname` is configured to the server's public ip. Add jenkins user: $ sudo groupadd -g 910 jenkins @@ -53,7 +55,6 @@ Install and Configure Puppet: EOF -Ensure `hostname` is configured to the server's public ip. - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16126) Blog post on Cassandra Usage Report 2020
[ https://issues.apache.org/jira/browse/CASSANDRA-16126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198224#comment-17198224 ] Michael Semb Wever commented on CASSANDRA-16126: Ninja-fix (for quotes and images) applied, after the staged review, applied. ref: https://github.com/apache/cassandra-website/pull/22#issuecomment-694743749 > Blog post on Cassandra Usage Report 2020 > > > Key: CASSANDRA-16126 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16126 > Project: Cassandra > Issue Type: Task > Components: Documentation/Blog >Reporter: Melissa Logan >Assignee: Melissa Logan >Priority: Normal > Fix For: 4.0-beta3 > > > Blog post on the 2020 Cassandra Usage Report. > ML: > [https://lists.apache.org/thread.html/r8a91b1d63079739c8dfbdab1833c66d58326de91a659afa3fb2a41a7%40%3Cdev.cassandra.apache.org%3E] > > PR: [https://github.com/apache/cassandra-website/pull/21] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-website] branch master updated: ninja-fix to 'Blog Cassandra Usage Report 2020' after staging check: fixing blockquotes and images, from Melissa Logan
This is an automated email from the ASF dual-hosted git repository. mck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/cassandra-website.git The following commit(s) were added to refs/heads/master by this push: new 06395ed ninja-fix to 'Blog Cassandra Usage Report 2020' after staging check: fixing blockquotes and images, from Melissa Logan 06395ed is described below commit 06395edd1f9b7fb82ac30f3c1b11a3908df81c4f Author: mck AuthorDate: Fri Sep 18 10:48:32 2020 +0200 ninja-fix to 'Blog Cassandra Usage Report 2020' after staging check: fixing blockquotes and images, from Melissa Logan --- ...2020-09-17-cassandra-usage-report-2020.markdown | 13 ++--- src/img/blog-post-usage-report-2020/image1.jpg | Bin 77307 -> 73535 bytes src/img/blog-post-usage-report-2020/image4.jpg | Bin 105206 -> 100861 bytes src/img/blog-post-usage-report-2020/image5.jpg | Bin 176257 -> 172291 bytes 4 files changed, 6 insertions(+), 7 deletions(-) diff --git a/src/_posts/2020-09-17-cassandra-usage-report-2020.markdown b/src/_posts/2020-09-17-cassandra-usage-report-2020.markdown index 0f4a337..6245aef 100644 --- a/src/_posts/2020-09-17-cassandra-usage-report-2020.markdown +++ b/src/_posts/2020-09-17-cassandra-usage-report-2020.markdown @@ -9,7 +9,7 @@ categories: blog Apache Cassandra is the open source NoSQL database for mission critical data. Today the community announced findings from a comprehensive global survey of 901 practitioners on Cassandra usage. It’s the first of what will become an annual survey that provides a baseline understanding of who, how, and why organizations use Cassandra. -"I saw zero downtime at global scale with Apache Cassandra. That's a powerful statement to make. For our business that’s quite crucial." - Practitioner, London +> "I saw zero downtime at global scale with Apache Cassandra. That's a powerful statement to make. For our business that’s quite crucial." - Practitioner, London ### Key Themes @@ -60,7 +60,7 @@ People from organizations that self-identified as being in a “highly advanced Cassandra predominates in very important or mission critical apps. Among practitioners, 31% use Cassandra for their mission critical applications, 55% for their very important applications, 38% for their somewhat important applications, and 20% for their least important applications. -"We're scheduling 100s of millions of messages to be sent. Per day. If it's two weeks, we're talking about a couple billion. So for this, we use Cassandra." - Practitioner, Amsterdam +> "We're scheduling 100s of millions of messages to be sent. Per day. If it's two weeks, we're talking about a couple billion. So for this, we use Cassandra." - Practitioner, Amsterdam  @@ -69,23 +69,22 @@ Cassandra predominates in very important or mission critical apps. Among practit The top reasons practitioners use Cassandra for mission critical apps are “good hybrid solutions” (62%), “very secure” (60%), “highly scalable” (57%), “fast” (57%), and “easy to build apps with” (55%). -“High traffic, high data environments where really you're just looking for very simplistic key value persistence of your data. It's going to be a great fit for you, I can promise that.” - Global SVP Engineering +> “High traffic, high data environments where really you're just looking for very simplistic key value persistence of your data. It's going to be a great fit for you, I can promise that.” - Global SVP Engineering  For companies in a highly advanced stage of digital transformation, 58% cite “won’t lose data” as the top reason, followed by “gives me confidence” (56%), “cloud native” (56%), and “very secure” (56%). -"It can’t lose anything, it has to be able to capture everything. It can’t have any security defects. It needs to be somewhat compatible with the environment. If we adopt a new database, it can’t be a duplicate of the data we already have.… So: Cassandra." - Practitioner, San Francisco +> "It can’t lose anything, it has to be able to capture everything. It can’t have any security defects. It needs to be somewhat compatible with the environment. If we adopt a new database, it can’t be a duplicate of the data we already have.… So: Cassandra." - Practitioner, San Francisco However, 36% of practitioners currently using Cassandra for mission critical apps say that a lack of Cassandra-skilled team members may deter adoption. -“We don’t have time to train a ton of developers, so that time to deploy, time to onboard, that's really key. All the other stuff, scalability, that all sounds fine.” – Practitioner, London +> “We don’t have time to train a ton of developers, so that time to deploy, time to onboard,
[cassandra-in-jvm-dtest-api] branch master updated: Add method to ignore uncaught throwables (#18)
This is an automated email from the ASF dual-hosted git repository.
marcuse pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-in-jvm-dtest-api.git
The following commit(s) were added to refs/heads/master by this push:
new 15d9595 Add method to ignore uncaught throwables (#18)
15d9595 is described below
commit 15d959578258d053eb66ae94a43f299e1f099814
Author: Marcus Eriksson
AuthorDate: Fri Sep 18 09:19:13 2020 +0200
Add method to ignore uncaught throwables (#18)
Patch by marcuse; reviewed by David Capwell for CASSANDRA-16101
---
.../org/apache/cassandra/distributed/api/ICluster.java | 14 ++
.../org/apache/cassandra/distributed/api/IInstance.java| 1 -
2 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/src/main/java/org/apache/cassandra/distributed/api/ICluster.java
b/src/main/java/org/apache/cassandra/distributed/api/ICluster.java
index a586311..3fc056e 100644
--- a/src/main/java/org/apache/cassandra/distributed/api/ICluster.java
+++ b/src/main/java/org/apache/cassandra/distributed/api/ICluster.java
@@ -24,6 +24,8 @@ import java.net.InetSocketAddress;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
+import java.util.function.BiPredicate;
+import java.util.function.Predicate;
import java.util.stream.Stream;
public interface ICluster extends AutoCloseable
@@ -54,6 +56,18 @@ public interface ICluster extends
AutoCloseable
IMessageFilters filters();
+/**
+ * dynamically sets the current uncaught exceptions filter
+ *
+ * the predicate should return true if we should ignore the given
throwable on the given instance
+ */
+default void setUncaughtExceptionsFilter(BiPredicate
ignoreThrowable) {}
+default void setUncaughtExceptionsFilter(Predicate
ignoreThrowable)
+{
+setUncaughtExceptionsFilter((ignored, throwable) ->
ignoreThrowable.test(throwable));
+}
+default void checkAndResetUncaughtExceptions() {}
+
static void setup() throws Throwable
{
setupLogging();
diff --git a/src/main/java/org/apache/cassandra/distributed/api/IInstance.java
b/src/main/java/org/apache/cassandra/distributed/api/IInstance.java
index 5db0bcf..496d33d 100644
--- a/src/main/java/org/apache/cassandra/distributed/api/IInstance.java
+++ b/src/main/java/org/apache/cassandra/distributed/api/IInstance.java
@@ -90,5 +90,4 @@ public interface IInstance extends IIsolatedExecutor
void forceCompact(String keyspace, String table);
-List getUncaughtExceptions();
}
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16057) Should update in-jvm dtest to expose stdout and stderr for nodetool
[ https://issues.apache.org/jira/browse/CASSANDRA-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198180#comment-17198180 ] Stefan Miklosovic commented on CASSANDRA-16057: --- Ah, looking at Netstats I have _yet again_ some work to align this with the changes in this PR when that one comes through. I start to believe 15406 is just not meant to be ever merged or something :D > Should update in-jvm dtest to expose stdout and stderr for nodetool > --- > > Key: CASSANDRA-16057 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16057 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest/java >Reporter: David Capwell >Assignee: Yifan Cai >Priority: Normal > Time Spent: 2h 10m > Remaining Estimate: 0h > > Many nodetool commands output to stdout or stderr so running nodetool using > in-jvm dtest should expose that to tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16036) Add flag to disable chunk cache and disable by default
[
https://issues.apache.org/jira/browse/CASSANDRA-16036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198173#comment-17198173
]
ZhaoYang edited comment on CASSANDRA-16036 at 9/18/20, 7:16 AM:
!16036_128mb.png!
Above is write perf in mixed-read-write test using 128mb cache between
16036-disable-chunk-cache and its base line. Disabling chunk cache
significantly improves latency. (read perf is similar to write perf)
!15229_128mb.png!
Above is write perf in mixed-read-write test using 128mb cache between
15229-disable-chunk-cache and 15229-improved-buffer-pool. Disabling chunk cache
show some improvement on latency. (read perf is similar to write perf)
was (Author: jasonstack):
!16036_128mb.png!
Above is write perf in mixed-read-write test between 16036-disable-chunk-cache
and its base line. Disabling chunk cache significantly improves latency. (read
perf is similar to write perf)
!15229_128mb.png!
Above is write perf in mixed-read-write test between 15229-disable-chunk-cache
and 15229-improved-buffer-pool. Disabling chunk cache show some improvement on
latency. (read perf is similar to write perf)
> Add flag to disable chunk cache and disable by default
> --
>
> Key: CASSANDRA-16036
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16036
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: 15229_128mb.png, 16036_128mb.png,
> async-profile.collapsed.svg,
> clustering-in-clause_latency_selects_baseline.png,
> clustering-in-clause_latency_selects_baseline_attempt3.png,
> clustering-in-clause_latency_under90_selects_baseline.png,
> clustering-in-clause_latency_under90_selects_baseline_attempt3.png,
> clustering-slice_latency_selects_baseline.png,
> clustering-slice_latency_under90_selects_baseline.png,
> medium-blobs_latency_selects_baseline.png,
> medium-blobs_latency_under90_selects_baseline.png,
> partition-single-row-read_latency_selects_baseline.png,
> partition-single-row-read_latency_under90_selects_baseline.png
>
>
> Chunk cache is enabled by default and doesn’t have a flag to disable without
> impacting networking. In performance testing 4.0 against 3.0 I found that
> reads were slower in 4.0 and after profiling found that the ChunkCache was
> partially to blame; after disabling the chunk cache, read performance had
> improved.
> {code}
> 40_w_cc-selects.hdr
> #[Mean= 11.50063, StdDeviation = 13.44014]
> #[Max =482.41254, Total count= 316477]
> #[Buckets = 25, SubBuckets = 262144]
> 40_wo_cc-selects.hdr
> #[Mean= 9.82115, StdDeviation = 10.14270]
> #[Max =522.36493, Total count= 317444]
> #[Buckets = 25, SubBuckets = 262144]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16036) Add flag to disable chunk cache and disable by default
[
https://issues.apache.org/jira/browse/CASSANDRA-16036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198175#comment-17198175
]
ZhaoYang commented on CASSANDRA-16036:
--
+1 to disable chunk cache until we get CASSANDRA-15229 and other improvements
(eg. fixed buffer size) into chunk cache.
> Add flag to disable chunk cache and disable by default
> --
>
> Key: CASSANDRA-16036
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16036
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: 15229_128mb.png, 16036_128mb.png,
> async-profile.collapsed.svg,
> clustering-in-clause_latency_selects_baseline.png,
> clustering-in-clause_latency_selects_baseline_attempt3.png,
> clustering-in-clause_latency_under90_selects_baseline.png,
> clustering-in-clause_latency_under90_selects_baseline_attempt3.png,
> clustering-slice_latency_selects_baseline.png,
> clustering-slice_latency_under90_selects_baseline.png,
> medium-blobs_latency_selects_baseline.png,
> medium-blobs_latency_under90_selects_baseline.png,
> partition-single-row-read_latency_selects_baseline.png,
> partition-single-row-read_latency_under90_selects_baseline.png
>
>
> Chunk cache is enabled by default and doesn’t have a flag to disable without
> impacting networking. In performance testing 4.0 against 3.0 I found that
> reads were slower in 4.0 and after profiling found that the ChunkCache was
> partially to blame; after disabling the chunk cache, read performance had
> improved.
> {code}
> 40_w_cc-selects.hdr
> #[Mean= 11.50063, StdDeviation = 13.44014]
> #[Max =482.41254, Total count= 316477]
> #[Buckets = 25, SubBuckets = 262144]
> 40_wo_cc-selects.hdr
> #[Mean= 9.82115, StdDeviation = 10.14270]
> #[Max =522.36493, Total count= 317444]
> #[Buckets = 25, SubBuckets = 262144]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16036) Add flag to disable chunk cache and disable by default
[
https://issues.apache.org/jira/browse/CASSANDRA-16036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198173#comment-17198173
]
ZhaoYang commented on CASSANDRA-16036:
--
!16036_128mb.png!
Above is write perf in mixed-read-write test between 16036-disable-chunk-cache
and its base line. Disabling chunk cache significantly improves latency. (read
perf is similar to write perf)
!15229_128mb.png!
Above is write perf in mixed-read-write test between 15229-disable-chunk-cache
and 15229-improved-buffer-pool. Disabling chunk cache show some improvement on
latency. (read perf is similar to write perf)
> Add flag to disable chunk cache and disable by default
> --
>
> Key: CASSANDRA-16036
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16036
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: 15229_128mb.png, 16036_128mb.png,
> async-profile.collapsed.svg,
> clustering-in-clause_latency_selects_baseline.png,
> clustering-in-clause_latency_selects_baseline_attempt3.png,
> clustering-in-clause_latency_under90_selects_baseline.png,
> clustering-in-clause_latency_under90_selects_baseline_attempt3.png,
> clustering-slice_latency_selects_baseline.png,
> clustering-slice_latency_under90_selects_baseline.png,
> medium-blobs_latency_selects_baseline.png,
> medium-blobs_latency_under90_selects_baseline.png,
> partition-single-row-read_latency_selects_baseline.png,
> partition-single-row-read_latency_under90_selects_baseline.png
>
>
> Chunk cache is enabled by default and doesn’t have a flag to disable without
> impacting networking. In performance testing 4.0 against 3.0 I found that
> reads were slower in 4.0 and after profiling found that the ChunkCache was
> partially to blame; after disabling the chunk cache, read performance had
> improved.
> {code}
> 40_w_cc-selects.hdr
> #[Mean= 11.50063, StdDeviation = 13.44014]
> #[Max =482.41254, Total count= 316477]
> #[Buckets = 25, SubBuckets = 262144]
> 40_wo_cc-selects.hdr
> #[Mean= 9.82115, StdDeviation = 10.14270]
> #[Max =522.36493, Total count= 317444]
> #[Buckets = 25, SubBuckets = 262144]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16057) Should update in-jvm dtest to expose stdout and stderr for nodetool
[ https://issues.apache.org/jira/browse/CASSANDRA-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198157#comment-17198157 ] Stefan Miklosovic commented on CASSANDRA-16057: --- [~ifesdjeen] [~yifanc] I have implemented this here https://github.com/apache/cassandra/pull/711/files If you go over that PR you will see what I mean. Maybe that one might go in instead? Anyway, I like the idea this would be directly in dtest api, we had an intention to add it there _sometimes_ as I do not have time to do that so if we do it _the proper way_ and I can reuse it in my PR, that would be awesome. We are just exposing there stdout and stderr. Please keep in mind that I need to do this _against an instance_, meaning I need to be able to do it such way that it will be executed in the "scope" of that node because I can spin up more clusters and I need to have "stdout" per each "jvm / classloader". If all this is met, we might reuse whatever solution but please keep in mind that I would love to merge 15406 very soon because it was there literally for months and it was blocked by few other tickets so I am personally little impatient here :) Looking forward to have a feedback. > Should update in-jvm dtest to expose stdout and stderr for nodetool > --- > > Key: CASSANDRA-16057 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16057 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest/java >Reporter: David Capwell >Assignee: Yifan Cai >Priority: Normal > Time Spent: 2h 10m > Remaining Estimate: 0h > > Many nodetool commands output to stdout or stderr so running nodetool using > in-jvm dtest should expose that to tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15386) Use multiple data directories in the in-jvm dtests
[ https://issues.apache.org/jira/browse/CASSANDRA-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198140#comment-17198140 ] Alex Petrov commented on CASSANDRA-15386: - +1 > Use multiple data directories in the in-jvm dtests > -- > > Key: CASSANDRA-15386 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15386 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction, Test/dtest/java >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Labels: pull-request-available > Fix For: 3.11.x, 4.x > > > We should default to using 3 data directories when running the in-jvm dtests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16120) Add ability for jvm-dtest to grep instance logs
[
https://issues.apache.org/jira/browse/CASSANDRA-16120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov updated CASSANDRA-16120:
Status: Ready to Commit (was: Review In Progress)
> Add ability for jvm-dtest to grep instance logs
> ---
>
> Key: CASSANDRA-16120
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16120
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/dtest/java
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-beta
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> One of the main gaps between python dtest and jvm dtest is python dtest
> supports the ability to grep the logs of an instance; we need this capability
> as some tests require validating logs were triggered.
> Pydocs for common log methods
> {code}
> | grep_log(self, expr, filename='system.log', from_mark=None)
> | Returns a list of lines matching the regular expression in parameter
> | in the Cassandra log of this node
> |
> | grep_log_for_errors(self, filename='system.log')
> | Returns a list of errors with stack traces
> | in the Cassandra log of this node
> |
> | grep_log_for_errors_from(self, filename='system.log', seek_start=0)
> {code}
> {code}
> | watch_log_for(self, exprs, from_mark=None, timeout=600, process=None,
> verbose=False, filename='system.log')
> | Watch the log until one or more (regular) expression are found.
> | This methods when all the expressions have been found or the method
> | timeouts (a TimeoutError is then raised). On successful completion,
> | a list of pair (line matched, match object) is returned.
> {code}
> Below is a POC showing a way to do such logic
> {code}
> package org.apache.cassandra.distributed.test;
> import java.io.BufferedReader;
> import java.io.FileInputStream;
> import java.io.IOException;
> import java.io.InputStreamReader;
> import java.io.UncheckedIOException;
> import java.nio.charset.StandardCharsets;
> import java.util.Iterator;
> import java.util.Spliterator;
> import java.util.Spliterators;
> import java.util.regex.Matcher;
> import java.util.regex.Pattern;
> import java.util.stream.Stream;
> import java.util.stream.StreamSupport;
> import com.google.common.io.Closeables;
> import org.junit.Test;
> import org.apache.cassandra.distributed.Cluster;
> import org.apache.cassandra.utils.AbstractIterator;
> public class AllTheLogs extends TestBaseImpl
> {
>@Test
>public void test() throws IOException
>{
>try (final Cluster cluster = init(Cluster.build(1).start()))
>{
>String tag = System.getProperty("cassandra.testtag",
> "cassandra.testtag_IS_UNDEFINED");
>String suite = System.getProperty("suitename",
> "suitename_IS_UNDEFINED");
>String log = String.format("build/test/logs/%s/TEST-%s.log", tag,
> suite);
>grep(log, "Enqueuing flush of tables").forEach(l ->
> System.out.println("I found the thing: " + l));
>}
>}
>private static Stream grep(String file, String regex) throws
> IOException
>{
>return grep(file, Pattern.compile(regex));
>}
>private static Stream grep(String file, Pattern regex) throws
> IOException
>{
>BufferedReader reader = new BufferedReader(new InputStreamReader(new
> FileInputStream(file), StandardCharsets.UTF_8));
>Iterator it = new AbstractIterator()
>{
>protected String computeNext()
>{
>try
>{
>String s;
>while ((s = reader.readLine()) != null)
>{
>Matcher m = regex.matcher(s);
>if (m.find())
>return s;
>}
>reader.close();
>return endOfData();
>}
>catch (IOException e)
>{
>Closeables.closeQuietly(reader);
>throw new UncheckedIOException(e);
>}
>}
>};
>return StreamSupport.stream(Spliterators.spliteratorUnknownSize(it,
> Spliterator.ORDERED), false);
>}
> }
> {code}
> And
> {code}
> @Test
>public void test() throws IOException
>{
>try (final Cluster cluster = init(Cluster.build(1).start()))
>{
>String tag = System.getProperty("cassandra.testtag",
> "cassandra.testtag_IS_UNDEFINED");
>String suite = System.getProperty("suitename",
> "suitename_IS_UNDEFINED");
>//TODO missing way to get node id
> //cluster.get(1);
>String log =
>
