[jira] [Commented] (CASSANDRA-16176) Python dtests should run at debug

[Berenguer Blasi (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17229032#comment-17229032
 ] 

Berenguer Blasi commented on CASSANDRA-16176:
-

TRACE is indeed heavier so I am happy to leave it at DEBUG and we can change it 
later if needed. I have pushed and ran CI for low, mid and high res. The 3 
display DEBUG for the nodes logs, ccm and the py test itself. Look for 
{{test_split}} in the CI run {{Artifacts}} tab. Then look inside the 
{{stdout.txt}} for that executor for {{Run stress to ensure data is readable}} 
to locate a py test DEBUG message:
- HIGH: 
[j11|https://app.circleci.com/pipelines/github/bereng/cassandra/179/workflows/61825d09-50b5-475f-b8af-779381a741c8]
 
[j8|https://app.circleci.com/pipelines/github/bereng/cassandra/179/workflows/9bf31e85-9b86-43da-82a7-130af992cc1d]
- MID: 
[j11|https://app.circleci.com/pipelines/github/bereng/cassandra/178/workflows/5791ee2e-a23a-4474-b2e2-0a12f3eafe36]
 
[j8|https://app.circleci.com/pipelines/github/bereng/cassandra/178/workflows/7a5e5dce-ce05-49c8-8bb1-5ff23b839666]
- LOW: 
[j11|https://app.circleci.com/pipelines/github/bereng/cassandra/177/workflows/663735e6-0acf-455c-8b85-545bbd1d9ea7]
 
[j8|https://app.circleci.com/pipelines/github/bereng/cassandra/177/workflows/34090602-f24c-4b10-9066-51a3f6090f0f]

If this is what you needed it is ready to merge imo.

> Python dtests should run at debug
> -
>
> Key: CASSANDRA-16176
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16176
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CI
>Reporter: Brandon Williams
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> At least in our circle configs, we are specifying --log-level="INFO" which 
> often leaves us with nothing to go on when we have a flaky test that only 
> fails in a certain environment.  In general it would be more useful to always 
> run at DEBUG. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16259) tablehistograms cause ArrayIndexOutOfBoundsException

[Erick Ramirez (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228993#comment-17228993
 ] 

Erick Ramirez commented on CASSANDRA-16259:
---

Noting here that this came out of a [discussion on ASF Slack in the #cassandra 
channel|https://the-asf.slack.com/archives/CJZLTM05A/p1604884605220500].

> tablehistograms cause ArrayIndexOutOfBoundsException
> 
>
> Key: CASSANDRA-16259
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16259
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Justin
>Priority: Normal
>
> After upgrading some nodes in the cluster from 3.11.8 to 3.11.9 a new error 
> appeared on the upgraded nodes when trying to access _*tablehistograms*_. I 
> confirmed running the same commands on my non-upgraded .8 nodes returns as 
> expected, only the upgraded .9 nodes fail. Not all tables fail when queried, 
> but about 90% of them do.
> I use Datastax MCAC which must query histograms every 30 seconds, this 
> outputs to the system.log:
> {noformat}
> WARN  [insights-3-1] 2020-11-09 01:11:22,331 UnixSocketClient.java:830 - 
> Error reporting:
> java.lang.ArrayIndexOutOfBoundsException: 115
> at 
> org.apache.cassandra.metrics.TableMetrics.combineHistograms(TableMetrics.java:261)
>  ~[apache-cassandra-3.11.9.jar:3.11.9]
> at 
> org.apache.cassandra.metrics.TableMetrics.access$000(TableMetrics.java:48) 
> ~[apache-cassandra-3.11.9.jar:3.11.9]
> at 
> org.apache.cassandra.metrics.TableMetrics$11.getValue(TableMetrics.java:376) 
> ~[apache-cassandra-3.11.9.jar:3.11.9]
> at 
> org.apache.cassandra.metrics.TableMetrics$11.getValue(TableMetrics.java:373) 
> ~[apache-cassandra-3.11.9.jar:3.11.9]
> at 
> com.datastax.mcac.UnixSocketClient.writeMetric(UnixSocketClient.java:839) 
> [datastax-mcac-agent.jar:na]
> at 
> com.datastax.mcac.UnixSocketClient.access$700(UnixSocketClient.java:78) 
> [datastax-mcac-agent.jar:na]
> at 
> com.datastax.mcac.UnixSocketClient$2.lambda$onGaugeAdded$0(UnixSocketClient.java:626)
>  ~[datastax-mcac-agent.jar:na]
> at 
> com.datastax.mcac.UnixSocketClient.writeGroup(UnixSocketClient.java:819) 
> [datastax-mcac-agent.jar:na]
> at 
> com.datastax.mcac.UnixSocketClient.lambda$restartMetricReporting$2(UnixSocketClient.java:798)
>  [datastax-mcac-agent.jar:na]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_272]
> at 
> io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:126)
>  ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
>  ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:307) 
> ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
>  ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at 
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
>  ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_272]{noformat}
> Manually trying a histogram from the CLI:
> {noformat}
> nodetool tablehistograms logdata log_height_index
> error: 115
> -- StackTrace --
> java.lang.ArrayIndexOutOfBoundsException: 115
>   at 
> org.apache.cassandra.metrics.TableMetrics.combineHistograms(TableMetrics.java:261)
>   at 
> org.apache.cassandra.metrics.TableMetrics.access$000(TableMetrics.java:48)
>   at 
> org.apache.cassandra.metrics.TableMetrics$11.getValue(TableMetrics.java:376)
>   at 
> org.apache.cassandra.metrics.TableMetrics$11.getValue(TableMetrics.java:373)
>   at 
> org.apache.cassandra.metrics.CassandraMetricsRegistry$JmxGauge.getValue(CassandraMetricsRegistry.java:250)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276)
>   at 
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
>

[jira] [Updated] (CASSANDRA-16259) tablehistograms cause ArrayIndexOutOfBoundsException

[Justin (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justin updated CASSANDRA-16259:
---
Description: 
After upgrading some nodes in the cluster from 3.11.8 to 3.11.9 a new error 
appeared on the upgraded nodes when trying to access _*tablehistograms*_. I 
confirmed running the same commands on my non-upgraded .8 nodes returns as 
expected, only the upgraded .9 nodes fail. Not all tables fail when queried, 
but about 90% of them do.

I use Datastax MCAC which must query histograms every 30 seconds, this outputs 
to the system.log:
{noformat}
WARN  [insights-3-1] 2020-11-09 01:11:22,331 UnixSocketClient.java:830 - Error 
reporting:
java.lang.ArrayIndexOutOfBoundsException: 115
at 
org.apache.cassandra.metrics.TableMetrics.combineHistograms(TableMetrics.java:261)
 ~[apache-cassandra-3.11.9.jar:3.11.9]
at 
org.apache.cassandra.metrics.TableMetrics.access$000(TableMetrics.java:48) 
~[apache-cassandra-3.11.9.jar:3.11.9]
at 
org.apache.cassandra.metrics.TableMetrics$11.getValue(TableMetrics.java:376) 
~[apache-cassandra-3.11.9.jar:3.11.9]
at 
org.apache.cassandra.metrics.TableMetrics$11.getValue(TableMetrics.java:373) 
~[apache-cassandra-3.11.9.jar:3.11.9]
at 
com.datastax.mcac.UnixSocketClient.writeMetric(UnixSocketClient.java:839) 
[datastax-mcac-agent.jar:na]
at com.datastax.mcac.UnixSocketClient.access$700(UnixSocketClient.java:78) 
[datastax-mcac-agent.jar:na]
at 
com.datastax.mcac.UnixSocketClient$2.lambda$onGaugeAdded$0(UnixSocketClient.java:626)
 ~[datastax-mcac-agent.jar:na]
at com.datastax.mcac.UnixSocketClient.writeGroup(UnixSocketClient.java:819) 
[datastax-mcac-agent.jar:na]
at 
com.datastax.mcac.UnixSocketClient.lambda$restartMetricReporting$2(UnixSocketClient.java:798)
 [datastax-mcac-agent.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_272]
at 
io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:126) 
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:307) 
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_272]{noformat}
Manually trying a histogram from the CLI:
{noformat}
nodetool tablehistograms logdata log_height_index
error: 115
-- StackTrace --
java.lang.ArrayIndexOutOfBoundsException: 115
at 
org.apache.cassandra.metrics.TableMetrics.combineHistograms(TableMetrics.java:261)
at 
org.apache.cassandra.metrics.TableMetrics.access$000(TableMetrics.java:48)
at 
org.apache.cassandra.metrics.TableMetrics$11.getValue(TableMetrics.java:376)
at 
org.apache.cassandra.metrics.TableMetrics$11.getValue(TableMetrics.java:373)
at 
org.apache.cassandra.metrics.CassandraMetricsRegistry$JmxGauge.getValue(CassandraMetricsRegistry.java:250)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276)
at 
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at 
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at 
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at 
com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
at 
com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
at 
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at 
com.sun.jmx.remote.security.MBeanServerAccessController.getAttribute(MBeanServerAccessController.java:320)
at 
javax

[jira] [Updated] (CASSANDRA-16259) tablehistograms cause ArrayIndexOutOfBoundsException

[Justin (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justin updated CASSANDRA-16259:
---
Description: 
After upgrading some nodes in the cluster from 3.11.8 to 3.11.9 a new error 
appeared on the upgraded nodes. I confirmed running the same commands on my 
non-upgraded .8 nodes returns as expected, only the upgraded .9 nodes fail. Not 
all tables fail when queried, but about 90% of them do.

I use Datastax MCAC which must query histograms every 30 seconds, this outputs 
to the system.log:
{noformat}
WARN  [insights-3-1] 2020-11-09 01:11:22,331 UnixSocketClient.java:830 - Error 
reporting:
java.lang.ArrayIndexOutOfBoundsException: 115
at 
org.apache.cassandra.metrics.TableMetrics.combineHistograms(TableMetrics.java:261)
 ~[apache-cassandra-3.11.9.jar:3.11.9]
at 
org.apache.cassandra.metrics.TableMetrics.access$000(TableMetrics.java:48) 
~[apache-cassandra-3.11.9.jar:3.11.9]
at 
org.apache.cassandra.metrics.TableMetrics$11.getValue(TableMetrics.java:376) 
~[apache-cassandra-3.11.9.jar:3.11.9]
at 
org.apache.cassandra.metrics.TableMetrics$11.getValue(TableMetrics.java:373) 
~[apache-cassandra-3.11.9.jar:3.11.9]
at 
com.datastax.mcac.UnixSocketClient.writeMetric(UnixSocketClient.java:839) 
[datastax-mcac-agent.jar:na]
at com.datastax.mcac.UnixSocketClient.access$700(UnixSocketClient.java:78) 
[datastax-mcac-agent.jar:na]
at 
com.datastax.mcac.UnixSocketClient$2.lambda$onGaugeAdded$0(UnixSocketClient.java:626)
 ~[datastax-mcac-agent.jar:na]
at com.datastax.mcac.UnixSocketClient.writeGroup(UnixSocketClient.java:819) 
[datastax-mcac-agent.jar:na]
at 
com.datastax.mcac.UnixSocketClient.lambda$restartMetricReporting$2(UnixSocketClient.java:798)
 [datastax-mcac-agent.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_272]
at 
io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:126) 
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:307) 
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_272]{noformat}
Manually trying a histogram from the CLI:
{noformat}
nodetool tablehistograms logdata log_height_index
error: 115
-- StackTrace --
java.lang.ArrayIndexOutOfBoundsException: 115
at 
org.apache.cassandra.metrics.TableMetrics.combineHistograms(TableMetrics.java:261)
at 
org.apache.cassandra.metrics.TableMetrics.access$000(TableMetrics.java:48)
at 
org.apache.cassandra.metrics.TableMetrics$11.getValue(TableMetrics.java:376)
at 
org.apache.cassandra.metrics.TableMetrics$11.getValue(TableMetrics.java:373)
at 
org.apache.cassandra.metrics.CassandraMetricsRegistry$JmxGauge.getValue(CassandraMetricsRegistry.java:250)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276)
at 
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at 
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at 
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at 
com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
at 
com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
at 
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at 
com.sun.jmx.remote.security.MBeanServerAccessController.getAttribute(MBeanServerAccessController.java:320)
at 
javax.management.remote.rmi.RMIConnectionImpl.do

[jira] [Updated] (CASSANDRA-16259) tablehistograms cause ArrayIndexOutOfBoundsException

[Justin (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justin updated CASSANDRA-16259:
---
Description: 
After upgrading some nodes in the cluster from 3.11.8 to 3.11.9 a new error 
appeared on the upgraded nodes. I confirmed running the same commands on my 
non-upgraded .8 nodes returns as expected, only the upgraded .9 nodes fail.

I use Datastax MCAC which must query histograms every 30 seconds is why I first 
noticed this in the system.log:
{noformat}
WARN  [insights-3-1] 2020-11-09 01:11:22,331 UnixSocketClient.java:830 - Error 
reporting:
java.lang.ArrayIndexOutOfBoundsException: 115
at 
org.apache.cassandra.metrics.TableMetrics.combineHistograms(TableMetrics.java:261)
 ~[apache-cassandra-3.11.9.jar:3.11.9]
at 
org.apache.cassandra.metrics.TableMetrics.access$000(TableMetrics.java:48) 
~[apache-cassandra-3.11.9.jar:3.11.9]
at 
org.apache.cassandra.metrics.TableMetrics$11.getValue(TableMetrics.java:376) 
~[apache-cassandra-3.11.9.jar:3.11.9]
at 
org.apache.cassandra.metrics.TableMetrics$11.getValue(TableMetrics.java:373) 
~[apache-cassandra-3.11.9.jar:3.11.9]
at 
com.datastax.mcac.UnixSocketClient.writeMetric(UnixSocketClient.java:839) 
[datastax-mcac-agent.jar:na]
at com.datastax.mcac.UnixSocketClient.access$700(UnixSocketClient.java:78) 
[datastax-mcac-agent.jar:na]
at 
com.datastax.mcac.UnixSocketClient$2.lambda$onGaugeAdded$0(UnixSocketClient.java:626)
 ~[datastax-mcac-agent.jar:na]
at com.datastax.mcac.UnixSocketClient.writeGroup(UnixSocketClient.java:819) 
[datastax-mcac-agent.jar:na]
at 
com.datastax.mcac.UnixSocketClient.lambda$restartMetricReporting$2(UnixSocketClient.java:798)
 [datastax-mcac-agent.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_272]
at 
io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:126) 
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:307) 
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_272]{noformat}
I then tried to manually run a histogram from the CLI:
{noformat}
nodetool tablehistograms logdata log_height_index
error: 115
-- StackTrace --
java.lang.ArrayIndexOutOfBoundsException: 115
at 
org.apache.cassandra.metrics.TableMetrics.combineHistograms(TableMetrics.java:261)
at 
org.apache.cassandra.metrics.TableMetrics.access$000(TableMetrics.java:48)
at 
org.apache.cassandra.metrics.TableMetrics$11.getValue(TableMetrics.java:376)
at 
org.apache.cassandra.metrics.TableMetrics$11.getValue(TableMetrics.java:373)
at 
org.apache.cassandra.metrics.CassandraMetricsRegistry$JmxGauge.getValue(CassandraMetricsRegistry.java:250)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276)
at 
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at 
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at 
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at 
com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
at 
com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
at 
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at 
com.sun.jmx.remote.security.MBeanServerAccessController.getAttribute(MBeanServerAccessController.java:320)
at 
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1

[jira] [Created] (CASSANDRA-16259) tablehistograms cause ArrayIndexOutOfBoundsException

[Justin (Jira)]

Justin created CASSANDRA-16259:
--

 Summary: tablehistograms cause ArrayIndexOutOfBoundsException
 Key: CASSANDRA-16259
 URL: https://issues.apache.org/jira/browse/CASSANDRA-16259
 Project: Cassandra
  Issue Type: Bug
Reporter: Justin


After upgrading some nodes in the cluster from 3.11.8 to 3.11.9 a new error 
appeared on the upgraded nodes. I confirmed running the same commands on my 
non-upgraded .8 nodes returns as expected, only the upgraded .9 nodes fail.

I use Datastax MCAC which must query histograms every 30 seconds is why I first 
noticed this in the system.log:
{noformat}
WARN  [insights-3-1] 2020-11-09 01:11:22,331 UnixSocketClient.java:830 - Error 
reporting:java.lang.ArrayIndexOutOfBoundsException: 115at 
org.apache.cassandra.metrics.TableMetrics.combineHistograms(TableMetrics.java:261)
 ~[apache-cassandra-3.11.9.jar:3.11.9]at 
org.apache.cassandra.metrics.TableMetrics.access$000(TableMetrics.java:48) 
~[apache-cassandra-3.11.9.jar:3.11.9]at 
org.apache.cassandra.metrics.TableMetrics$11.getValue(TableMetrics.java:376) 
~[apache-cassandra-3.11.9.jar:3.11.9]at 
org.apache.cassandra.metrics.TableMetrics$11.getValue(TableMetrics.java:373) 
~[apache-cassandra-3.11.9.jar:3.11.9]at 
com.datastax.mcac.UnixSocketClient.writeMetric(UnixSocketClient.java:839) 
[datastax-mcac-agent.jar:na]at 
com.datastax.mcac.UnixSocketClient.access$700(UnixSocketClient.java:78) 
[datastax-mcac-agent.jar:na]at 
com.datastax.mcac.UnixSocketClient$2.lambda$onGaugeAdded$0(UnixSocketClient.java:626)
 ~[datastax-mcac-agent.jar:na]at 
com.datastax.mcac.UnixSocketClient.writeGroup(UnixSocketClient.java:819) 
[datastax-mcac-agent.jar:na]at 
com.datastax.mcac.UnixSocketClient.lambda$restartMetricReporting$2(UnixSocketClient.java:798)
 [datastax-mcac-agent.jar:na]at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_272]at 
io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:126) 
~[netty-all-4.0.44.Final.jar:4.0.44.Final]at 
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]at 
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:307) 
~[netty-all-4.0.44.Final.jar:4.0.44.Final]at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]at 
java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_272]{noformat}
I then tried to manually run a histogram from the CLI:
{noformat}
nodetool tablehistograms logdata log_height_index
error: 115
-- StackTrace --
java.lang.ArrayIndexOutOfBoundsException: 115
at 
org.apache.cassandra.metrics.TableMetrics.combineHistograms(TableMetrics.java:261)
at 
org.apache.cassandra.metrics.TableMetrics.access$000(TableMetrics.java:48)
at 
org.apache.cassandra.metrics.TableMetrics$11.getValue(TableMetrics.java:376)
at 
org.apache.cassandra.metrics.TableMetrics$11.getValue(TableMetrics.java:373)
at 
org.apache.cassandra.metrics.CassandraMetricsRegistry$JmxGauge.getValue(CassandraMetricsRegistry.java:250)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276)
at 
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at 
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at 
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at 
com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
at 
com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
at 
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at 
com.sun.jmx.remote.security.MBeanServerAccessController.getAttribute(MBeanServerAcc

[jira] [Updated] (CASSANDRA-16241) ArrayClustering does not properly handle null clustering key elements left over from tables created WITH COMPACT STORAGE

[Caleb Rackliffe (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Caleb Rackliffe updated CASSANDRA-16241:

Status: Ready to Commit  (was: Review In Progress)

> ArrayClustering does not properly handle null clustering key elements left 
> over from tables created WITH COMPACT STORAGE
> 
>
> Key: CASSANDRA-16241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16241
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Compaction
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
>  Labels: compaction
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The only way we can produce null clustering key elements is leaving them 
> empty on insert while a table is still compact. If we subsequently DROP 
> COMPACT STORAGE, those null elements linger, and {{ArrayClustering}} does not 
> handle them appropriately on compaction. 
> If you run the test 
> [here|https://github.com/maedhroz/cassandra/commit/e247b7868cae383168153bbe8bbbaa47a660f76b],
>  you should be able to observe an exception that looks roughly like this:
> {noformat}
> java.lang.NullPointerException
>   at java.base/java.nio.ByteBuffer.wrap(ByteBuffer.java:422)
>   at 
> org.apache.cassandra.db.AbstractArrayClusteringPrefix.getBufferArray(AbstractArrayClusteringPrefix.java:45)
>   at 
> org.apache.cassandra.io.sstable.metadata.MetadataCollector.finalizeMetadata(MetadataCollector.java:246)
>   at 
> org.apache.cassandra.io.sstable.format.SSTableWriter.finalizeMetadata(SSTableWriter.java:315)
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$200(BigTableWriter.java:52)
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:415)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)
>   at 
> org.apache.cassandra.io.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:283)
>   at 
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:380)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)
>   at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:118)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:179)
>   at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:128)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:225)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> {noformat}
> There are already numerous places where we respect the fact that clustering 
> elements may be null, so this should be pretty straightforward to fix, and 
> the tests that accompany it will probably be more complex than the fix itself.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16241) ArrayClustering does not properly handle null clustering key elements left over from tables created WITH COMPACT STORAGE

[Caleb Rackliffe (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Caleb Rackliffe updated CASSANDRA-16241:

Reviewers: Jordan West, Yifan Cai  (was: Jordan West)

> ArrayClustering does not properly handle null clustering key elements left 
> over from tables created WITH COMPACT STORAGE
> 
>
> Key: CASSANDRA-16241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16241
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Compaction
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
>  Labels: compaction
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The only way we can produce null clustering key elements is leaving them 
> empty on insert while a table is still compact. If we subsequently DROP 
> COMPACT STORAGE, those null elements linger, and {{ArrayClustering}} does not 
> handle them appropriately on compaction. 
> If you run the test 
> [here|https://github.com/maedhroz/cassandra/commit/e247b7868cae383168153bbe8bbbaa47a660f76b],
>  you should be able to observe an exception that looks roughly like this:
> {noformat}
> java.lang.NullPointerException
>   at java.base/java.nio.ByteBuffer.wrap(ByteBuffer.java:422)
>   at 
> org.apache.cassandra.db.AbstractArrayClusteringPrefix.getBufferArray(AbstractArrayClusteringPrefix.java:45)
>   at 
> org.apache.cassandra.io.sstable.metadata.MetadataCollector.finalizeMetadata(MetadataCollector.java:246)
>   at 
> org.apache.cassandra.io.sstable.format.SSTableWriter.finalizeMetadata(SSTableWriter.java:315)
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$200(BigTableWriter.java:52)
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:415)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)
>   at 
> org.apache.cassandra.io.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:283)
>   at 
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:380)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)
>   at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:118)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:179)
>   at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:128)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:225)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> {noformat}
> There are already numerous places where we respect the fact that clustering 
> elements may be null, so this should be pretty straightforward to fix, and 
> the tests that accompany it will probably be more complex than the fix itself.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16241) ArrayClustering does not properly handle null clustering key elements left over from tables created WITH COMPACT STORAGE

[Caleb Rackliffe (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228900#comment-17228900
 ] 

Caleb Rackliffe commented on CASSANDRA-16241:
-

[~jwest] Would you be able to commit this? (It's just trunk...)

> ArrayClustering does not properly handle null clustering key elements left 
> over from tables created WITH COMPACT STORAGE
> 
>
> Key: CASSANDRA-16241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16241
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Compaction
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
>  Labels: compaction
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The only way we can produce null clustering key elements is leaving them 
> empty on insert while a table is still compact. If we subsequently DROP 
> COMPACT STORAGE, those null elements linger, and {{ArrayClustering}} does not 
> handle them appropriately on compaction. 
> If you run the test 
> [here|https://github.com/maedhroz/cassandra/commit/e247b7868cae383168153bbe8bbbaa47a660f76b],
>  you should be able to observe an exception that looks roughly like this:
> {noformat}
> java.lang.NullPointerException
>   at java.base/java.nio.ByteBuffer.wrap(ByteBuffer.java:422)
>   at 
> org.apache.cassandra.db.AbstractArrayClusteringPrefix.getBufferArray(AbstractArrayClusteringPrefix.java:45)
>   at 
> org.apache.cassandra.io.sstable.metadata.MetadataCollector.finalizeMetadata(MetadataCollector.java:246)
>   at 
> org.apache.cassandra.io.sstable.format.SSTableWriter.finalizeMetadata(SSTableWriter.java:315)
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$200(BigTableWriter.java:52)
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:415)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)
>   at 
> org.apache.cassandra.io.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:283)
>   at 
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:380)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)
>   at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:118)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:179)
>   at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:128)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:225)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> {noformat}
> There are already numerous places where we respect the fact that clustering 
> elements may be null, so this should be pretty straightforward to fix, and 
> the tests that accompany it will probably be more complex than the fix itself.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16258) Running read on a mixed cluster with C* version 3.11.6 and version 4.0-alpha1 failed with multiple exceptions

[Yongle Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongle Zhang updated CASSANDRA-16258:
-
Severity: Normal

> Running read on a mixed cluster with C* version 3.11.6 and version 4.0-alpha1 
> failed with multiple exceptions
> -
>
> Key: CASSANDRA-16258
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16258
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Yongle Zhang
>Priority: Normal
>
> Steps to reproduce: 
>  # Set up a  cluster with 2 nodes (1 seed using 3.11.6, the other node using 
> 4.0-alpha1). 
>  # Run stress testing on the seed node running C* 3.11.6. 
> Error messages on each node: 
> Seed (3.11.6):
>  
> {code:java}
> ERROR [RequestResponseStage-1] 2020-06-25 01:58:56,745 
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread 
> Thread[RequestResponseStage-1,5,main]
> java.lang.IllegalArgumentException: Unknown request failure reason error 
> code: 3
>   at 
> org.apache.cassandra.exceptions.RequestFailureReason.fromCode(RequestFailureReason.java:49)
>  ~[main/:na]
>   at org.apache.cassandra.net.MessageIn.getFailureReason(MessageIn.java:189) 
> ~[main/:na]
>   at 
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:47)
>  ~[main/:na]
>   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) 
> ~[main/:na]
>   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_222]
>   at 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165)
>  ~[main/:na]
>   at 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:137)
>  [main/:na]
>   at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:113) 
> [main/:na]
>   at java.lang.Thread.run(Thread.java:748) [na:1.8.0_222]{code}
>  
>  
> Looks like this error message might be because the enum class 
> RequestFailureReason is incompatible between 3.11.6 and 4.0-alpha1: In 3.11.6 
> it contains only 2 possible values 
> ([https://github.com/apache/cassandra/blob/cassandra-3.11.6/src/java/org/apache/cassandra/exceptions/RequestFailureReason.java#L21]),
>  while in 4.0-alpha1 it contains 4 possible values 
> ([https://github.com/apache/cassandra/blob/cassandra-4.0-alpha1/src/java/org/apache/cassandra/exceptions/RequestFailureReason.java#L33]).
>  
>  
> Non-seed node (4.0-alpha1):
>  
> {code:java}
> INFO  [Messaging-EventLoop-3-5] 2020-06-25 01:58:56,743 NoSpamLogger.java:91 
> - 250.16.238.1:7000->250.16.238.2:7000-LEGACY_MESSAGES-0e4d7ae2 incompatible 
> schema encountered while deserializing a message
> org.apache.cassandra.exceptions.UnknownTableException: Couldn't find table 
> with id 5bc52802-de25-35ed-aeab-188eecebb090. If a table was just created, 
> this is likely due to the schemanot being fully propagated.  Please wait for 
> schema agreement on table creation.
>   at 
> org.apache.cassandra.schema.Schema.getExistingTableMetadata(Schema.java:456)
>   at 
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:614)
>   at 
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:344)
>   at 
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:362)
>   at 
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:326)
>   at 
> org.apache.cassandra.net.Message$Serializer.deserializePayloadPre40(Message.java:949)
>   at 
> org.apache.cassandra.net.Message$Serializer.deserializePre40(Message.java:930)
>   at 
> org.apache.cassandra.net.Message$Serializer.deserializePre40(Message.java:918)
>   at org.apache.cassandra.net.Message$Serializer.deserialize(Message.java:618)
>   at 
> org.apache.cassandra.net.InboundMessageHandler.processSmallMessage(InboundMessageHandler.java:321)
>   at 
> org.apache.cassandra.net.InboundMessageHandler.processOneContainedMessage(InboundMessageHandler.java:304)
>   at 
> org.apache.cassandra.net.InboundMessageHandler.processFrameOfContainedMessages(InboundMessageHandler.java:271)
>   at 
> org.apache.cassandra.net.InboundMessageHandler.processIntactFrame(InboundMessageHandler.java:256)
>   at 
> org.apache.cassandra.net.InboundMessageHandler.process(InboundMessageHandler.java:247)
>   at org.apache.cassandra.net.FrameDecoder.deliver(FrameDecoder.java:320)
>   at org.apache.cassandra.net.FrameDecoder.channelRead(FrameDecoder.java:284)
>   at org.apache.cassandra.net.FrameDecoder.channelRead(FrameDecoder.java:268)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.jav

[jira] [Created] (CASSANDRA-16258) Running read on a mixed cluster with C* version 3.11.6 and version 4.0-alpha1 failed with multiple exceptions

[Yongle Zhang (Jira)]

Yongle Zhang created CASSANDRA-16258:


 Summary: Running read on a mixed cluster with C* version 3.11.6 
and version 4.0-alpha1 failed with multiple exceptions
 Key: CASSANDRA-16258
 URL: https://issues.apache.org/jira/browse/CASSANDRA-16258
 Project: Cassandra
  Issue Type: Bug
Reporter: Yongle Zhang


Steps to reproduce: 
 # Set up a  cluster with 2 nodes (1 seed using 3.11.6, the other node using 
4.0-alpha1). 
 # Run stress testing on the seed node running C* 3.11.6. 

Error messages on each node: 

Seed (3.11.6):

 
{code:java}
ERROR [RequestResponseStage-1] 2020-06-25 01:58:56,745 
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread 
Thread[RequestResponseStage-1,5,main]
java.lang.IllegalArgumentException: Unknown request failure reason error code: 3
  at 
org.apache.cassandra.exceptions.RequestFailureReason.fromCode(RequestFailureReason.java:49)
 ~[main/:na]
  at org.apache.cassandra.net.MessageIn.getFailureReason(MessageIn.java:189) 
~[main/:na]
  at 
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:47)
 ~[main/:na]
  at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) 
~[main/:na]
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_222]
  at 
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165)
 ~[main/:na]
  at 
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:137)
 [main/:na]
  at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:113) 
[main/:na]
  at java.lang.Thread.run(Thread.java:748) [na:1.8.0_222]{code}
 

 

Looks like this error message might be because the enum class 
RequestFailureReason is incompatible between 3.11.6 and 4.0-alpha1: In 3.11.6 
it contains only 2 possible values 
([https://github.com/apache/cassandra/blob/cassandra-3.11.6/src/java/org/apache/cassandra/exceptions/RequestFailureReason.java#L21]),
 while in 4.0-alpha1 it contains 4 possible values 
([https://github.com/apache/cassandra/blob/cassandra-4.0-alpha1/src/java/org/apache/cassandra/exceptions/RequestFailureReason.java#L33]).
 

 

Non-seed node (4.0-alpha1):

 
{code:java}
INFO  [Messaging-EventLoop-3-5] 2020-06-25 01:58:56,743 NoSpamLogger.java:91 - 
250.16.238.1:7000->250.16.238.2:7000-LEGACY_MESSAGES-0e4d7ae2 incompatible 
schema encountered while deserializing a message
org.apache.cassandra.exceptions.UnknownTableException: Couldn't find table with 
id 5bc52802-de25-35ed-aeab-188eecebb090. If a table was just created, this is 
likely due to the schemanot being fully propagated.  Please wait for schema 
agreement on table creation.
  at 
org.apache.cassandra.schema.Schema.getExistingTableMetadata(Schema.java:456)
  at 
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:614)
  at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:344)
  at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:362)
  at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:326)
  at 
org.apache.cassandra.net.Message$Serializer.deserializePayloadPre40(Message.java:949)
  at 
org.apache.cassandra.net.Message$Serializer.deserializePre40(Message.java:930)
  at 
org.apache.cassandra.net.Message$Serializer.deserializePre40(Message.java:918)
  at org.apache.cassandra.net.Message$Serializer.deserialize(Message.java:618)
  at 
org.apache.cassandra.net.InboundMessageHandler.processSmallMessage(InboundMessageHandler.java:321)
  at 
org.apache.cassandra.net.InboundMessageHandler.processOneContainedMessage(InboundMessageHandler.java:304)
  at 
org.apache.cassandra.net.InboundMessageHandler.processFrameOfContainedMessages(InboundMessageHandler.java:271)
  at 
org.apache.cassandra.net.InboundMessageHandler.processIntactFrame(InboundMessageHandler.java:256)
  at 
org.apache.cassandra.net.InboundMessageHandler.process(InboundMessageHandler.java:247)
  at org.apache.cassandra.net.FrameDecoder.deliver(FrameDecoder.java:320)
  at org.apache.cassandra.net.FrameDecoder.channelRead(FrameDecoder.java:284)
  at org.apache.cassandra.net.FrameDecoder.channelRead(FrameDecoder.java:268)
  at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
  at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360
  at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
  at 
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
  at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
  at 
io.

[jira] [Commented] (CASSANDRA-16241) ArrayClustering does not properly handle null clustering key elements left over from tables created WITH COMPACT STORAGE

[Yifan Cai (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228891#comment-17228891
 ] 

Yifan Cai commented on CASSANDRA-16241:
---

Beside leaving clustering key empty on insertion, there is one other case that 
generates null values in clustering key. 
When dropping the {{COMPACT STORAGE}} from a table that has no clustering 
columns, a new clustering column is added to the table, and its value could be 
null. I modified your test to simulate that scenario, but compaction does not 
have a problem with it (using trunk code base). 

The patch makes sense to me. +1 and thanks.


> ArrayClustering does not properly handle null clustering key elements left 
> over from tables created WITH COMPACT STORAGE
> 
>
> Key: CASSANDRA-16241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16241
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Compaction
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
>  Labels: compaction
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The only way we can produce null clustering key elements is leaving them 
> empty on insert while a table is still compact. If we subsequently DROP 
> COMPACT STORAGE, those null elements linger, and {{ArrayClustering}} does not 
> handle them appropriately on compaction. 
> If you run the test 
> [here|https://github.com/maedhroz/cassandra/commit/e247b7868cae383168153bbe8bbbaa47a660f76b],
>  you should be able to observe an exception that looks roughly like this:
> {noformat}
> java.lang.NullPointerException
>   at java.base/java.nio.ByteBuffer.wrap(ByteBuffer.java:422)
>   at 
> org.apache.cassandra.db.AbstractArrayClusteringPrefix.getBufferArray(AbstractArrayClusteringPrefix.java:45)
>   at 
> org.apache.cassandra.io.sstable.metadata.MetadataCollector.finalizeMetadata(MetadataCollector.java:246)
>   at 
> org.apache.cassandra.io.sstable.format.SSTableWriter.finalizeMetadata(SSTableWriter.java:315)
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$200(BigTableWriter.java:52)
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:415)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)
>   at 
> org.apache.cassandra.io.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:283)
>   at 
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:380)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)
>   at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:118)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:179)
>   at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:128)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:225)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> {noformat}
> There are already numerous places where we respect the fact that clustering 
> elements may be null, so this should be pretty straightforward to fix, and 
> the tests that accompany it will probably be more complex than the fix itself.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-16257) When upgrading from 2.1.0 to 2.2.0, UnknownColumnFamilyException encountered on upgraded nodes

[Zhuqi Jin (Jira)]

Zhuqi Jin created CASSANDRA-16257:
-

 Summary: When upgrading from 2.1.0 to 2.2.0, 
UnknownColumnFamilyException encountered on upgraded nodes
 Key: CASSANDRA-16257
 URL: https://issues.apache.org/jira/browse/CASSANDRA-16257
 Project: Cassandra
  Issue Type: Bug
Reporter: Zhuqi Jin


Both using 2.1.0 node as seed and using 2.2.0 node as seed will fail with this 
similar error:
{code:java}
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find 
cfId=49a44701-d6bc-11ea-848f-4766f428c026 (when using 2.1.0 node as seed)
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find 
cfId=727d9450-d6c7-11ea-9370-4766f428c026 (when using 2.2.0 node as seed)at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
 ~[apache-cassandra-2.1.0.jar:2.1.0]at 
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
 ~[apache-cassandra-2.1.0.jar:2.1.0]at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
 ~[apache-cassandra-2.1.0.jar:2.1.0]at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
 ~[apache-cassandra-2.1.0.jar:2.1.0]at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
 ~[apache-cassandra-2.1.0.jar:2.1.0]at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
 ~[apache-cassandra-2.1.0.jar:2.1.0]at 
org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) 
~[apache-cassandra-2.1.0.jar:2.1.0]at 
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:165)
 ~[apache-cassandra-2.1.0.jar:2.1.0]at 
org.apache.cassandra.net.IncomingTcpConnection.handleModernVersion(IncomingTcpConnection.java:147)
 ~[apache-cassandra-2.1.0.jar:2.1.0]at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
 ~[apache-cassandra-2.1.0.jar:2.1.0]{code}
This seems to originate from the difference in the settings in the stress 
tests, and is not a data format bug. The field `cfId` is a UUID string 
generated from another two fields: `ksName` and `cfName`. These two names are 
not the interior settings in the cluster, instead they come from user 
specification.

In 2.1.0, the stress test configures these two names to be “Keyspace1” and 
“Standard1”. See 
[https://github.com/apache/cassandra/blob/cassandra-2.1.0/tools/stress/src/org/apache/cassandra/stress/settings/SettingsSchema.java#L84]
 . Here is my log:
{code:java}
WARN  [MigrationStage:1] 2020-08-05 03:17:20,514 Schema.java:330 - Schema.load: 
Adding org.apache.cassandra.config.CFMetaData@6299796f…..., key = 
(Keyspace1,Standard1), val = 49a44701-d6bc-11ea-848f-4766f428c026
{code}
In 2.2.0, the stress test configures these two names to be “keyspace1” and 
“standard1” (note that the names are case-sensitive). See 
https://github.com/apache/cassandra/blob/cassandra-2.2.0/tools/stress/src/org/apache/cassandra/stress/settings/SettingsSchema.java#L225
  . Here is my log:
{code:java}
WARN  [MigrationStage:1] 2020-08-05 02:57:49,680 Schema.java:350 - Schema.load: 
Adding org.apache.cassandra.config.CFMetaData@719822c2……..., key = 
(keyspace1,standard1), val = 727d9450-d6c7-11ea-9370-4766f428c026
{code}
When a 2.2.0 message reaches a 2.1.0 node, or vice versa, the receiver will 
look up the `cfId` field in a map. Since the key is not present in its own map, 
the look up will fail with the above message. Unluckily this problem is quite 
hard to fix, since many other places also depend on the name. So I can not 
overcome this problem and find actual data format bugs between these two 
versions currently.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16256) jvm dtest is strict on properties which causes upgrade tests to fail

[David Capwell (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228866#comment-17228866
 ] 

David Capwell commented on CASSANDRA-16256:
---

since [~e.dimitrova] was the reporter wanted to get her review as well, once 
she approves will merge (expected tomorrow).

> jvm dtest is strict on properties which causes upgrade tests to fail
> 
>
> Key: CASSANDRA-16256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16256
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest/java
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> {code}
> ConfigurationException: Invalid yaml. Please remove properties 
> [cdc_raw_directory, diagnostic_events_enabled] from your cassandra.yaml
> {code}
> This was done in CASSANDRA-16152 to make sure we don’t have typo’s in 
> configs, but this makes upgrades harder as InstanceConfig is used and this 
> isn’t version aware.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16256) jvm dtest is strict on properties which causes upgrade tests to fail

[David Capwell (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Capwell updated CASSANDRA-16256:
--
Reviewers: Caleb Rackliffe, Jordan West, Yifan Cai, David Capwell  (was: 
Caleb Rackliffe, David Capwell, Jordan West, Yifan Cai)
   Caleb Rackliffe, Jordan West, Yifan Cai, David Capwell  (was: 
Caleb Rackliffe, Yifan Cai)
   Status: Review In Progress  (was: Patch Available)

> jvm dtest is strict on properties which causes upgrade tests to fail
> 
>
> Key: CASSANDRA-16256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16256
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest/java
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> {code}
> ConfigurationException: Invalid yaml. Please remove properties 
> [cdc_raw_directory, diagnostic_events_enabled] from your cassandra.yaml
> {code}
> This was done in CASSANDRA-16152 to make sure we don’t have typo’s in 
> configs, but this makes upgrades harder as InstanceConfig is used and this 
> isn’t version aware.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16256) jvm dtest is strict on properties which causes upgrade tests to fail

[David Capwell (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Capwell updated CASSANDRA-16256:
--
Reviewers: Caleb Rackliffe, Jordan West, Yifan Cai  (was: Caleb Rackliffe, 
David Capwell, Jordan West, Yifan Cai)

> jvm dtest is strict on properties which causes upgrade tests to fail
> 
>
> Key: CASSANDRA-16256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16256
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest/java
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> {code}
> ConfigurationException: Invalid yaml. Please remove properties 
> [cdc_raw_directory, diagnostic_events_enabled] from your cassandra.yaml
> {code}
> This was done in CASSANDRA-16152 to make sure we don’t have typo’s in 
> configs, but this makes upgrades harder as InstanceConfig is used and this 
> isn’t version aware.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-15214) OOMs caught and not rethrown

[David Capwell (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228865#comment-17228865
 ] 

David Capwell commented on CASSANDRA-15214:
---

+1

Need second reviewer, can merge after.

> OOMs caught and not rethrown
> 
>
> Key: CASSANDRA-15214
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15214
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Client, Messaging/Internode
>Reporter: Benedict Elliott Smith
>Assignee: Yifan Cai
>Priority: Normal
> Fix For: 4.0, 4.0-rc
>
> Attachments: oom-experiments.zip
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Netty (at least, and perhaps elsewhere in Executors) catches all exceptions, 
> so presently there is no way to ensure that an OOM reaches the JVM handler to 
> trigger a crash/heapdump.
> It may be that the simplest most consistent way to do this would be to have a 
> single thread spawned at startup that waits for any exceptions we must 
> propagate to the Runtime.
> We could probably submit a patch upstream to Netty, but for a guaranteed 
> future proof approach, it may be worth paying the cost of a single thread.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16256) jvm dtest is strict on properties which causes upgrade tests to fail

[Jon Meredith (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228853#comment-17228853
 ] 

Jon Meredith commented on CASSANDRA-16256:
--

Definitely, thanks for the speedy fix. Sorry it got into the codebase.

> jvm dtest is strict on properties which causes upgrade tests to fail
> 
>
> Key: CASSANDRA-16256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16256
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest/java
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> {code}
> ConfigurationException: Invalid yaml. Please remove properties 
> [cdc_raw_directory, diagnostic_events_enabled] from your cassandra.yaml
> {code}
> This was done in CASSANDRA-16152 to make sure we don’t have typo’s in 
> configs, but this makes upgrades harder as InstanceConfig is used and this 
> isn’t version aware.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16256) jvm dtest is strict on properties which causes upgrade tests to fail

[Jordan West (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228848#comment-17228848
 ] 

Jordan West commented on CASSANDRA-16256:
-

+1. Thanks for the quick fix. 

> jvm dtest is strict on properties which causes upgrade tests to fail
> 
>
> Key: CASSANDRA-16256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16256
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest/java
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> {code}
> ConfigurationException: Invalid yaml. Please remove properties 
> [cdc_raw_directory, diagnostic_events_enabled] from your cassandra.yaml
> {code}
> This was done in CASSANDRA-16152 to make sure we don’t have typo’s in 
> configs, but this makes upgrades harder as InstanceConfig is used and this 
> isn’t version aware.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16256) jvm dtest is strict on properties which causes upgrade tests to fail

[Yifan Cai (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yifan Cai updated CASSANDRA-16256:
--
Reviewers: Caleb Rackliffe, Yifan Cai  (was: Caleb Rackliffe)

> jvm dtest is strict on properties which causes upgrade tests to fail
> 
>
> Key: CASSANDRA-16256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16256
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest/java
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> {code}
> ConfigurationException: Invalid yaml. Please remove properties 
> [cdc_raw_directory, diagnostic_events_enabled] from your cassandra.yaml
> {code}
> This was done in CASSANDRA-16152 to make sure we don’t have typo’s in 
> configs, but this makes upgrades harder as InstanceConfig is used and this 
> isn’t version aware.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16256) jvm dtest is strict on properties which causes upgrade tests to fail

[Yifan Cai (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228847#comment-17228847
 ] 

Yifan Cai commented on CASSANDRA-16256:
---

Patches lgtm. +1 
I was able to run the upgrade test locally using David's branches. 
We might want to relocate the {{Constants}} into the dtest-api later. 

> jvm dtest is strict on properties which causes upgrade tests to fail
> 
>
> Key: CASSANDRA-16256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16256
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest/java
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> {code}
> ConfigurationException: Invalid yaml. Please remove properties 
> [cdc_raw_directory, diagnostic_events_enabled] from your cassandra.yaml
> {code}
> This was done in CASSANDRA-16152 to make sure we don’t have typo’s in 
> configs, but this makes upgrades harder as InstanceConfig is used and this 
> isn’t version aware.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16192) Add more tests to cover compaction metrics

[Adam Holmberg (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Holmberg updated CASSANDRA-16192:
--
Reviewers: Adam Holmberg, Adam Holmberg  (was: Adam Holmberg)
   Adam Holmberg, Adam Holmberg  (was: Adam Holmberg)
   Status: Review In Progress  (was: Patch Available)

> Add more tests to cover compaction metrics
> --
>
> Key: CASSANDRA-16192
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16192
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Test/unit
>Reporter: Benjamin Lerer
>Assignee: Mohamed Zafraan
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: 0001-added-unit-tests-to-cover-compaction-metrics.patch
>
>
> Some compaction metrics do not seems to be tested.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16237) Fix flaky test mixedModeReadRepairUpdate - org.apache.cassandra.distributed.upgrade.MixedModeReadRepairTest

[Caleb Rackliffe (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228834#comment-17228834
 ] 

Caleb Rackliffe commented on CASSANDRA-16237:
-

[~adelapena] I'm going to take a shot at splitting this up. I'll probably be 
asking for your review soon...

> Fix flaky test mixedModeReadRepairUpdate - 
> org.apache.cassandra.distributed.upgrade.MixedModeReadRepairTest
> ---
>
> Key: CASSANDRA-16237
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16237
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest/java
>Reporter: David Capwell
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 4.0-beta
>
>
> [https://app.circleci.com/pipelines/github/dcapwell/cassandra/745/workflows/1c7e589e-b5af-4a56-b40a-43da424602c7/jobs/4233]
>  
> {code}
> java.lang.RuntimeException: java.lang.OutOfMemoryError: Metaspace
>   at 
> org.apache.cassandra.distributed.impl.Instance.lambda$startup$11(Instance.java:514)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.OutOfMemoryError: Metaspace
>   at java.lang.ClassLoader.defineClass1(Native Method)
>   at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
>   at 
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
>   at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
>   at 
> org.apache.cassandra.distributed.shared.InstanceClassLoader.loadClassInternal(InstanceClassLoader.java:101)
>   at 
> org.apache.cassandra.distributed.shared.InstanceClassLoader.loadClass(InstanceClassLoader.java:87)
>   at 
> org.apache.cassandra.net.OutboundConnections$UnusedConnectionMonitor.(OutboundConnections.java:265)
>   at 
> org.apache.cassandra.net.OutboundConnections.scheduleUnusedConnectionMonitoring(OutboundConnections.java:312)
>   at 
> org.apache.cassandra.net.MessagingService.(MessagingService.java:262)
>   at 
> org.apache.cassandra.net.MessagingService$MSHandle.(MessagingService.java:226)
>   at 
> org.apache.cassandra.net.MessagingService.instance(MessagingService.java:231)
>   at 
> org.apache.cassandra.distributed.impl.Instance.registerMockMessaging(Instance.java:253)
>   at 
> org.apache.cassandra.distributed.impl.Instance.lambda$startup$11(Instance.java:466)
>   at 
> org.apache.cassandra.distributed.impl.Instance$$Lambda$16500/806573706.run(Unknown
>  Source)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16183) Add tests to cover ClientRequest metrics

[Adam Holmberg (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228820#comment-17228820
 ] 

Adam Holmberg commented on CASSANDRA-16183:
---

Thanks for the review. I pushed a commit with most of the suggested tweaks. One 
exception, I left the nominal tests as-written to see if you had any further 
thoughts on this bit:
https://github.com/aholmberg/cassandra-dtest/pull/1#discussion_r518930951

> Add tests to cover ClientRequest metrics 
> -
>
> Key: CASSANDRA-16183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16183
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Test/dtest/java
>Reporter: Benjamin Lerer
>Assignee: Adam Holmberg
>Priority: Normal
> Fix For: 4.0-beta
>
>
> We do not have test that covers the ClientRequest metrics.
> * ClientRequestMetrics
> * CASClientRequestMetrics
> * CASClientWriteRequestMetrics
> * ClientWriteRequestMetrics
> * ViewWriteMetrics



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-16237) Fix flaky test mixedModeReadRepairUpdate - org.apache.cassandra.distributed.upgrade.MixedModeReadRepairTest

[Caleb Rackliffe (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Caleb Rackliffe reassigned CASSANDRA-16237:
---

Assignee: Caleb Rackliffe

> Fix flaky test mixedModeReadRepairUpdate - 
> org.apache.cassandra.distributed.upgrade.MixedModeReadRepairTest
> ---
>
> Key: CASSANDRA-16237
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16237
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest/java
>Reporter: David Capwell
>Assignee: Caleb Rackliffe
>Priority: Normal
> Fix For: 4.0-beta
>
>
> [https://app.circleci.com/pipelines/github/dcapwell/cassandra/745/workflows/1c7e589e-b5af-4a56-b40a-43da424602c7/jobs/4233]
>  
> {code}
> java.lang.RuntimeException: java.lang.OutOfMemoryError: Metaspace
>   at 
> org.apache.cassandra.distributed.impl.Instance.lambda$startup$11(Instance.java:514)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.OutOfMemoryError: Metaspace
>   at java.lang.ClassLoader.defineClass1(Native Method)
>   at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
>   at 
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
>   at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
>   at 
> org.apache.cassandra.distributed.shared.InstanceClassLoader.loadClassInternal(InstanceClassLoader.java:101)
>   at 
> org.apache.cassandra.distributed.shared.InstanceClassLoader.loadClass(InstanceClassLoader.java:87)
>   at 
> org.apache.cassandra.net.OutboundConnections$UnusedConnectionMonitor.(OutboundConnections.java:265)
>   at 
> org.apache.cassandra.net.OutboundConnections.scheduleUnusedConnectionMonitoring(OutboundConnections.java:312)
>   at 
> org.apache.cassandra.net.MessagingService.(MessagingService.java:262)
>   at 
> org.apache.cassandra.net.MessagingService$MSHandle.(MessagingService.java:226)
>   at 
> org.apache.cassandra.net.MessagingService.instance(MessagingService.java:231)
>   at 
> org.apache.cassandra.distributed.impl.Instance.registerMockMessaging(Instance.java:253)
>   at 
> org.apache.cassandra.distributed.impl.Instance.lambda$startup$11(Instance.java:466)
>   at 
> org.apache.cassandra.distributed.impl.Instance$$Lambda$16500/806573706.run(Unknown
>  Source)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16256) jvm dtest is strict on properties which causes upgrade tests to fail

[David Capwell (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228818#comment-17228818
 ] 

David Capwell commented on CASSANDRA-16256:
---

bq. metaspace problem that I'm guessing would g away on re-run

nope, this is a known issue, see CASSANDRA-16237.  

> jvm dtest is strict on properties which causes upgrade tests to fail
> 
>
> Key: CASSANDRA-16256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16256
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest/java
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> {code}
> ConfigurationException: Invalid yaml. Please remove properties 
> [cdc_raw_directory, diagnostic_events_enabled] from your cassandra.yaml
> {code}
> This was done in CASSANDRA-16152 to make sure we don’t have typo’s in 
> configs, but this makes upgrades harder as InstanceConfig is used and this 
> isn’t version aware.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16256) jvm dtest is strict on properties which causes upgrade tests to fail

[Caleb Rackliffe (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228816#comment-17228816
 ] 

Caleb Rackliffe commented on CASSANDRA-16256:
-

+1 on all 4 branches (which look more or less identical), and the Circle run 
looks fine, albeit with a metaspace problem that I'm guessing would g away on 
re-run. I was initially wondering if we might just maintain a list of 
properties to ignore on older versions, but the proposed patch avoiding that 
altogether and ignoring the checks altogether for upgrade tests seems like it 
should be easier to maintain. Thanks!

> jvm dtest is strict on properties which causes upgrade tests to fail
> 
>
> Key: CASSANDRA-16256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16256
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest/java
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> {code}
> ConfigurationException: Invalid yaml. Please remove properties 
> [cdc_raw_directory, diagnostic_events_enabled] from your cassandra.yaml
> {code}
> This was done in CASSANDRA-16152 to make sure we don’t have typo’s in 
> configs, but this makes upgrades harder as InstanceConfig is used and this 
> isn’t version aware.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16253) Fix documentation to reflect real defaults for commit logs

[Michael Semb Wever (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Semb Wever updated CASSANDRA-16253:
---
  Fix Version/s: (was: 4.0.x)
 (was: 3.11.x)
 3.11.9
 4.0-beta4
Source Control Link: 
https://github.com/apache/cassandra/commit/73a90b987dddedf6a13101c7d90114da669f52f5
 Resolution: Fixed
 Status: Resolved  (was: Ready to Commit)

Committed as 
[73a90b987dddedf6a13101c7d90114da669f52f5|https://github.com/apache/cassandra/commit/73a90b987dddedf6a13101c7d90114da669f52f5].

Thanks [~arodrime]!

> Fix documentation to reflect real defaults for commit logs
> --
>
> Key: CASSANDRA-16253
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16253
> Project: Cassandra
>  Issue Type: Task
>  Components: Documentation/Website
>Reporter: Alain RODRIGUEZ
>Assignee: Michael Semb Wever
>Priority: Normal
> Fix For: 4.0-beta4, 3.11.9
>
>
> About the documentation on commit logs:
>  * Commit logs default is every 10 sec - *periodic*, not batch 
> [https://github.com/apache/cassandra/blob/cassandra-3.11.8/conf/cassandra.yaml#L385-L386]
>  (same in 3.11.9 or trunk). It can lead to data losses but still makes sense 
> to me because fsync-ing every write is very expensive, and (even in batches) 
> should be reserved for the super paranoid where handling multiple power-out 
> (or sudden disk failures) simultaneously on multiple nodes needs to be 
> handled (as [~mck] explained to me).
>   * Nonetheless, apart from the theory behind it, in a more practical 
> perspective, *the doc is wrong* and says *Default Value: batch* which mislead 
> me and this could happen to more people: 
> [https://cassandra.apache.org/doc/latest/architecture/storage_engine.html#commit-log]
>  
> I think this is important enough for us to quickly fix the documentation in 
> trunk. As this information is fully missing in the 3.11 branch documentation, 
> I suggest we add it as well on that branch (based on information about commit 
> logs from trunk).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-16253) Fix documentation to reflect real defaults for commit logs

[Michael Semb Wever (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228808#comment-17228808
 ] 

Michael Semb Wever edited comment on CASSANDRA-16253 at 11/9/20, 8:55 PM:
--

Perfect!

Committed as 
[73a90b987dddedf6a13101c7d90114da669f52f5|https://github.com/apache/cassandra/commit/73a90b987dddedf6a13101c7d90114da669f52f5].

Thanks [~arodrime]!


was (Author: michaelsembwever):
Committed as 
[73a90b987dddedf6a13101c7d90114da669f52f5|https://github.com/apache/cassandra/commit/73a90b987dddedf6a13101c7d90114da669f52f5].

Thanks [~arodrime]!

> Fix documentation to reflect real defaults for commit logs
> --
>
> Key: CASSANDRA-16253
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16253
> Project: Cassandra
>  Issue Type: Task
>  Components: Documentation/Website
>Reporter: Alain RODRIGUEZ
>Assignee: Michael Semb Wever
>Priority: Normal
> Fix For: 3.11.9, 4.0-beta4
>
>
> About the documentation on commit logs:
>  * Commit logs default is every 10 sec - *periodic*, not batch 
> [https://github.com/apache/cassandra/blob/cassandra-3.11.8/conf/cassandra.yaml#L385-L386]
>  (same in 3.11.9 or trunk). It can lead to data losses but still makes sense 
> to me because fsync-ing every write is very expensive, and (even in batches) 
> should be reserved for the super paranoid where handling multiple power-out 
> (or sudden disk failures) simultaneously on multiple nodes needs to be 
> handled (as [~mck] explained to me).
>   * Nonetheless, apart from the theory behind it, in a more practical 
> perspective, *the doc is wrong* and says *Default Value: batch* which mislead 
> me and this could happen to more people: 
> [https://cassandra.apache.org/doc/latest/architecture/storage_engine.html#commit-log]
>  
> I think this is important enough for us to quickly fix the documentation in 
> trunk. As this information is fully missing in the 3.11 branch documentation, 
> I suggest we add it as well on that branch (based on information about commit 
> logs from trunk).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch trunk updated (159e021 -> 4dc1b5c)

[mck]

This is an automated email from the ASF dual-hosted git repository.

mck pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from 159e021  Merge branch 'cassandra-3.11' into trunk
 new 73a90b9  Fix documentation to reflect real defaults for commit logs
 new 4dc1b5c  Merge branch 'cassandra-3.11' into trunk

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 doc/source/architecture/storage_engine.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch cassandra-3.11 updated: Fix documentation to reflect real defaults for commit logs

[mck]

This is an automated email from the ASF dual-hosted git repository.

mck pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/cassandra-3.11 by this push:
 new 73a90b9  Fix documentation to reflect real defaults for commit logs
73a90b9 is described below

commit 73a90b987dddedf6a13101c7d90114da669f52f5
Author: Alain Rodriguez 
AuthorDate: Mon Nov 9 11:22:44 2020 +0100

Fix documentation to reflect real defaults for commit logs

 patch by Alain Rodriguez; reviewed by Mick Semb Wever for CASSANDRA-16253
---
 doc/source/architecture/storage_engine.rst | 49 +-
 1 file changed, 48 insertions(+), 1 deletion(-)

diff --git a/doc/source/architecture/storage_engine.rst 
b/doc/source/architecture/storage_engine.rst
index e4114e5..2bd429d 100644
--- a/doc/source/architecture/storage_engine.rst
+++ b/doc/source/architecture/storage_engine.rst
@@ -22,7 +22,54 @@ Storage Engine
 CommitLog
 ^
 
-.. todo:: todo
+Commitlogs are an append only log of all mutations local to a Cassandra node. 
Any data written to Cassandra will first be written to a commit log before 
being written to a memtable. This provides durability in the case of unexpected 
shutdown. On startup, any mutations in the commit log will be applied to 
memtables.
+
+All mutations write optimized by storing in commitlog segments, reducing the 
number of seeks needed to write to disk. Commitlog Segments are limited by the 
"commitlog_segment_size_in_mb" option, once the size is reached, a new 
commitlog segment is created. Commitlog segments can be archived, deleted, or 
recycled once all its data has been flushed to SSTables.  Commitlog segments 
are truncated when Cassandra has written data older than a certain point to the 
SSTables. Running "nodetool dr [...]
+
+- ``commitlog_segment_size_in_mb``: The default size is 32, which is almost 
always fine, but if you are archiving commitlog segments (see 
commitlog_archiving.properties), then you probably want a finer granularity of 
archiving; 8 or 16 MB is reasonable. Max mutation size is also configurable via 
max_mutation_size_in_kb setting in cassandra.yaml. The default is half the size 
commitlog_segment_size_in_mb * 1024.
+
+***NOTE: If max_mutation_size_in_kb is set explicitly then 
commitlog_segment_size_in_mb must be set to at least twice the size of 
max_mutation_size_in_kb / 1024***
+
+*Default Value:* 32
+
+Commitlogs are an append only log of all mutations local to a Cassandra node. 
Any data written to Cassandra will first be written to a commit log before 
being written to a memtable. This provides durability in the case of unexpected 
shutdown. On startup, any mutations in the commit log will be applied.
+
+- ``commitlog_sync``: may be either “periodic” or “batch.”
+
+  - ``batch``: In batch mode, Cassandra won’t ack writes until the commit log 
has been fsynced to disk. It will wait "commitlog_sync_batch_window_in_ms" 
milliseconds between fsyncs. This window should be kept short because the 
writer threads will be unable to do extra work while waiting. You may need to 
increase concurrent_writes for the same reason.
+
+- ``commitlog_sync_batch_window_in_ms``: Time to wait between "batch" 
fsyncs
+*Default Value:* 2
+
+  - ``periodic``: In periodic mode, writes are immediately ack'ed, and the 
CommitLog is simply synced every "commitlog_sync_period_in_ms" milliseconds.
+
+- ``commitlog_sync_period_in_ms``: Time to wait between "periodic" fsyncs
+*Default Value:* 1
+
+*Default Value:* periodic
+
+*** NOTE: In the event of an unexpected shutdown, Cassandra can lose up to the 
sync period or more if the sync is delayed. If using "batch" mode, it is 
recommended to store commitlogs in a separate, dedicated device.**
+
+
+- ``commitlog_directory``: This option is commented out by default When 
running on magnetic HDD, this should be a separate spindle than the data 
directories. If not set, the default directory is 
$CASSANDRA_HOME/data/commitlog.
+
+*Default Value:* /var/lib/cassandra/commitlog
+
+- ``commitlog_compression``: Compression to apply to the commitlog. If 
omitted, the commit log will be written uncompressed. LZ4, Snappy, Deflate and 
Zstd compressors are supported.
+
+(Default Value: (complex option)::
+
+#   - class_name: LZ4Compressor
+# parameters:
+# -
+
+- ``commitlog_total_space_in_mb``: Total space to use for commit logs on disk.
+
+If space gets above this value, Cassandra will flush every dirty CF in the 
oldest segment and remove it. So a small total commitlog space will tend to 
cause more flush activity on less-active columnfamilies.
+
+The default value is the smaller of 8192, and 1/4 of the total space of the 
commitlog volume.
+
+*Default Value:* 8192
 
 .. _memtables:
 


-
To unsubscribe, e-mail: commits-un

[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk

[mck]

This is an automated email from the ASF dual-hosted git repository.

mck pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 4dc1b5c3119affd22e40f5fbe8661b9da052aa2b
Merge: 159e021 73a90b9
Author: Mick Semb Wever 
AuthorDate: Mon Nov 9 21:34:47 2020 +0100

Merge branch 'cassandra-3.11' into trunk

 doc/source/architecture/storage_engine.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)



-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15158) Wait for schema agreement rather than in flight schema requests when bootstrapping

[Blake Eggleston (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Blake Eggleston updated CASSANDRA-15158:

  Since Version: 3.0.0  (was: 3.1)
Source Control Link: 
https://github.com/apache/cassandra/commit/08450080614250a8bfaba23dbca741a4d9315e3c
 Resolution: Fixed
 Status: Resolved  (was: Ready to Commit)

[~stefan.miklosovic] it does, thanks to you and [~aleksey] for the reviews. 
Committed to 3.0 and merged up to trunk.

> Wait for schema agreement rather than in flight schema requests when 
> bootstrapping
> --
>
> Key: CASSANDRA-15158
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15158
> Project: Cassandra
>  Issue Type: Bug
>  Components: Cluster/Gossip, Cluster/Schema
>Reporter: Vincent White
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently when a node is bootstrapping we use a set of latches 
> (org.apache.cassandra.service.MigrationTask#inflightTasks) to keep track of 
> in-flight schema pull requests, and we don't proceed with 
> bootstrapping/stream until all the latches are released (or we timeout 
> waiting for each one). One issue with this is that if we have a large schema, 
> or the retrieval of the schema from the other nodes was unexpectedly slow 
> then we have no explicit check in place to ensure we have actually received a 
> schema before we proceed.
> While it's possible to increase "migration_task_wait_in_seconds" to force the 
> node to wait on each latche longer, there are cases where this doesn't help 
> because the callbacks for the schema pull requests have expired off the 
> messaging service's callback map 
> (org.apache.cassandra.net.MessagingService#callbacks) after 
> request_timeout_in_ms (default 10 seconds) before the other nodes were able 
> to respond to the new node.
> This patch checks for schema agreement between the bootstrapping node and the 
> rest of the live nodes before proceeding with bootstrapping. It also adds a 
> check to prevent the new node from flooding existing nodes with simultaneous 
> schema pull requests as can happen in large clusters.
> Removing the latch system should also prevent new nodes in large clusters 
> getting stuck for extended amounts of time as they wait 
> `migration_task_wait_in_seconds` on each of the latches left orphaned by the 
> timed out callbacks.
>  
> ||3.11||
> |[PoC|https://github.com/apache/cassandra/compare/cassandra-3.11...vincewhite:check_for_schema]|
> |[dtest|https://github.com/apache/cassandra-dtest/compare/master...vincewhite:wait_for_schema_agreement]|
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk

[bdeggleston]

This is an automated email from the ASF dual-hosted git repository.

bdeggleston pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 159e021aa36a44cdc7674dd234d4a6e189478280
Merge: 85f69cf fe70155
Author: Blake Eggleston 
AuthorDate: Mon Nov 9 12:22:43 2020 -0800

Merge branch 'cassandra-3.11' into trunk

 CHANGES.txt|   1 +
 .../cassandra/schema/MigrationCoordinator.java | 516 +
 .../apache/cassandra/schema/MigrationManager.java  | 149 +-
 .../org/apache/cassandra/schema/MigrationTask.java | 111 -
 src/java/org/apache/cassandra/schema/Schema.java   |   2 +-
 .../cassandra/schema/SchemaMigrationEvent.java |   7 +-
 .../apache/cassandra/service/StorageService.java   |  63 ++-
 .../cassandra/service/StorageServiceMBean.java |   7 +
 .../cassandra/distributed/action/GossipHelper.java |  10 +-
 .../cassandra/schema/MigrationCoordinatorTest.java | 319 +
 10 files changed, 908 insertions(+), 277 deletions(-)

diff --cc CHANGES.txt
index f90ffd4,b81b0c8..1b08837
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,41 -1,10 +1,42 @@@
 -3.11.10
 +4.0-beta4
 + * Add a ratelimiter to snapshot creation and deletion (CASSANDRA-13019)
 + * Produce consistent tombstone for reads to avoid digest mistmatch 
(CASSANDRA-15369)
 + * Fix SSTableloader issue when restoring a table named backups 
(CASSANDRA-16235)
 + * Invalid serialized size for responses caused by increasing message time by 
1ms which caused extra bytes in size calculation (CASSANDRA-16103)
 + * Throw BufferOverflowException from DataOutputBuffer for better visibility 
(CASSANDRA-16214)
 + * TLS connections to the storage port on a node without server encryption 
configured causes java.io.IOException accessing missing keystore 
(CASSANDRA-16144)
 +Merged from 3.11:
  Merged from 3.0:
 - * Fix invalid cell value skipping when reading from disk (CASSANDRA-16223)
++ * Wait for schema agreement when bootstrapping (CASSANDRA-15158)
   * Prevent invoking enable/disable gossip when not in NORMAL (CASSANDRA-16146)
  
 -3.11.9
 - * Synchronize Keyspace instance store/clear (CASSANDRA-16210)
 +4.0-beta3
 + * Segregate Network and Chunk Cache BufferPools and Recirculate Partially 
Freed Chunks (CASSANDRA-15229)
 + * Fail truncation requests when they fail on a replica (CASSANDRA-16208)
 + * Move compact storage validation earlier in startup process 
(CASSANDRA-16063)
 + * Fix ByteBufferAccessor cast exceptions are thrown when trying to query a 
virtual table (CASSANDRA-16155)
 + * Consolidate node liveness check for forced repair (CASSANDRA-16113)
 + * Use unsigned short in ValueAccessor.sliceWithShortLength (CASSANDRA-16147)
 + * Abort repairs when getting a truncation request (CASSANDRA-15854)
 + * Remove bad assert when getting active compactions for an sstable 
(CASSANDRA-15457)
 + * Avoid failing compactions with very large partitions (CASSANDRA-15164)
 + * Prevent NPE in StreamMessage in type lookup (CASSANDRA-16131)
 + * Avoid invalid state transition exception during incremental repair 
(CASSANDRA-16067)
 + * Allow zero padding in timestamp serialization (CASSANDRA-16105)
 + * Add byte array backed cells (CASSANDRA-15393)
 + * Correctly handle pending ranges with adjacent range movements 
(CASSANDRA-14801)
 + * Avoid adding locahost when streaming trivial ranges (CASSANDRA-16099)
 + * Add nodetool getfullquerylog (CASSANDRA-15988)
 + * Fix yaml format and alignment in tpstats (CASSANDRA-11402)
 + * Avoid trying to keep track of RTs for endpoints we won't write to during 
read repair (CASSANDRA-16084)
 + * When compaction gets interrupted, the exception should include the 
compactionId (CASSANDRA-15954)
 + * Make Table/Keyspace Metric Names Consistent With Each Other 
(CASSANDRA-15909)
 + * Mutating sstable component may race with entire-sstable-streaming(ZCS) 
causing checksum validation failure (CASSANDRA-15861)
 + * NPE thrown while updating speculative execution time if keyspace is 
removed during task execution (CASSANDRA-15949)
 + * Show the progress of data streaming and index build (CASSANDRA-15406)
 + * Add flag to disable chunk cache and disable by default (CASSANDRA-16036)
 + * Upgrade to snakeyaml >= 1.26 version for CVE-2017-18640 fix 
(CASSANDRA-16150)
 +Merged from 3.11:
   * Fix ColumnFilter to avoid querying cells of unselected complex columns 
(CASSANDRA-15977)
   * Fix memory leak in CompressedChunkReader (CASSANDRA-15880)
   * Don't attempt value skipping with mixed version cluster (CASSANDRA-15833)
diff --cc src/java/org/apache/cassandra/schema/MigrationCoordinator.java
index 000,000..9fb348e
new file mode 100644
--- /dev/null
+++ b/src/java/org/apache/cassandra/schema/MigrationCoordinator.java
@@@ -1,0 -1,0 +1,516 @@@
++/*
++ * Licensed to the Apache Software Foundation (ASF) under one
++ * or more contributor license agreements.  See the NOTICE file
++ * distributed with this work for additional in

[cassandra] 01/01: Merge branch 'cassandra-3.0' into cassandra-3.11

[bdeggleston]

This is an automated email from the ASF dual-hosted git repository.

bdeggleston pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit fe701556a1d5c3955f79f957f835406319bb97ed
Merge: 94f940c 0845008
Author: Blake Eggleston 
AuthorDate: Mon Nov 9 12:18:24 2020 -0800

Merge branch 'cassandra-3.0' into cassandra-3.11

 CHANGES.txt|   1 +
 .../cassandra/service/MigrationCoordinator.java| 506 +
 .../apache/cassandra/service/MigrationManager.java | 145 +-
 .../apache/cassandra/service/MigrationTask.java| 115 -
 .../apache/cassandra/service/StorageService.java   |  64 ++-
 .../cassandra/service/StorageServiceMBean.java |   4 +
 .../service/MigrationCoordinatorTest.java  | 319 +
 7 files changed, 883 insertions(+), 271 deletions(-)

diff --cc CHANGES.txt
index e46731b,07ee9af..b81b0c8
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -3,14 -3,7 +3,15 @@@ Merged from 3.0
   * Fix invalid cell value skipping when reading from disk (CASSANDRA-16223)
   * Prevent invoking enable/disable gossip when not in NORMAL (CASSANDRA-16146)
  
 -3.0.23:
 +3.11.9
 + * Synchronize Keyspace instance store/clear (CASSANDRA-16210)
 + * Fix ColumnFilter to avoid querying cells of unselected complex columns 
(CASSANDRA-15977)
 + * Fix memory leak in CompressedChunkReader (CASSANDRA-15880)
 + * Don't attempt value skipping with mixed version cluster (CASSANDRA-15833)
 + * Avoid failing compactions with very large partitions (CASSANDRA-15164)
 + * Make sure LCS handles duplicate sstable added/removed notifications 
correctly (CASSANDRA-14103)
 +Merged from 3.0:
++ * Wait for schema agreement when bootstrapping (CASSANDRA-15158)
   * Fix OOM when terminating repair session (CASSANDRA-15902)
   * Avoid marking shutting down nodes as up after receiving gossip shutdown 
message (CASSANDRA-16094)
   * Check SSTables for latest version before dropping compact storage 
(CASSANDRA-16063)
diff --cc src/java/org/apache/cassandra/service/MigrationCoordinator.java
index 000,18ffdbb..3730f62
mode 00,100644..100644
--- a/src/java/org/apache/cassandra/service/MigrationCoordinator.java
+++ b/src/java/org/apache/cassandra/service/MigrationCoordinator.java
@@@ -1,0 -1,501 +1,506 @@@
+ /*
+  * Licensed to the Apache Software Foundation (ASF) under one
+  * or more contributor license agreements.  See the NOTICE file
+  * distributed with this work for additional information
+  * regarding copyright ownership.  The ASF licenses this file
+  * to you under the Apache License, Version 2.0 (the
+  * "License"); you may not use this file except in compliance
+  * with the License.  You may obtain a copy of the License at
+  *
+  * http://www.apache.org/licenses/LICENSE-2.0
+  *
+  * Unless required by applicable law or agreed to in writing, software
+  * distributed under the License is distributed on an "AS IS" BASIS,
+  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  * See the License for the specific language governing permissions and
+  * limitations under the License.
+  */
+ 
+ package org.apache.cassandra.service;
+ 
+ import java.lang.management.ManagementFactory;
+ import java.lang.management.RuntimeMXBean;
+ import java.net.InetAddress;
+ import java.util.ArrayDeque;
+ import java.util.ArrayList;
+ import java.util.Collection;
+ import java.util.Deque;
+ import java.util.HashMap;
+ import java.util.List;
+ import java.util.Map;
+ import java.util.Set;
+ import java.util.UUID;
+ import java.util.concurrent.Future;
+ import java.util.concurrent.FutureTask;
+ import java.util.concurrent.TimeUnit;
+ 
+ import com.google.common.annotations.VisibleForTesting;
+ import com.google.common.collect.ImmutableSet;
+ import com.google.common.collect.Sets;
+ import com.google.common.util.concurrent.Futures;
+ import org.slf4j.Logger;
+ import org.slf4j.LoggerFactory;
+ 
+ import org.apache.cassandra.concurrent.LocalAwareExecutorService;
+ import org.apache.cassandra.concurrent.ScheduledExecutors;
+ import org.apache.cassandra.concurrent.Stage;
+ import org.apache.cassandra.concurrent.StageManager;
+ import org.apache.cassandra.config.Schema;
+ import org.apache.cassandra.db.Mutation;
++import org.apache.cassandra.exceptions.RequestFailureReason;
+ import org.apache.cassandra.gms.ApplicationState;
+ import org.apache.cassandra.gms.EndpointState;
+ import org.apache.cassandra.gms.FailureDetector;
+ import org.apache.cassandra.gms.Gossiper;
++import org.apache.cassandra.gms.IEndpointStateChangeSubscriber;
+ import org.apache.cassandra.gms.VersionedValue;
+ import org.apache.cassandra.net.IAsyncCallbackWithFailure;
+ import org.apache.cassandra.net.MessageIn;
+ import org.apache.cassandra.net.MessageOut;
+ import org.apache.cassandra.net.MessagingService;
+ import org.apache.cassandra.schema.SchemaKeyspace;
+ import org.apache.cassandra.utils.FBUtilities;
+ import org.apache.cas

[cassandra] branch cassandra-3.11 updated (94f940c -> fe70155)

[bdeggleston]

This is an automated email from the ASF dual-hosted git repository.

bdeggleston pushed a change to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from 94f940c  Merge branch 'cassandra-3.0' into cassandra-3.11
 new a960d87  add schema coordinator
 new 0845008  Wait for schema agreement when bootstrapping
 new fe70155  Merge branch 'cassandra-3.0' into cassandra-3.11

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|   1 +
 .../cassandra/service/MigrationCoordinator.java| 506 +
 .../apache/cassandra/service/MigrationManager.java | 145 +-
 .../apache/cassandra/service/MigrationTask.java| 115 -
 .../apache/cassandra/service/StorageService.java   |  64 ++-
 .../cassandra/service/StorageServiceMBean.java |   4 +
 .../service/MigrationCoordinatorTest.java  | 319 +
 7 files changed, 883 insertions(+), 271 deletions(-)
 create mode 100644 
src/java/org/apache/cassandra/service/MigrationCoordinator.java
 delete mode 100644 src/java/org/apache/cassandra/service/MigrationTask.java
 create mode 100644 
test/unit/org/apache/cassandra/service/MigrationCoordinatorTest.java


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch trunk updated (85f69cf -> 159e021)

[bdeggleston]

This is an automated email from the ASF dual-hosted git repository.

bdeggleston pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from 85f69cf  Add a ratelimiter to snapshot creation and deletion
 new a960d87  add schema coordinator
 new 0845008  Wait for schema agreement when bootstrapping
 new fe70155  Merge branch 'cassandra-3.0' into cassandra-3.11
 new 159e021  Merge branch 'cassandra-3.11' into trunk

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|   1 +
 .../cassandra/schema/MigrationCoordinator.java | 516 +
 .../apache/cassandra/schema/MigrationManager.java  | 149 +-
 .../org/apache/cassandra/schema/MigrationTask.java | 111 -
 src/java/org/apache/cassandra/schema/Schema.java   |   2 +-
 .../cassandra/schema/SchemaMigrationEvent.java |   7 +-
 .../apache/cassandra/service/StorageService.java   |  63 ++-
 .../cassandra/service/StorageServiceMBean.java |   7 +
 .../cassandra/distributed/action/GossipHelper.java |  10 +-
 .../cassandra/schema/MigrationCoordinatorTest.java | 319 +
 10 files changed, 908 insertions(+), 277 deletions(-)
 create mode 100644 
src/java/org/apache/cassandra/schema/MigrationCoordinator.java
 delete mode 100644 src/java/org/apache/cassandra/schema/MigrationTask.java
 create mode 100644 
test/unit/org/apache/cassandra/schema/MigrationCoordinatorTest.java


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 01/02: add schema coordinator

[bdeggleston]

This is an automated email from the ASF dual-hosted git repository.

bdeggleston pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit a960d87e05f01000a758d8e7e5a58be57d13eb33
Author: Blake Eggleston 
AuthorDate: Mon May 18 23:02:51 2020 +0200

add schema coordinator

Co-authored-by: Stefan Miklosovic 
---
 src/java/org/apache/cassandra/config/Schema.java   |   8 +
 .../cassandra/service/MigrationCoordinator.java| 501 +
 .../apache/cassandra/service/MigrationManager.java | 124 +
 .../apache/cassandra/service/MigrationTask.java| 116 -
 .../apache/cassandra/service/StorageService.java   |  69 ++-
 .../cassandra/service/StorageServiceMBean.java |   4 +
 .../service/MigrationCoordinatorTest.java  | 319 +
 7 files changed, 895 insertions(+), 246 deletions(-)

diff --git a/src/java/org/apache/cassandra/config/Schema.java 
b/src/java/org/apache/cassandra/config/Schema.java
index 6d91d8d..2d50b32 100644
--- a/src/java/org/apache/cassandra/config/Schema.java
+++ b/src/java/org/apache/cassandra/config/Schema.java
@@ -574,6 +574,14 @@ public class Schema
 }
 
 /**
+ * Checks whether the current schema is empty.
+ */
+public boolean isEmpty()
+{
+return emptyVersion.equals(version);
+}
+
+/**
  * Read schema from system keyspace and calculate MD5 digest of every row, 
resulting digest
  * will be converted into UUID which would act as content-based version of 
the schema.
  */
diff --git a/src/java/org/apache/cassandra/service/MigrationCoordinator.java 
b/src/java/org/apache/cassandra/service/MigrationCoordinator.java
new file mode 100644
index 000..18ffdbb
--- /dev/null
+++ b/src/java/org/apache/cassandra/service/MigrationCoordinator.java
@@ -0,0 +1,501 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.RuntimeMXBean;
+import java.net.InetAddress;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.Future;
+import java.util.concurrent.FutureTask;
+import java.util.concurrent.TimeUnit;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Sets;
+import com.google.common.util.concurrent.Futures;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.concurrent.LocalAwareExecutorService;
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.concurrent.Stage;
+import org.apache.cassandra.concurrent.StageManager;
+import org.apache.cassandra.config.Schema;
+import org.apache.cassandra.db.Mutation;
+import org.apache.cassandra.gms.ApplicationState;
+import org.apache.cassandra.gms.EndpointState;
+import org.apache.cassandra.gms.FailureDetector;
+import org.apache.cassandra.gms.Gossiper;
+import org.apache.cassandra.gms.VersionedValue;
+import org.apache.cassandra.net.IAsyncCallbackWithFailure;
+import org.apache.cassandra.net.MessageIn;
+import org.apache.cassandra.net.MessageOut;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.schema.SchemaKeyspace;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.concurrent.WaitQueue;
+
+public class MigrationCoordinator
+{
+private static final Logger logger = 
LoggerFactory.getLogger(MigrationCoordinator.class);
+private static final RuntimeMXBean runtimeMXBean = 
ManagementFactory.getRuntimeMXBean();
+private static final Future FINISHED_FUTURE = 
Futures.immediateFuture(null);
+
+private static final int MIGRATION_DELAY_IN_MS = 6;
+private static final int MAX_OUTSTANDING_VERSION_REQUESTS = 3;
+
+public static final MigrationCoordinator instance = new 
MigrationCoordinator();
+
+static class VersionInfo
+{
+final UUID version;
+
+final Set endpoi

[cassandra] branch cassandra-3.0 updated (e5ab8c1 -> 0845008)

[bdeggleston]

This is an automated email from the ASF dual-hosted git repository.

bdeggleston pushed a change to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from e5ab8c1  Fix tests broken by CASSANDRA-16146
 new a960d87  add schema coordinator
 new 0845008  Wait for schema agreement when bootstrapping

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|   1 +
 src/java/org/apache/cassandra/config/Schema.java   |   8 +
 .../cassandra/service/MigrationCoordinator.java| 501 +
 .../apache/cassandra/service/MigrationManager.java | 124 +
 .../apache/cassandra/service/MigrationTask.java| 116 -
 .../apache/cassandra/service/StorageService.java   |  69 ++-
 .../cassandra/service/StorageServiceMBean.java |   4 +
 .../service/MigrationCoordinatorTest.java  | 319 +
 8 files changed, 896 insertions(+), 246 deletions(-)
 create mode 100644 
src/java/org/apache/cassandra/service/MigrationCoordinator.java
 delete mode 100644 src/java/org/apache/cassandra/service/MigrationTask.java
 create mode 100644 
test/unit/org/apache/cassandra/service/MigrationCoordinatorTest.java


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 02/02: Wait for schema agreement when bootstrapping

[bdeggleston]

This is an automated email from the ASF dual-hosted git repository.

bdeggleston pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 08450080614250a8bfaba23dbca741a4d9315e3c
Author: Blake Eggleston 
AuthorDate: Mon Nov 9 12:15:21 2020 -0800

Wait for schema agreement when bootstrapping

Patch by Blake Eggleston & Stefan Miklosovic;
Reviewed by Aleksey Yeschenko and Stefan Miklosovic for CASSANDRA-15158

Co-authored-by: Stefan Miklosovic 
---
 CHANGES.txt | 1 +
 1 file changed, 1 insertion(+)

diff --git a/CHANGES.txt b/CHANGES.txt
index 696e50f..07ee9af 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.24:
+ * Wait for schema agreement when bootstrapping (CASSANDRA-15158)
  * Fix invalid cell value skipping when reading from disk (CASSANDRA-16223)
  * Prevent invoking enable/disable gossip when not in NORMAL (CASSANDRA-16146)
 


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16253) Fix documentation to reflect real defaults for commit logs

[Michael Semb Wever (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Semb Wever updated CASSANDRA-16253:
---
Status: Ready to Commit  (was: Review In Progress)

> Fix documentation to reflect real defaults for commit logs
> --
>
> Key: CASSANDRA-16253
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16253
> Project: Cassandra
>  Issue Type: Task
>  Components: Documentation/Website
>Reporter: Alain RODRIGUEZ
>Assignee: Michael Semb Wever
>Priority: Normal
> Fix For: 3.11.x, 4.0.x
>
>
> About the documentation on commit logs:
>  * Commit logs default is every 10 sec - *periodic*, not batch 
> [https://github.com/apache/cassandra/blob/cassandra-3.11.8/conf/cassandra.yaml#L385-L386]
>  (same in 3.11.9 or trunk). It can lead to data losses but still makes sense 
> to me because fsync-ing every write is very expensive, and (even in batches) 
> should be reserved for the super paranoid where handling multiple power-out 
> (or sudden disk failures) simultaneously on multiple nodes needs to be 
> handled (as [~mck] explained to me).
>   * Nonetheless, apart from the theory behind it, in a more practical 
> perspective, *the doc is wrong* and says *Default Value: batch* which mislead 
> me and this could happen to more people: 
> [https://cassandra.apache.org/doc/latest/architecture/storage_engine.html#commit-log]
>  
> I think this is important enough for us to quickly fix the documentation in 
> trunk. As this information is fully missing in the 3.11 branch documentation, 
> I suggest we add it as well on that branch (based on information about commit 
> logs from trunk).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16256) jvm dtest is strict on properties which causes upgrade tests to fail

[David Capwell (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228772#comment-17228772
 ] 

David Capwell commented on CASSANDRA-16256:
---

patch working locally, will trigger CI 

{code}
testclasslist:
 [echo] Number of test runners: 1
[mkdir] Created dir: 
/Users/davidcapwell/src/github/apache/cassandra-trunk/build/test/cassandra
[mkdir] Created dir: 
/Users/davidcapwell/src/github/apache/cassandra-trunk/build/test/output
[junit-timeout] Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
[junit-timeout] Testsuite: 
org.apache.cassandra.distributed.upgrade.CompactStorage3to4UpgradeTest
[junit-timeout] Testsuite: 
org.apache.cassandra.distributed.upgrade.CompactStorage3to4UpgradeTest Tests 
run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.934 sec
[junit-timeout]

BUILD SUCCESSFUL
Total time: 35 seconds
{code}

> jvm dtest is strict on properties which causes upgrade tests to fail
> 
>
> Key: CASSANDRA-16256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16256
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest/java
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> {code}
> ConfigurationException: Invalid yaml. Please remove properties 
> [cdc_raw_directory, diagnostic_events_enabled] from your cassandra.yaml
> {code}
> This was done in CASSANDRA-16152 to make sure we don’t have typo’s in 
> configs, but this makes upgrades harder as InstanceConfig is used and this 
> isn’t version aware.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra-dtest] branch trunk updated: CASSANDRA-15158 give node a bit more time to come up

[bdeggleston]

This is an automated email from the ASF dual-hosted git repository.

bdeggleston pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra-dtest.git


The following commit(s) were added to refs/heads/trunk by this push:
 new 73fd2db  CASSANDRA-15158 give node a bit more time to come up
73fd2db is described below

commit 73fd2db54ae71c204184814bb0760e19fe3e7e57
Author: Blake Eggleston 
AuthorDate: Fri Oct 9 10:12:01 2020 -0700

CASSANDRA-15158 give node a bit more time to come up
---
 bootstrap_test.py | 18 +++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/bootstrap_test.py b/bootstrap_test.py
index 1cda916..51ee32e 100644
--- a/bootstrap_test.py
+++ b/bootstrap_test.py
@@ -482,11 +482,17 @@ class TestBootstrap(Tester):
 
 node3 = new_node(cluster, data_center='dc2')
 node3.start(jvm_args=["-Dcassandra.write_survey=true"], 
no_wait=True)
-time.sleep(5)
 
+node3_seen = False
+for _ in range(30):  # give node3 up to 30 seconds to start
+ntout = node1.nodetool('status').stdout
+if re.search(r'UJ\s+' + node3.ip_addr, ntout):
+node3_seen = True
+break
+time.sleep(1)
+
+assert node3_seen, "expected {} in 
status:\n{}".format(node3.ip_addr, ntout)
 
-ntout = node1.nodetool('status').stdout
-assert re.search(r'UJ\s+' + node3.ip_addr, ntout), ntout
 out, err, _ = node1.stress(['user', 'profile=' + 
stress_config.name, 'ops(insert=1)',
 'n=10k', 'no-warmup', 
'cl=LOCAL_QUORUM',
 '-rate', 'threads=10',
@@ -768,6 +774,12 @@ class TestBootstrap(Tester):
 node2 = new_node(cluster)
 node2.start()
 
+for _ in range(30):  # wait until node2 shows up
+ntout = node1.nodetool('status').stdout
+if re.search(r'UJ\s+' + node2.ip_addr, ntout):
+break
+time.sleep(0.1)
+
 node3 = new_node(cluster, remote_debug_port='2003')
 try:
 node3.start(wait_other_notice=False, verbose=False)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16256) jvm dtest is strict on properties which causes upgrade tests to fail

[David Capwell (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Capwell updated CASSANDRA-16256:
--
Test and Documentation Plan: testing
 Status: Patch Available  (was: Open)

> jvm dtest is strict on properties which causes upgrade tests to fail
> 
>
> Key: CASSANDRA-16256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16256
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest/java
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> {code}
> ConfigurationException: Invalid yaml. Please remove properties 
> [cdc_raw_directory, diagnostic_events_enabled] from your cassandra.yaml
> {code}
> This was done in CASSANDRA-16152 to make sure we don’t have typo’s in 
> configs, but this makes upgrades harder as InstanceConfig is used and this 
> isn’t version aware.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16256) jvm dtest is strict on properties which causes upgrade tests to fail

[Caleb Rackliffe (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Caleb Rackliffe updated CASSANDRA-16256:

Reviewers: Caleb Rackliffe

> jvm dtest is strict on properties which causes upgrade tests to fail
> 
>
> Key: CASSANDRA-16256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16256
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest/java
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>
> {code}
> ConfigurationException: Invalid yaml. Please remove properties 
> [cdc_raw_directory, diagnostic_events_enabled] from your cassandra.yaml
> {code}
> This was done in CASSANDRA-16152 to make sure we don’t have typo’s in 
> configs, but this makes upgrades harder as InstanceConfig is used and this 
> isn’t version aware.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16256) jvm dtest is strict on properties which causes upgrade tests to fail

[David Capwell (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Capwell updated CASSANDRA-16256:
--
 Bug Category: Parent values: Correctness(12982)Level 1 values: Test 
Failure(12990)
   Complexity: Low Hanging Fruit
Discovered By: Unit Test
 Severity: Normal
   Status: Open  (was: Triage Needed)

> jvm dtest is strict on properties which causes upgrade tests to fail
> 
>
> Key: CASSANDRA-16256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16256
> Project: Cassandra
>  Issue Type: Bug
>  Components: Test/dtest/java
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>
> {code}
> ConfigurationException: Invalid yaml. Please remove properties 
> [cdc_raw_directory, diagnostic_events_enabled] from your cassandra.yaml
> {code}
> This was done in CASSANDRA-16152 to make sure we don’t have typo’s in 
> configs, but this makes upgrades harder as InstanceConfig is used and this 
> isn’t version aware.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-16256) jvm dtest is strict on properties which causes upgrade tests to fail

[David Capwell (Jira)]

David Capwell created CASSANDRA-16256:
-

 Summary: jvm dtest is strict on properties which causes upgrade 
tests to fail
 Key: CASSANDRA-16256
 URL: https://issues.apache.org/jira/browse/CASSANDRA-16256
 Project: Cassandra
  Issue Type: Bug
  Components: Test/dtest/java
Reporter: David Capwell
Assignee: David Capwell


{code}
ConfigurationException: Invalid yaml. Please remove properties 
[cdc_raw_directory, diagnostic_events_enabled] from your cassandra.yaml
{code}

This was done in CASSANDRA-16152 to make sure we don’t have typo’s in configs, 
but this makes upgrades harder as InstanceConfig is used and this isn’t version 
aware.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16213) Cannot replace_address /X because it doesn't exist in gossip

[David Capwell (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228747#comment-17228747
 ] 

David Capwell commented on CASSANDRA-16213:
---

Thanks for the feedback

bq. not required as endpoint states with heartbeat version -1 need not be sent 
by examineGossiper

I think that was important for assassinate, so will try to look closer at your 
patch and how it works with assassinate

bq. HostReplacementOfDowedClusterTest.hostReplacementOfDeadNode tests an 
orderly shutdown, ... Can we maybe also test with an abrupt shutdown, that is 
when the shutdown state is not broadcast and the node to be replaced is on 
NORMAL state?

Makes a lot of sense, I can try working on this test (I assume I can just block 
gossip messages as I can't kill -9 in jvm-dtest).

> Cannot replace_address /X because it doesn't exist in gossip
> 
>
> Key: CASSANDRA-16213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16213
> Project: Cassandra
>  Issue Type: Bug
>  Components: Cluster/Gossip, Cluster/Membership
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Fix For: 4.0-beta
>
>
> We see this exception around nodes crashing and trying to do a host 
> replacement; this error appears to be correlated around multiple node 
> failures.
> A simplified case to trigger this is the following
> *) Have a N node cluster
> *) Shutdown all N nodes
> *) Bring up N-1 nodes (at least 1 seed, else replace seed)
> *) Host replace the N-1th node -> this will fail with the above
> The reason this happens is that the N-1th node isn’t gossiping anymore, and 
> the existing nodes do not have its details in gossip (but have the details in 
> the peers table), so the host replacement fails as the node isn’t known in 
> gossip.
> This affects all versions (tested 3.0 and trunk, assume 2.2 as well)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16225) Followup CASSANDRA-14554

[Brandon Williams (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-16225:
-
Reviewers: Brandon Williams, Brandon Williams  (was: Brandon Williams)
   Brandon Williams, Brandon Williams
   Status: Review In Progress  (was: Patch Available)

> Followup CASSANDRA-14554
> 
>
> Key: CASSANDRA-16225
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16225
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Internode
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
>
> As per [~stefania]'s advice, additional synchronization should be added to   
> LogTransaction.java. Without synchronization, we could have corrupted txn log 
> files with JBOD.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16241) ArrayClustering does not properly handle null clustering key elements left over from tables created WITH COMPACT STORAGE

[Caleb Rackliffe (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228678#comment-17228678
 ] 

Caleb Rackliffe commented on CASSANDRA-16241:
-

[~yifanc] I've rebased and started the JVM upgrade tests again here: 
https://app.circleci.com/pipelines/github/maedhroz/cassandra/142/workflows/e657431c-1e91-45e8-b8f4-321f89408b5f

> ArrayClustering does not properly handle null clustering key elements left 
> over from tables created WITH COMPACT STORAGE
> 
>
> Key: CASSANDRA-16241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16241
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Compaction
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
>  Labels: compaction
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The only way we can produce null clustering key elements is leaving them 
> empty on insert while a table is still compact. If we subsequently DROP 
> COMPACT STORAGE, those null elements linger, and {{ArrayClustering}} does not 
> handle them appropriately on compaction. 
> If you run the test 
> [here|https://github.com/maedhroz/cassandra/commit/e247b7868cae383168153bbe8bbbaa47a660f76b],
>  you should be able to observe an exception that looks roughly like this:
> {noformat}
> java.lang.NullPointerException
>   at java.base/java.nio.ByteBuffer.wrap(ByteBuffer.java:422)
>   at 
> org.apache.cassandra.db.AbstractArrayClusteringPrefix.getBufferArray(AbstractArrayClusteringPrefix.java:45)
>   at 
> org.apache.cassandra.io.sstable.metadata.MetadataCollector.finalizeMetadata(MetadataCollector.java:246)
>   at 
> org.apache.cassandra.io.sstable.format.SSTableWriter.finalizeMetadata(SSTableWriter.java:315)
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$200(BigTableWriter.java:52)
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:415)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)
>   at 
> org.apache.cassandra.io.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:283)
>   at 
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:380)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)
>   at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:118)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)
>   at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:179)
>   at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:128)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:225)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> {noformat}
> There are already numerous places where we respect the fact that clustering 
> elements may be null, so this should be pretty straightforward to fix, and 
> the tests that accompany it will probably be more complex than the fix itself.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-16225) Followup CASSANDRA-14554

[Ekaterina Dimitrova (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228655#comment-17228655
 ] 

Ekaterina Dimitrova edited comment on CASSANDRA-16225 at 11/9/20, 3:15 PM:
---

Transaction files on JBOD may become out of sync, that is lines are not in the 
same order.

To prevent that we synchronize LogTransaction.java
 [trunk 
|https://github.com/ekaterinadimitrova2/cassandra/commit/7d4007e942202809925b0d715bc519fd8f7d5ba0]
 | [JAVA8 CI 
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/483/workflows/0fc7c223-7b2f-450a-9697-36cb2f295ad1]
 | [JAVA11 CI 
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/483/workflows/2e4ca8af-217f-4778-8da9-064b24c2fbf4]

[3.11 
|https://github.com/ekaterinadimitrova2/cassandra/commit/bbf11d406dc88bc646cad45c4dfb8cf7135ff72c]
 | [CI 
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/482/workflows/5b9f1098-9b79-4477-9ca9-ef9ff8c57e84]
 [3.0 
|https://github.com/ekaterinadimitrova2/cassandra/commit/6b72d5ebe82e7ed031708a1ad4cd45f6c76ca525]
 | [CI 
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/481/workflows/3b4e63f5-5f09-4520-8e0c-764410febec3]

No new failures.

[~brandon.williams], can you, please, review?


was (Author: e.dimitrova):
Transaction files on JBOD may become out of sync, that is lines are not in the 
same order.

To prevent that we synchronize LogTransaction.java
 [trunk 
|https://github.com/ekaterinadimitrova2/cassandra/commit/7d4007e942202809925b0d715bc519fd8f7d5ba0]
 | [JAVA8 CI 
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/483/workflows/0fc7c223-7b2f-450a-9697-36cb2f295ad1]
 | [JAVA11CI 
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/483/workflows/2e4ca8af-217f-4778-8da9-064b24c2fbf4]

[3.11 
|https://github.com/ekaterinadimitrova2/cassandra/commit/bbf11d406dc88bc646cad45c4dfb8cf7135ff72c]
 | [CI 
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/482/workflows/5b9f1098-9b79-4477-9ca9-ef9ff8c57e84]
 [3.0 
|https://github.com/ekaterinadimitrova2/cassandra/commit/6b72d5ebe82e7ed031708a1ad4cd45f6c76ca525]
 | [CI 
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/481/workflows/3b4e63f5-5f09-4520-8e0c-764410febec3]

No new failures.

[~brandon.williams], can you, please, review?

> Followup CASSANDRA-14554
> 
>
> Key: CASSANDRA-16225
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16225
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Internode
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
>
> As per [~stefania]'s advice, additional synchronization should be added to   
> LogTransaction.java. Without synchronization, we could have corrupted txn log 
> files with JBOD.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-16225) Followup CASSANDRA-14554

[Ekaterina Dimitrova (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228655#comment-17228655
 ] 

Ekaterina Dimitrova edited comment on CASSANDRA-16225 at 11/9/20, 3:11 PM:
---

Transaction files on JBOD may become out of sync, that is lines are not in the 
same order.

To prevent that we synchronize LogTransaction.java
 [trunk 
|https://github.com/ekaterinadimitrova2/cassandra/commit/7d4007e942202809925b0d715bc519fd8f7d5ba0]
 | [JAVA8 CI 
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/483/workflows/0fc7c223-7b2f-450a-9697-36cb2f295ad1]
 | [JAVA11CI 
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/483/workflows/2e4ca8af-217f-4778-8da9-064b24c2fbf4]

[3.11 
|https://github.com/ekaterinadimitrova2/cassandra/commit/bbf11d406dc88bc646cad45c4dfb8cf7135ff72c]
 | [CI 
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/482/workflows/5b9f1098-9b79-4477-9ca9-ef9ff8c57e84]
 [3.0 
|https://github.com/ekaterinadimitrova2/cassandra/commit/6b72d5ebe82e7ed031708a1ad4cd45f6c76ca525]
 | [CI 
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/481/workflows/3b4e63f5-5f09-4520-8e0c-764410febec3]

No new failures.

[~brandon.williams], can you, please, review?


was (Author: e.dimitrova):
Transaction files on JBOD may become out of sync, that is lines are not in the 
same order
 [trunk 
|https://github.com/ekaterinadimitrova2/cassandra/commit/7d4007e942202809925b0d715bc519fd8f7d5ba0]
 | [JAVA8 CI 
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/483/workflows/0fc7c223-7b2f-450a-9697-36cb2f295ad1]
 | [JAVA11CI 
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/483/workflows/2e4ca8af-217f-4778-8da9-064b24c2fbf4]

[3.11 
|https://github.com/ekaterinadimitrova2/cassandra/commit/bbf11d406dc88bc646cad45c4dfb8cf7135ff72c]
 | [CI 
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/482/workflows/5b9f1098-9b79-4477-9ca9-ef9ff8c57e84]
 [3.0 
|https://github.com/ekaterinadimitrova2/cassandra/commit/6b72d5ebe82e7ed031708a1ad4cd45f6c76ca525]
 | [CI 
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/481/workflows/3b4e63f5-5f09-4520-8e0c-764410febec3]

No new failures.


 [~brandon.williams], can you, please, review?

> Followup CASSANDRA-14554
> 
>
> Key: CASSANDRA-16225
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16225
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Internode
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
>
> As per [~stefania]'s advice, additional synchronization should be added to   
> LogTransaction.java. Without synchronization, we could have corrupted txn log 
> files with JBOD.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16225) Followup CASSANDRA-14554

[Ekaterina Dimitrova (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228655#comment-17228655
 ] 

Ekaterina Dimitrova commented on CASSANDRA-16225:
-

Transaction files on JBOD may become out of sync, that is lines are not in the 
same order
 [trunk 
|https://github.com/ekaterinadimitrova2/cassandra/commit/7d4007e942202809925b0d715bc519fd8f7d5ba0]
 | [JAVA8 CI 
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/483/workflows/0fc7c223-7b2f-450a-9697-36cb2f295ad1]
 | [JAVA11CI 
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/483/workflows/2e4ca8af-217f-4778-8da9-064b24c2fbf4]

[3.11 
|https://github.com/ekaterinadimitrova2/cassandra/commit/bbf11d406dc88bc646cad45c4dfb8cf7135ff72c]
 | [CI 
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/482/workflows/5b9f1098-9b79-4477-9ca9-ef9ff8c57e84]
 [3.0 
|https://github.com/ekaterinadimitrova2/cassandra/commit/6b72d5ebe82e7ed031708a1ad4cd45f6c76ca525]
 | [CI 
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/481/workflows/3b4e63f5-5f09-4520-8e0c-764410febec3]

No new failures.


 [~brandon.williams], can you, please, review?

> Followup CASSANDRA-14554
> 
>
> Key: CASSANDRA-16225
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16225
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Internode
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
>
> As per [~stefania]'s advice, additional synchronization should be added to   
> LogTransaction.java. Without synchronization, we could have corrupted txn log 
> files with JBOD.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16225) Followup CASSANDRA-14554

[Ekaterina Dimitrova (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ekaterina Dimitrova updated CASSANDRA-16225:

Test and Documentation Plan: 
https://issues.apache.org/jira/browse/CASSANDRA-16225?focusedCommentId=17228655&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17228655
 Status: Patch Available  (was: In Progress)

> Followup CASSANDRA-14554
> 
>
> Key: CASSANDRA-16225
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16225
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Internode
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
>
> As per [~stefania]'s advice, additional synchronization should be added to   
> LogTransaction.java. Without synchronization, we could have corrupted txn log 
> files with JBOD.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16255) Update jctools dependency

[Marcus Eriksson (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-16255:

Change Category: Quality Assurance
 Complexity: Low Hanging Fruit
Component/s: Local/Other
  Fix Version/s: 4.0-beta4
 Status: Open  (was: Triage Needed)

> Update jctools dependency
> -
>
> Key: CASSANDRA-16255
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16255
> Project: Cassandra
>  Issue Type: Task
>  Components: Local/Other
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 4.0-beta4
>
>
> CASSANDRA-15880 started using {{MpmcArrayQueue}} from jctools - before that 
> we only used it in cassandra-stress, we should probably update the dependency 
> as jctools-1.2.1 is more than 4 years old



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-16255) Update jctools dependency

[Marcus Eriksson (Jira)]

Marcus Eriksson created CASSANDRA-16255:
---

 Summary: Update jctools dependency
 Key: CASSANDRA-16255
 URL: https://issues.apache.org/jira/browse/CASSANDRA-16255
 Project: Cassandra
  Issue Type: Task
Reporter: Marcus Eriksson
Assignee: Marcus Eriksson


CASSANDRA-15880 started using {{MpmcArrayQueue}} from jctools - before that we 
only used it in cassandra-stress, we should probably update the dependency as 
jctools-1.2.1 is more than 4 years old



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16254) Make sure OOM errors are rethrown on truncation failure

[Marcus Eriksson (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-16254:

Test and Documentation Plan: new in-jvm dtest, cci run
 Status: Patch Available  (was: Open)

patch: https://github.com/krummas/cassandra/commits/marcuse/16254
cci: 
https://app.circleci.com/pipelines/github/krummas/cassandra?branch=marcuse%2F16254

> Make sure OOM errors are rethrown on truncation failure
> ---
>
> Key: CASSANDRA-16254
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16254
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Other
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 4.0-beta4
>
>
> Seems we don't rethrow OOM errors in [truncation verb 
> handler|https://github.com/apache/cassandra/blob/609876275738589fdfb9a3e20cb2f594aa404037/src/java/org/apache/cassandra/db/TruncateVerbHandler.java#L44]
> We also send both success and failure messages if the failure reason is not 
> an FSError



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16254) Make sure OOM errors are rethrown on truncation failure

[Marcus Eriksson (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-16254:

 Bug Category: Parent values: Degradation(12984)Level 1 values: Other 
Exception(12998)
   Complexity: Low Hanging Fruit
Discovered By: Code Inspection
Fix Version/s: 4.0-beta4
 Severity: Low
   Status: Open  (was: Triage Needed)

> Make sure OOM errors are rethrown on truncation failure
> ---
>
> Key: CASSANDRA-16254
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16254
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Other
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 4.0-beta4
>
>
> Seems we don't rethrow OOM errors in [truncation verb 
> handler|https://github.com/apache/cassandra/blob/609876275738589fdfb9a3e20cb2f594aa404037/src/java/org/apache/cassandra/db/TruncateVerbHandler.java#L44]
> We also send both success and failure messages if the failure reason is not 
> an FSError



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-16254) Make sure OOM errors are rethrown on truncation failure

[Marcus Eriksson (Jira)]

Marcus Eriksson created CASSANDRA-16254:
---

 Summary: Make sure OOM errors are rethrown on truncation failure
 Key: CASSANDRA-16254
 URL: https://issues.apache.org/jira/browse/CASSANDRA-16254
 Project: Cassandra
  Issue Type: Bug
  Components: Local/Other
Reporter: Marcus Eriksson
Assignee: Marcus Eriksson


Seems we don't rethrow OOM errors in [truncation verb 
handler|https://github.com/apache/cassandra/blob/609876275738589fdfb9a3e20cb2f594aa404037/src/java/org/apache/cassandra/db/TruncateVerbHandler.java#L44]

We also send both success and failure messages if the failure reason is not an 
FSError



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13019) Improve clearsnapshot to delete the snapshot files slowly

[Marcus Eriksson (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-13019:

  Fix Version/s: (was: 4.0-beta)
 (was: 4.x)
 4.0-beta4
Source Control Link: 
https://github.com/apache/cassandra/commit/85f69cf394a8035280a741b117e7dfc71fb848c3
 Resolution: Fixed
 Status: Resolved  (was: Ready to Commit)

committed, thanks

> Improve clearsnapshot to delete the snapshot files slowly 
> --
>
> Key: CASSANDRA-13019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13019
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Legacy/Core
>Reporter: Dikang Gu
>Assignee: Jeff Jirsa
>Priority: Normal
>  Labels: pull-request-available
> Fix For: 4.0-beta4
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> In our environment, we are creating snapshots for backup, after we finish the 
> backup, we are running {{clearsnapshot}} to delete the snapshot files. At 
> that time we may have thousands of files to delete, and it's causing sudden 
> disk usage spike. As a result, we are experiencing a spike of drop messages 
> from Cassandra.
> I think we should implement something like {{slowrm}} to delete the snapshot 
> files slowly, avoid the sudden disk usage spike.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch trunk updated: Add a ratelimiter to snapshot creation and deletion

[marcuse]

This is an automated email from the ASF dual-hosted git repository.

marcuse pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/trunk by this push:
 new 85f69cf  Add a ratelimiter to snapshot creation and deletion
85f69cf is described below

commit 85f69cf394a8035280a741b117e7dfc71fb848c3
Author: Jeff Jirsa 
AuthorDate: Sat Nov 9 21:01:37 2019 -0800

Add a ratelimiter to snapshot creation and deletion

Patch by Jeff Jirsa; Reviewed by Aleksey Yeschenko, Chris Lohfink,
maxwellguo for CASSANDRA-13019
---
 CHANGES.txt|  1 +
 NEWS.txt   |  1 +
 conf/cassandra.yaml|  7 
 src/java/org/apache/cassandra/config/Config.java   |  1 +
 .../cassandra/config/DatabaseDescriptor.java   | 22 ++
 .../org/apache/cassandra/db/ColumnFamilyStore.java | 35 
 src/java/org/apache/cassandra/db/Directories.java  |  5 ++-
 src/java/org/apache/cassandra/db/Keyspace.java | 12 --
 .../org/apache/cassandra/db/SystemKeyspace.java|  2 +-
 .../org/apache/cassandra/io/util/FileUtils.java| 47 +++---
 .../apache/cassandra/service/StorageService.java   | 19 -
 .../cassandra/service/StorageServiceMBean.java | 16 
 src/java/org/apache/cassandra/tools/NodeProbe.java | 10 +
 src/java/org/apache/cassandra/tools/NodeTool.java  |  2 +
 .../tools/nodetool/GetSnapshotThrottle.java| 36 +
 .../tools/nodetool/SetSnapshotThrottle.java| 36 +
 16 files changed, 230 insertions(+), 22 deletions(-)

diff --git a/CHANGES.txt b/CHANGES.txt
index 635fa71..f90ffd4 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 4.0-beta4
+ * Add a ratelimiter to snapshot creation and deletion (CASSANDRA-13019)
  * Produce consistent tombstone for reads to avoid digest mistmatch 
(CASSANDRA-15369)
  * Fix SSTableloader issue when restoring a table named backups 
(CASSANDRA-16235)
  * Invalid serialized size for responses caused by increasing message time by 
1ms which caused extra bytes in size calculation (CASSANDRA-16103)
diff --git a/NEWS.txt b/NEWS.txt
index 4b5264f..7d16a97 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -112,6 +112,7 @@ New features
   Added --python option to cqlsh so users can specify the path to their 
chosen Python interpreter.
   See CASSANDRA-10190 for details.
 - Support for server side DESCRIBE statements has been added. See 
CASSANDRA-14825
+- It is now possible to rate limit snapshot creation/clearing. See 
CASSANDRA-13019
 
 Upgrading
 -
diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml
index 37b18f9..11e54ad 100644
--- a/conf/cassandra.yaml
+++ b/conf/cassandra.yaml
@@ -791,6 +791,13 @@ snapshot_before_compaction: false
 # lose data on truncation or drop.
 auto_snapshot: true
 
+# The act of creating or clearing a snapshot involves creating or removing
+# potentially tens of thousands of links, which can cause significant 
performance
+# impact, especially on consumer grade SSDs. A non-zero value here can
+# be used to throttle these links to avoid negative performance impact of
+# taking and clearing snapshots
+snapshot_links_per_second: 0
+
 # Granularity of the collation index of rows within a partition.
 # Increase if your rows are large, or if you have a very large
 # number of rows per partition.  The competing goals are these:
diff --git a/src/java/org/apache/cassandra/config/Config.java 
b/src/java/org/apache/cassandra/config/Config.java
index 26854da..759a41a 100644
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@ -204,6 +204,7 @@ public class Config
 
 public boolean snapshot_before_compaction = false;
 public boolean auto_snapshot = true;
+public volatile long snapshot_links_per_second = 0;
 
 /* if the size of columns or super-columns are more than this, indexing 
will kick in */
 public int column_index_size_in_kb = 64;
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java 
b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index 5fbb220..2be169a 100644
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@ -34,6 +34,7 @@ import com.google.common.base.Preconditions;
 import com.google.common.collect.ImmutableSet;
 import com.google.common.primitives.Ints;
 import com.google.common.primitives.Longs;
+import com.google.common.util.concurrent.RateLimiter;
 
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -772,6 +773,9 @@ public class DatabaseDescriptor
 }
 }
 
+if (conf.snapshot_links_per_second < 0)
+throw new ConfigurationException("snapshot_links_per_second must 
be >= 0");
+
 if (conf.max_va

[jira] [Commented] (CASSANDRA-15950) unable to start the cassandra on windows 10 machine

[Brandon Williams (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-15950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228595#comment-17228595
 ] 

Brandon Williams commented on CASSANDRA-15950:
--

8u271  is known to cause problems with SSL sockets not being closed either, I 
would consider it bad and likely the cause here.

> unable to start the cassandra on windows 10 machine
> ---
>
> Key: CASSANDRA-15950
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15950
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Mandar Kulkarni
>Priority: Normal
>
> Os - Windows 10
> Cassandra -  apache-cassandra-3.11.4
> java version - java version "1.8.0_261"
> Java(TM) SE Runtime Environment (build 1.8.0_261-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.261-b12, mixed mode)
> python version - Python 3.8.4
> I am getting below error when starting cassandra -f
> INFO [main] 2020-07-15 17:20:54,566 SigarLibrary.java:44 - Initializing SIGAR 
> library
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x10014ed4, 
> pid=13704, tid=0x25f4
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_261-b12) (build 
> 1.8.0_261-b12)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.261-b12 mixed mode 
> windows-amd64 compressed oops)
> # Problematic frame:
> # C [sigar-amd64-winnt.dll+0x14ed4]
> #
> # Failed to write core dump. Minidumps are not enabled by default on client 
> versions of Windows
> #
> # An error report file with more information is saved as:
> # C:\cass311\apache-cassandra-3.11.4\bin\hs_err_pid13704.log
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.java.com/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15950) unable to start the cassandra on windows 10 machine

[Brandon Williams (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-15950:
-
Status: Triage Needed  (was: Awaiting Feedback)

> unable to start the cassandra on windows 10 machine
> ---
>
> Key: CASSANDRA-15950
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15950
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Mandar Kulkarni
>Priority: Normal
>
> Os - Windows 10
> Cassandra -  apache-cassandra-3.11.4
> java version - java version "1.8.0_261"
> Java(TM) SE Runtime Environment (build 1.8.0_261-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.261-b12, mixed mode)
> python version - Python 3.8.4
> I am getting below error when starting cassandra -f
> INFO [main] 2020-07-15 17:20:54,566 SigarLibrary.java:44 - Initializing SIGAR 
> library
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x10014ed4, 
> pid=13704, tid=0x25f4
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_261-b12) (build 
> 1.8.0_261-b12)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.261-b12 mixed mode 
> windows-amd64 compressed oops)
> # Problematic frame:
> # C [sigar-amd64-winnt.dll+0x14ed4]
> #
> # Failed to write core dump. Minidumps are not enabled by default on client 
> versions of Windows
> #
> # An error report file with more information is saved as:
> # C:\cass311\apache-cassandra-3.11.4\bin\hs_err_pid13704.log
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.java.com/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16252) centos 7

[Brandon Williams (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-16252:
-
Resolution: Invalid
Status: Resolved  (was: Triage Needed)

This JIRA is not a support system.  Please use the mailing list or slack for 
inquires about your problem: https://cassandra.apache.org/community/

> centos 7 
> -
>
> Key: CASSANDRA-16252
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16252
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Karim Chowdhury
>Priority: Normal
>
> We are using centos 7. we have an 10 node Cassandra Cluster and currently we 
> are facing issues with one of them with following error:
>  
> Fatal exception in thread Thread[CompactionExecutor:1,1,main]
>  java.io.IOError: java.io.EOFException
>   
>  or 
>   
>  java.lang.AssertionError: Added column does not sort as the last colum I 
> already tried to remove the data folder, start scrub and repair. Error 
> happens after 24h again.
>   
>  Need root cause and workaround for fixed



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16176) Python dtests should run at debug

[Brandon Williams (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228587#comment-17228587
 ] 

Brandon Williams commented on CASSANDRA-16176:
--

bq. Why not TRACE instead of DEBUG?

I thought that may be a bit heavy, but if not I'm ok with that, as long as it 
is consistent between circle and jenkins.

> Python dtests should run at debug
> -
>
> Key: CASSANDRA-16176
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16176
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CI
>Reporter: Brandon Williams
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> At least in our circle configs, we are specifying --log-level="INFO" which 
> often leaves us with nothing to go on when we have a flaky test that only 
> fails in a certain environment.  In general it would be more useful to always 
> run at DEBUG. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13019) Improve clearsnapshot to delete the snapshot files slowly

[Aleksey Yeschenko (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228557#comment-17228557
 ] 

Aleksey Yeschenko commented on CASSANDRA-13019:
---

Sanity checked.

> Improve clearsnapshot to delete the snapshot files slowly 
> --
>
> Key: CASSANDRA-13019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13019
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Legacy/Core
>Reporter: Dikang Gu
>Assignee: Jeff Jirsa
>Priority: Normal
>  Labels: pull-request-available
> Fix For: 4.x, 4.0-beta
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> In our environment, we are creating snapshots for backup, after we finish the 
> backup, we are running {{clearsnapshot}} to delete the snapshot files. At 
> that time we may have thousands of files to delete, and it's causing sudden 
> disk usage spike. As a result, we are experiencing a spike of drop messages 
> from Cassandra.
> I think we should implement something like {{slowrm}} to delete the snapshot 
> files slowly, avoid the sudden disk usage spike.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-16205) Offline token allocation strategy generator tool

[Michael Semb Wever (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228531#comment-17228531
 ] 

Michael Semb Wever edited comment on CASSANDRA-16205 at 11/9/20, 12:02 PM:
---

Patch updated. Added `--verbose` mode that prints token ownership distributions.
CI run 
[here|https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-devbranch/detail/Cassandra-devbranch/182/pipeline].


was (Author: michaelsembwever):
Patch updated. Added `--verbose` mode that prints token ownership distributions.

> Offline token allocation strategy generator tool
> 
>
> Key: CASSANDRA-16205
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16205
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Local/Config, Local/Scripts
>Reporter: Michael Semb Wever
>Assignee: Michael Semb Wever
>Priority: Normal
>
> A command line tool to generate tokens (using the 
> allocate_tokens_for_local_replication_factor algorithm) for pre-configuration 
> of {{initial_tokens}} in cassandra.yaml.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16205) Offline token allocation strategy generator tool

[Michael Semb Wever (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228531#comment-17228531
 ] 

Michael Semb Wever commented on CASSANDRA-16205:


Patch updated. Added `--verbose` mode that prints token ownership distributions.

> Offline token allocation strategy generator tool
> 
>
> Key: CASSANDRA-16205
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16205
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Local/Config, Local/Scripts
>Reporter: Michael Semb Wever
>Assignee: Michael Semb Wever
>Priority: Normal
>
> A command line tool to generate tokens (using the 
> allocate_tokens_for_local_replication_factor algorithm) for pre-configuration 
> of {{initial_tokens}} in cassandra.yaml.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16253) Fix documentation to reflect real defaults for commit logs

[Alain RODRIGUEZ (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alain RODRIGUEZ updated CASSANDRA-16253:

Description: 
About the documentation on commit logs:
 * Commit logs default is every 10 sec - *periodic*, not batch 
[https://github.com/apache/cassandra/blob/cassandra-3.11.8/conf/cassandra.yaml#L385-L386]
 (same in 3.11.9 or trunk). It can lead to data losses but still makes sense to 
me because fsync-ing every write is very expensive, and (even in batches) 
should be reserved for the super paranoid where handling multiple power-out (or 
sudden disk failures) simultaneously on multiple nodes needs to be handled (as 
[~mck] explained to me).

  * Nonetheless, apart from the theory behind it, in a more practical 
perspective, *the doc is wrong* and says *Default Value: batch* which mislead 
me and this could happen to more people: 
[https://cassandra.apache.org/doc/latest/architecture/storage_engine.html#commit-log]

 

I think this is important enough for us to quickly fix the documentation in 
trunk. As this information is fully missing in the 3.11 branch documentation, I 
suggest we add it as well on that branch (based on information about commit 
logs from trunk).

  was:
About the documentation on commit logs:
 * Commit logs default is every 10 sec - *periodic*, not batch 
[https://github.com/apache/cassandra/blob/cassandra-3.11.8/conf/cassandra.yaml#L385-L386]
 (same in 3.11.9 or trunk). It can lead to data losses but still makes sense to 
me because fsync-ing every write is very expensive, and (even in batches) 
should be reserved for the super paranoid where handling multiple power-out (or 
sudden disk failures) simultaneously on multiple nodes needs to be handled (as 
[~mck] explained to me.

  * Nonetheless, apart from the theory behind it, in a more practical 
perspective, *the doc is wrong* and says *Default Value: batch* which mislead 
me and this could happen to more people: 
[https://cassandra.apache.org/doc/latest/architecture/storage_engine.html#commit-log]

 

I think this is important enough for us to quickly fix the documentation in 
trunk. As this information is fully missing in the 3.11 branch documentation, I 
suggest we add it as well on that branch (based on information about commit 
logs from trunk).


> Fix documentation to reflect real defaults for commit logs
> --
>
> Key: CASSANDRA-16253
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16253
> Project: Cassandra
>  Issue Type: Task
>  Components: Documentation/Website
>Reporter: Alain RODRIGUEZ
>Assignee: Michael Semb Wever
>Priority: Normal
> Fix For: 3.11.x, 4.0.x
>
>
> About the documentation on commit logs:
>  * Commit logs default is every 10 sec - *periodic*, not batch 
> [https://github.com/apache/cassandra/blob/cassandra-3.11.8/conf/cassandra.yaml#L385-L386]
>  (same in 3.11.9 or trunk). It can lead to data losses but still makes sense 
> to me because fsync-ing every write is very expensive, and (even in batches) 
> should be reserved for the super paranoid where handling multiple power-out 
> (or sudden disk failures) simultaneously on multiple nodes needs to be 
> handled (as [~mck] explained to me).
>   * Nonetheless, apart from the theory behind it, in a more practical 
> perspective, *the doc is wrong* and says *Default Value: batch* which mislead 
> me and this could happen to more people: 
> [https://cassandra.apache.org/doc/latest/architecture/storage_engine.html#commit-log]
>  
> I think this is important enough for us to quickly fix the documentation in 
> trunk. As this information is fully missing in the 3.11 branch documentation, 
> I suggest we add it as well on that branch (based on information about commit 
> logs from trunk).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16253) Fix documentation to reflect real defaults for commit logs

[Michael Semb Wever (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Semb Wever updated CASSANDRA-16253:
---
Reviewers: Michael Semb Wever, Michael Semb Wever  (was: Michael Semb Wever)
   Michael Semb Wever, Michael Semb Wever
   Status: Review In Progress  (was: Patch Available)

> Fix documentation to reflect real defaults for commit logs
> --
>
> Key: CASSANDRA-16253
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16253
> Project: Cassandra
>  Issue Type: Task
>  Components: Documentation/Website
>Reporter: Alain RODRIGUEZ
>Assignee: Michael Semb Wever
>Priority: Normal
> Fix For: 3.11.x, 4.0.x
>
>
> About the documentation on commit logs:
>  * Commit logs default is every 10 sec - *periodic*, not batch 
> [https://github.com/apache/cassandra/blob/cassandra-3.11.8/conf/cassandra.yaml#L385-L386]
>  (same in 3.11.9 or trunk). It can lead to data losses but still makes sense 
> to me because fsync-ing every write is very expensive, and (even in batches) 
> should be reserved for the super paranoid where handling multiple power-out 
> (or sudden disk failures) simultaneously on multiple nodes needs to be 
> handled (as [~mck] explained to me.
>   * Nonetheless, apart from the theory behind it, in a more practical 
> perspective, *the doc is wrong* and says *Default Value: batch* which mislead 
> me and this could happen to more people: 
> [https://cassandra.apache.org/doc/latest/architecture/storage_engine.html#commit-log]
>  
> I think this is important enough for us to quickly fix the documentation in 
> trunk. As this information is fully missing in the 3.11 branch documentation, 
> I suggest we add it as well on that branch (based on information about commit 
> logs from trunk).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16253) Fix documentation to reflect real defaults for commit logs

[Michael Semb Wever (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Semb Wever updated CASSANDRA-16253:
---
Test and Documentation Plan: website previewing
 Status: Patch Available  (was: Open)

> Fix documentation to reflect real defaults for commit logs
> --
>
> Key: CASSANDRA-16253
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16253
> Project: Cassandra
>  Issue Type: Task
>  Components: Documentation/Website
>Reporter: Alain RODRIGUEZ
>Assignee: Michael Semb Wever
>Priority: Normal
> Fix For: 3.11.x, 4.0.x
>
>
> About the documentation on commit logs:
>  * Commit logs default is every 10 sec - *periodic*, not batch 
> [https://github.com/apache/cassandra/blob/cassandra-3.11.8/conf/cassandra.yaml#L385-L386]
>  (same in 3.11.9 or trunk). It can lead to data losses but still makes sense 
> to me because fsync-ing every write is very expensive, and (even in batches) 
> should be reserved for the super paranoid where handling multiple power-out 
> (or sudden disk failures) simultaneously on multiple nodes needs to be 
> handled (as [~mck] explained to me.
>   * Nonetheless, apart from the theory behind it, in a more practical 
> perspective, *the doc is wrong* and says *Default Value: batch* which mislead 
> me and this could happen to more people: 
> [https://cassandra.apache.org/doc/latest/architecture/storage_engine.html#commit-log]
>  
> I think this is important enough for us to quickly fix the documentation in 
> trunk. As this information is fully missing in the 3.11 branch documentation, 
> I suggest we add it as well on that branch (based on information about commit 
> logs from trunk).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16253) Fix documentation to reflect real defaults for commit logs

[Michael Semb Wever (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Semb Wever updated CASSANDRA-16253:
---
Change Category: Semantic
 Complexity: Normal
Component/s: Documentation/Website
  Fix Version/s: 4.0.x
 3.11.x
 Status: Open  (was: Triage Needed)

> Fix documentation to reflect real defaults for commit logs
> --
>
> Key: CASSANDRA-16253
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16253
> Project: Cassandra
>  Issue Type: Task
>  Components: Documentation/Website
>Reporter: Alain RODRIGUEZ
>Assignee: Michael Semb Wever
>Priority: Normal
> Fix For: 3.11.x, 4.0.x
>
>
> About the documentation on commit logs:
>  * Commit logs default is every 10 sec - *periodic*, not batch 
> [https://github.com/apache/cassandra/blob/cassandra-3.11.8/conf/cassandra.yaml#L385-L386]
>  (same in 3.11.9 or trunk). It can lead to data losses but still makes sense 
> to me because fsync-ing every write is very expensive, and (even in batches) 
> should be reserved for the super paranoid where handling multiple power-out 
> (or sudden disk failures) simultaneously on multiple nodes needs to be 
> handled (as [~mck] explained to me.
>   * Nonetheless, apart from the theory behind it, in a more practical 
> perspective, *the doc is wrong* and says *Default Value: batch* which mislead 
> me and this could happen to more people: 
> [https://cassandra.apache.org/doc/latest/architecture/storage_engine.html#commit-log]
>  
> I think this is important enough for us to quickly fix the documentation in 
> trunk. As this information is fully missing in the 3.11 branch documentation, 
> I suggest we add it as well on that branch (based on information about commit 
> logs from trunk).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-16253) Fix documentation to reflect real defaults for commit logs

[Alain RODRIGUEZ (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alain RODRIGUEZ reassigned CASSANDRA-16253:
---

Assignee: Michael Semb Wever  (was: Alain RODRIGUEZ)

> Fix documentation to reflect real defaults for commit logs
> --
>
> Key: CASSANDRA-16253
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16253
> Project: Cassandra
>  Issue Type: Task
>Reporter: Alain RODRIGUEZ
>Assignee: Michael Semb Wever
>Priority: Normal
>
> About the documentation on commit logs:
>  * Commit logs default is every 10 sec - *periodic*, not batch 
> [https://github.com/apache/cassandra/blob/cassandra-3.11.8/conf/cassandra.yaml#L385-L386]
>  (same in 3.11.9 or trunk). It can lead to data losses but still makes sense 
> to me because fsync-ing every write is very expensive, and (even in batches) 
> should be reserved for the super paranoid where handling multiple power-out 
> (or sudden disk failures) simultaneously on multiple nodes needs to be 
> handled (as [~mck] explained to me.
>   * Nonetheless, apart from the theory behind it, in a more practical 
> perspective, *the doc is wrong* and says *Default Value: batch* which mislead 
> me and this could happen to more people: 
> [https://cassandra.apache.org/doc/latest/architecture/storage_engine.html#commit-log]
>  
> I think this is important enough for us to quickly fix the documentation in 
> trunk. As this information is fully missing in the 3.11 branch documentation, 
> I suggest we add it as well on that branch (based on information about commit 
> logs from trunk).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16253) Fix documentation to reflect real defaults for commit logs

[Alain RODRIGUEZ (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228494#comment-17228494
 ] 

Alain RODRIGUEZ commented on CASSANDRA-16253:
-

Patch for 3.11: 
[https://github.com/arodrime/cassandra/tree/alain/16253-add-docs-3.11-commitlog-sync-mode]

Patch for trunk: 
[https://github.com/arodrime/cassandra/tree/alain/16253-fix-docs-commitlog-sync-mode]

I'm not a frequently patching Apache Cassandra or Documentation. Please let me 
know if this patch is as expected.


C*heers

> Fix documentation to reflect real defaults for commit logs
> --
>
> Key: CASSANDRA-16253
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16253
> Project: Cassandra
>  Issue Type: Task
>Reporter: Alain RODRIGUEZ
>Assignee: Alain RODRIGUEZ
>Priority: Normal
>
> About the documentation on commit logs:
>  * Commit logs default is every 10 sec - *periodic*, not batch 
> [https://github.com/apache/cassandra/blob/cassandra-3.11.8/conf/cassandra.yaml#L385-L386]
>  (same in 3.11.9 or trunk). It can lead to data losses but still makes sense 
> to me because fsync-ing every write is very expensive, and (even in batches) 
> should be reserved for the super paranoid where handling multiple power-out 
> (or sudden disk failures) simultaneously on multiple nodes needs to be 
> handled (as [~mck] explained to me.
>   * Nonetheless, apart from the theory behind it, in a more practical 
> perspective, *the doc is wrong* and says *Default Value: batch* which mislead 
> me and this could happen to more people: 
> [https://cassandra.apache.org/doc/latest/architecture/storage_engine.html#commit-log]
>  
> I think this is important enough for us to quickly fix the documentation in 
> trunk. As this information is fully missing in the 3.11 branch documentation, 
> I suggest we add it as well on that branch (based on information about commit 
> logs from trunk).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16253) Fix documentation to reflect real defaults for commit logs

[Alain RODRIGUEZ (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alain RODRIGUEZ updated CASSANDRA-16253:

Description: 
About the documentation on commit logs:
 * Commit logs default is every 10 sec - *periodic*, not batch 
[https://github.com/apache/cassandra/blob/cassandra-3.11.8/conf/cassandra.yaml#L385-L386]
 (same in 3.11.9 or trunk). It can lead to data losses but still makes sense to 
me because fsync-ing every write is very expensive, and (even in batches) 
should be reserved for the super paranoid where handling multiple power-out (or 
sudden disk failures) simultaneously on multiple nodes needs to be handled (as 
[~mck] explained to me.

  * Nonetheless, apart from the theory behind it, in a more practical 
perspective, *the doc is wrong* and says *Default Value: batch* which mislead 
me and this could happen to more people: 
[https://cassandra.apache.org/doc/latest/architecture/storage_engine.html#commit-log]

 

I think this is important enough for us to quickly fix the documentation in 
trunk. As this information is fully missing in the 3.11 branch documentation, I 
suggest we add it as well on that branch (based on information about commit 
logs from trunk).

  was:
About the documentation on commit logs:
 * Commit logs default is every 10 sec (periodic), not batch 
[https://github.com/apache/cassandra/blob/cassandra-3.11.8/conf/cassandra.yaml#L385-L386]
 (same in 3.11.9 or trunk). It can lead to data losses but still makes sense to 
me because fsync-ing every write is very expensive, and (even in batches) 
should be reserved for the super paranoid where handling multiple power-out (or 
sudden disk failures) simultaneously on multiple nodes needs to be handled (as 
[~mck] explained to me.

  * Nonetheless, apart from the theory behind it, in a more practical 
perspective, *the doc is wrong* and says default is `_Default Value:_ batch` 
which mislead me and this could happen to more people: 
[https://cassandra.apache.org/doc/latest/architecture/storage_engine.html#commit-log]

 

I think this is important enough for us to quickly fix the documentation in 
trunk. As this information is fully missing in the 3.11 branch documentation, I 
suggest we add it as well on that branch (based on information about commit 
logs from trunk).


> Fix documentation to reflect real defaults for commit logs
> --
>
> Key: CASSANDRA-16253
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16253
> Project: Cassandra
>  Issue Type: Task
>Reporter: Alain RODRIGUEZ
>Assignee: Alain RODRIGUEZ
>Priority: Normal
>
> About the documentation on commit logs:
>  * Commit logs default is every 10 sec - *periodic*, not batch 
> [https://github.com/apache/cassandra/blob/cassandra-3.11.8/conf/cassandra.yaml#L385-L386]
>  (same in 3.11.9 or trunk). It can lead to data losses but still makes sense 
> to me because fsync-ing every write is very expensive, and (even in batches) 
> should be reserved for the super paranoid where handling multiple power-out 
> (or sudden disk failures) simultaneously on multiple nodes needs to be 
> handled (as [~mck] explained to me.
>   * Nonetheless, apart from the theory behind it, in a more practical 
> perspective, *the doc is wrong* and says *Default Value: batch* which mislead 
> me and this could happen to more people: 
> [https://cassandra.apache.org/doc/latest/architecture/storage_engine.html#commit-log]
>  
> I think this is important enough for us to quickly fix the documentation in 
> trunk. As this information is fully missing in the 3.11 branch documentation, 
> I suggest we add it as well on that branch (based on information about commit 
> logs from trunk).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-16253) Fix documentation to reflect real defaults for commit logs

[Alain RODRIGUEZ (Jira)]

Alain RODRIGUEZ created CASSANDRA-16253:
---

 Summary: Fix documentation to reflect real defaults for commit logs
 Key: CASSANDRA-16253
 URL: https://issues.apache.org/jira/browse/CASSANDRA-16253
 Project: Cassandra
  Issue Type: Task
Reporter: Alain RODRIGUEZ
Assignee: Alain RODRIGUEZ


About the documentation on commit logs:
 * Commit logs default is every 10 sec (periodic), not batch 
[https://github.com/apache/cassandra/blob/cassandra-3.11.8/conf/cassandra.yaml#L385-L386]
 (same in 3.11.9 or trunk). It can lead to data losses but still makes sense to 
me because fsync-ing every write is very expensive, and (even in batches) 
should be reserved for the super paranoid where handling multiple power-out (or 
sudden disk failures) simultaneously on multiple nodes needs to be handled (as 
[~mck] explained to me.

  * Nonetheless, apart from the theory behind it, in a more practical 
perspective, *the doc is wrong* and says default is `_Default Value:_ batch` 
which mislead me and this could happen to more people: 
[https://cassandra.apache.org/doc/latest/architecture/storage_engine.html#commit-log]

 

I think this is important enough for us to quickly fix the documentation in 
trunk. As this information is fully missing in the 3.11 branch documentation, I 
suggest we add it as well on that branch (based on information about commit 
logs from trunk).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-16176) Python dtests should run at debug

[Berenguer Blasi (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228412#comment-17228412
 ] 

Berenguer Blasi edited comment on CASSANDRA-16176 at 11/9/20, 9:56 AM:
---

[~brandon.williams] I gave it a go. 
[Here|https://app.circleci.com/pipelines/github/bereng/cassandra/178/workflows/7a5e5dce-ce05-49c8-8bb1-5ff23b839666/jobs/1391/artifacts]
 is the run. 2 q:
 - Is this what you wanted? (we still need high and med res test runs before 
merging if you confirm)
 - Why not TRACE instead of DEBUG?


was (Author: bereng):
[~brandon.williams] I gave it a go. 
[Here|https://app.circleci.com/pipelines/github/bereng/cassandra/177/workflows/34090602-f24c-4b10-9066-51a3f6090f0f]
 is the run and 
[here|https://circle-production-customer-artifacts.s3.amazonaws.com/picard/5e9576564660060baefcf188/5fa8e624911a071fa18e84ff-0-build/artifacts/logs/org.apache.cassandra.distributed.test.RepairBoundaryTest/cluster-b8e5bbe5-bd37-45d9-bf6a-647f88b5cac7/node1/system.log?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20201109T083014Z&X-Amz-SignedHeaders=host&X-Amz-Expires=60&X-Amz-Credential=AKIAJR3Q6CR467H7Z55A%2F20201109%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=8f5eb93861a86d7bea969b7ad8f68bc57708424266318952b5834d66d546b522]
 a sample log file. 2 q:
 - Is this what you wanted? (we still need high and med res test runs before 
merging if you confirm)
 - Why not TRACE instead of DEBUG?

> Python dtests should run at debug
> -
>
> Key: CASSANDRA-16176
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16176
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CI
>Reporter: Brandon Williams
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> At least in our circle configs, we are specifying --log-level="INFO" which 
> often leaves us with nothing to go on when we have a flaky test that only 
> fails in a certain environment.  In general it would be more useful to always 
> run at DEBUG. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-16176) Python dtests should run at debug

[Berenguer Blasi (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228473#comment-17228473
 ] 

Berenguer Blasi edited comment on CASSANDRA-16176 at 11/9/20, 9:55 AM:
---

Apologies wrong link. 
[Here|https://app.circleci.com/pipelines/github/bereng/cassandra/178/workflows/7a5e5dce-ce05-49c8-8bb1-5ff23b839666/jobs/1391/artifacts]
 is the right one. I can see DEBUG info both in the log files and on stdout.txt 
for ccm and for tests.


was (Author: bereng):
Apologies wrong link. 
[https://app.circleci.com/pipelines/github/bereng/cassandra/178/workflows/7a5e5dce-ce05-49c8-8bb1-5ff23b839666/jobs/1391/artifacts|http://example.com]
 is the right one. I can see DEBUG info both in the log files and on stdout.txt 
for ccm and for tests.

> Python dtests should run at debug
> -
>
> Key: CASSANDRA-16176
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16176
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CI
>Reporter: Brandon Williams
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> At least in our circle configs, we are specifying --log-level="INFO" which 
> often leaves us with nothing to go on when we have a flaky test that only 
> fails in a certain environment.  In general it would be more useful to always 
> run at DEBUG. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16176) Python dtests should run at debug

[Berenguer Blasi (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228473#comment-17228473
 ] 

Berenguer Blasi commented on CASSANDRA-16176:
-

Apologies wrong link. 
[https://app.circleci.com/pipelines/github/bereng/cassandra/178/workflows/7a5e5dce-ce05-49c8-8bb1-5ff23b839666/jobs/1391/artifacts|http://example.com]
 is the right one. I can see DEBUG info both in the log files and on stdout.txt 
for ccm and for tests.

> Python dtests should run at debug
> -
>
> Key: CASSANDRA-16176
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16176
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CI
>Reporter: Brandon Williams
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> At least in our circle configs, we are specifying --log-level="INFO" which 
> often leaves us with nothing to go on when we have a flaky test that only 
> fails in a certain environment.  In general it would be more useful to always 
> run at DEBUG. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Issue Comment Deleted] (CASSANDRA-16176) Python dtests should run at debug

[Berenguer Blasi (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Berenguer Blasi updated CASSANDRA-16176:

Comment: was deleted

(was: Meh.. must be some expiring session or God knows what. Go to the [test 
run|https://app.circleci.com/pipelines/github/bereng/cassandra/177/workflows/34090602-f24c-4b10-9066-51a3f6090f0f/jobs/1381]
 -> artifacts -> select a random log file and check it has the DEBUG statements 
you're expecting.)

> Python dtests should run at debug
> -
>
> Key: CASSANDRA-16176
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16176
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CI
>Reporter: Brandon Williams
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> At least in our circle configs, we are specifying --log-level="INFO" which 
> often leaves us with nothing to go on when we have a flaky test that only 
> fails in a certain environment.  In general it would be more useful to always 
> run at DEBUG. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-16176) Python dtests should run at debug

[Berenguer Blasi (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228447#comment-17228447
 ] 

Berenguer Blasi edited comment on CASSANDRA-16176 at 11/9/20, 9:27 AM:
---

Meh.. must be some expiring session or God knows what. Go to the [test 
run|https://app.circleci.com/pipelines/github/bereng/cassandra/177/workflows/34090602-f24c-4b10-9066-51a3f6090f0f/jobs/1381]
 -> artifacts -> select a random log file and check it has the DEBUG statements 
you're expecting.


was (Author: bereng):
Meh.. must be some expiring session or God knows what. Go to the [test 
run|https://app.circleci.com/pipelines/github/bereng/cassandra/177/workflows/34090602-f24c-4b10-9066-51a3f6090f0f]
 -> artifacts -> select a random log file and check it has the DEBUG statements 
you're expecting.

> Python dtests should run at debug
> -
>
> Key: CASSANDRA-16176
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16176
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CI
>Reporter: Brandon Williams
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> At least in our circle configs, we are specifying --log-level="INFO" which 
> often leaves us with nothing to go on when we have a flaky test that only 
> fails in a certain environment.  In general it would be more useful to always 
> run at DEBUG. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-16176) Python dtests should run at debug

[Berenguer Blasi (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228447#comment-17228447
 ] 

Berenguer Blasi edited comment on CASSANDRA-16176 at 11/9/20, 9:26 AM:
---

Meh.. must be some expiring session or God knows what. Go to the [test 
run|https://app.circleci.com/pipelines/github/bereng/cassandra/177/workflows/34090602-f24c-4b10-9066-51a3f6090f0f]
 -> artifacts -> select a random log file and check it has the DEBUG statements 
you're expecting.


was (Author: bereng):
Meh.. must be some expiring session or God knows what. Go to the test run -> 
artifacts -> select a random log file and check it has the DEBUG statements 
you're expecting.

> Python dtests should run at debug
> -
>
> Key: CASSANDRA-16176
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16176
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CI
>Reporter: Brandon Williams
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> At least in our circle configs, we are specifying --log-level="INFO" which 
> often leaves us with nothing to go on when we have a flaky test that only 
> fails in a certain environment.  In general it would be more useful to always 
> run at DEBUG. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16176) Python dtests should run at debug

[Berenguer Blasi (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228447#comment-17228447
 ] 

Berenguer Blasi commented on CASSANDRA-16176:
-

Meh.. must be some expiring session or God knows what. Go to the test run -> 
artifacts -> select a random log file and check it has the DEBUG statements 
you're expecting.

> Python dtests should run at debug
> -
>
> Key: CASSANDRA-16176
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16176
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CI
>Reporter: Brandon Williams
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> At least in our circle configs, we are specifying --log-level="INFO" which 
> often leaves us with nothing to go on when we have a flaky test that only 
> fails in a certain environment.  In general it would be more useful to always 
> run at DEBUG. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16171) Remove Windows scripts

[Berenguer Blasi (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228445#comment-17228445
 ] 

Berenguer Blasi commented on CASSANDRA-16171:
-

Ok gtk. We don't need anything specific to windows. Thx for the info I noticed 
dev-branch also builds the artifacts. So [~yukim] a dev-branch test run should 
suffice imo.

> Remove Windows scripts
> --
>
> Key: CASSANDRA-16171
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16171
> Project: Cassandra
>  Issue Type: Task
>  Components: Packaging
>Reporter: Yuki Morishita
>Assignee: Yuki Morishita
>Priority: Normal
> Fix For: 4.0-rc
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> As per the email thread in cassandra-dev mailing list[1], remove windows 
> scripts from Cassandra 4.0 onwards, due to the lack of maintenance and tests.
> 1: https://www.mail-archive.com/dev@cassandra.apache.org/msg15583.html 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-16176) Python dtests should run at debug

[Michael Semb Wever (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Semb Wever reassigned CASSANDRA-16176:
--

Assignee: Berenguer Blasi  (was: Brandon Williams)

> Python dtests should run at debug
> -
>
> Key: CASSANDRA-16176
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16176
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CI
>Reporter: Brandon Williams
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> At least in our circle configs, we are specifying --log-level="INFO" which 
> often leaves us with nothing to go on when we have a flaky test that only 
> fails in a certain environment.  In general it would be more useful to always 
> run at DEBUG. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16176) Python dtests should run at debug

[Michael Semb Wever (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228437#comment-17228437
 ] 

Michael Semb Wever commented on CASSANDRA-16176:


bq. Here is the run and here a sample log file. 

I can't access the logs you have linked to. "AccessDenied" (expired).

> Python dtests should run at debug
> -
>
> Key: CASSANDRA-16176
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16176
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CI
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> At least in our circle configs, we are specifying --log-level="INFO" which 
> often leaves us with nothing to go on when we have a flaky test that only 
> fails in a certain environment.  In general it would be more useful to always 
> run at DEBUG. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16171) Remove Windows scripts

[Michael Semb Wever (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228433#comment-17228433
 ] 

Michael Semb Wever commented on CASSANDRA-16171:


bq. We don't have a way to test artifacts generation right Mick?

Tarball, debian and redhat, artefacts are generated in CI 
[here|https://ci-cassandra.apache.org/job/Cassandra-trunk-artifacts/], for both 
JDK8 and JDK11 in trunk.

These artefacts can also be downloaded from nightlies… for JDK8 
[here|https://nightlies.apache.org/cassandra/Cassandra-trunk-artifacts/jdk=jdk_1.8_latest,label=cassandra/]
 and for JDK11 
[here|https://nightlies.apache.org/cassandra/Cassandra-trunk-artifacts/jdk=jdk_11_latest,label=cassandra/].

But in context, there's nothing here specific to Windows.

> Remove Windows scripts
> --
>
> Key: CASSANDRA-16171
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16171
> Project: Cassandra
>  Issue Type: Task
>  Components: Packaging
>Reporter: Yuki Morishita
>Assignee: Yuki Morishita
>Priority: Normal
> Fix For: 4.0-rc
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> As per the email thread in cassandra-dev mailing list[1], remove windows 
> scripts from Cassandra 4.0 onwards, due to the lack of maintenance and tests.
> 1: https://www.mail-archive.com/dev@cassandra.apache.org/msg15583.html 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16171) Remove Windows scripts

[Berenguer Blasi (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228419#comment-17228419
 ] 

Berenguer Blasi commented on CASSANDRA-16171:
-

bq. Yes we should, and these two can be handled in CASSANDRA-16173

sgtm

{quote}Agree. I will submit CI to run tests.
I don't think we have automated test to check artifacts generation, 
right?{quote}

I think you're right but let me ping [~mck] who knows this stuff inside out for 
him to confirm. We don't have a way to test artifacts generation right Mick?

bq. There are many places that can be removed from the code, so eventually we 
can remove those in the future.

Why not create the ticket now and link it into these so that thing doesn't fall 
through the cracks? Wdyt?


> Remove Windows scripts
> --
>
> Key: CASSANDRA-16171
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16171
> Project: Cassandra
>  Issue Type: Task
>  Components: Packaging
>Reporter: Yuki Morishita
>Assignee: Yuki Morishita
>Priority: Normal
> Fix For: 4.0-rc
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> As per the email thread in cassandra-dev mailing list[1], remove windows 
> scripts from Cassandra 4.0 onwards, due to the lack of maintenance and tests.
> 1: https://www.mail-archive.com/dev@cassandra.apache.org/msg15583.html 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-16176) Python dtests should run at debug

[Berenguer Blasi (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228412#comment-17228412
 ] 

Berenguer Blasi edited comment on CASSANDRA-16176 at 11/9/20, 8:34 AM:
---

[~brandon.williams] I gave it a go. 
[Here|https://app.circleci.com/pipelines/github/bereng/cassandra/177/workflows/34090602-f24c-4b10-9066-51a3f6090f0f]
 is the run and 
[here|https://circle-production-customer-artifacts.s3.amazonaws.com/picard/5e9576564660060baefcf188/5fa8e624911a071fa18e84ff-0-build/artifacts/logs/org.apache.cassandra.distributed.test.RepairBoundaryTest/cluster-b8e5bbe5-bd37-45d9-bf6a-647f88b5cac7/node1/system.log?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20201109T083014Z&X-Amz-SignedHeaders=host&X-Amz-Expires=60&X-Amz-Credential=AKIAJR3Q6CR467H7Z55A%2F20201109%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=8f5eb93861a86d7bea969b7ad8f68bc57708424266318952b5834d66d546b522]
 a sample log file. 2 q:
 - Is this what you wanted? (we still need high and med res test runs before 
merging if you confirm)
 - Why not TRACE instead of DEBUG?


was (Author: bereng):
[~brandon.williams] I gave it a go. 
[Here|https://app.circleci.com/pipelines/github/bereng/cassandra/177/workflows/34090602-f24c-4b10-9066-51a3f6090f0f]
 is the run and 
[here|https://circle-production-customer-artifacts.s3.amazonaws.com/picard/5e9576564660060baefcf188/5fa8e624911a071fa18e84ff-0-build/artifacts/logs/org.apache.cassandra.distributed.test.RepairBoundaryTest/cluster-b8e5bbe5-bd37-45d9-bf6a-647f88b5cac7/node1/system.log?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20201109T083014Z&X-Amz-SignedHeaders=host&X-Amz-Expires=60&X-Amz-Credential=AKIAJR3Q6CR467H7Z55A%2F20201109%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=8f5eb93861a86d7bea969b7ad8f68bc57708424266318952b5834d66d546b522]
 a sample log file. 2 q:
 - Is this what you wanted? (we still need high and med res test runs if you 
confirm)
 - Why not TRACE instead of DEBUG?

> Python dtests should run at debug
> -
>
> Key: CASSANDRA-16176
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16176
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CI
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> At least in our circle configs, we are specifying --log-level="INFO" which 
> often leaves us with nothing to go on when we have a flaky test that only 
> fails in a certain environment.  In general it would be more useful to always 
> run at DEBUG. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-16176) Python dtests should run at debug

[Berenguer Blasi (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228412#comment-17228412
 ] 

Berenguer Blasi edited comment on CASSANDRA-16176 at 11/9/20, 8:33 AM:
---

[~brandon.williams] I gave it a go. 
[Here|https://app.circleci.com/pipelines/github/bereng/cassandra/177/workflows/34090602-f24c-4b10-9066-51a3f6090f0f]
 is the run and 
[here|https://circle-production-customer-artifacts.s3.amazonaws.com/picard/5e9576564660060baefcf188/5fa8e624911a071fa18e84ff-0-build/artifacts/logs/org.apache.cassandra.distributed.test.RepairBoundaryTest/cluster-b8e5bbe5-bd37-45d9-bf6a-647f88b5cac7/node1/system.log?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20201109T083014Z&X-Amz-SignedHeaders=host&X-Amz-Expires=60&X-Amz-Credential=AKIAJR3Q6CR467H7Z55A%2F20201109%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=8f5eb93861a86d7bea969b7ad8f68bc57708424266318952b5834d66d546b522]
 a sample log file. 2 q:
 - Is this what you wanted? (we still need high and med res test runs if you 
confirm)
 - Why not TRACE instead of DEBUG?


was (Author: bereng):
[~brandon.williams] I gave it a go. 
[Here|https://app.circleci.com/pipelines/github/bereng/cassandra/177/workflows/34090602-f24c-4b10-9066-51a3f6090f0f]
 is the run and 
[https://circle-production-customer-artifacts.s3.amazonaws.com/picard/5e9576564660060baefcf188/5fa8e624911a071fa18e84ff-0-build/artifacts/logs/org.apache.cassandra.distributed.test.RepairBoundaryTest/cluster-b8e5bbe5-bd37-45d9-bf6a-647f88b5cac7/node1/system.log?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20201109T083014Z&X-Amz-SignedHeaders=host&X-Amz-Expires=60&X-Amz-Credential=AKIAJR3Q6CR467H7Z55A%2F20201109%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=8f5eb93861a86d7bea969b7ad8f68bc57708424266318952b5834d66d546b522|http://example.com]
 a sample log file. 2 q:
 - Is this what you wanted? (we still need high and med res test runs if you 
confirm)
 - Why not TRACE instead of DEBUG?

> Python dtests should run at debug
> -
>
> Key: CASSANDRA-16176
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16176
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CI
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> At least in our circle configs, we are specifying --log-level="INFO" which 
> often leaves us with nothing to go on when we have a flaky test that only 
> fails in a certain environment.  In general it would be more useful to always 
> run at DEBUG. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-16176) Python dtests should run at debug

[Berenguer Blasi (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Berenguer Blasi updated CASSANDRA-16176:

Test and Documentation Plan: See PR
 Status: Patch Available  (was: In Progress)

> Python dtests should run at debug
> -
>
> Key: CASSANDRA-16176
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16176
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CI
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> At least in our circle configs, we are specifying --log-level="INFO" which 
> often leaves us with nothing to go on when we have a flaky test that only 
> fails in a certain environment.  In general it would be more useful to always 
> run at DEBUG. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16176) Python dtests should run at debug

[Berenguer Blasi (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228412#comment-17228412
 ] 

Berenguer Blasi commented on CASSANDRA-16176:
-

[~brandon.williams] I gave it a go. 
[Here|https://app.circleci.com/pipelines/github/bereng/cassandra/177/workflows/34090602-f24c-4b10-9066-51a3f6090f0f]
 is the run and 
[https://circle-production-customer-artifacts.s3.amazonaws.com/picard/5e9576564660060baefcf188/5fa8e624911a071fa18e84ff-0-build/artifacts/logs/org.apache.cassandra.distributed.test.RepairBoundaryTest/cluster-b8e5bbe5-bd37-45d9-bf6a-647f88b5cac7/node1/system.log?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20201109T083014Z&X-Amz-SignedHeaders=host&X-Amz-Expires=60&X-Amz-Credential=AKIAJR3Q6CR467H7Z55A%2F20201109%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=8f5eb93861a86d7bea969b7ad8f68bc57708424266318952b5834d66d546b522|http://example.com]
 a sample log file. 2 q:
 - Is this what you wanted? (we still need high and med res test runs if you 
confirm)
 - Why not TRACE instead of DEBUG?

> Python dtests should run at debug
> -
>
> Key: CASSANDRA-16176
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16176
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CI
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> At least in our circle configs, we are specifying --log-level="INFO" which 
> often leaves us with nothing to go on when we have a flaky test that only 
> fails in a certain environment.  In general it would be more useful to always 
> run at DEBUG. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-16171) Remove Windows scripts

[Yuki Morishita (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-16171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228404#comment-17228404
 ] 

Yuki Morishita commented on CASSANDRA-16171:


Thanks for the comment.

I'd like this ticket to just remove windows scripts (*.bat / *.ps1) from the 
release artifact, as those are broken and should be a blocker for 4.0 GA 
release.
{quote}We should add a note in docs about the removal, explanation and the 
workaround for current Windows users. NEWS file is not enough imo. Otherwise 
.bat scripts will just be gone with no explanation for them
We should remove from docs all references to Windows (grep -ri window in doc 
folder) imo
{quote}
Yes we should, and these two can be handled in CASSANDRA-16173
{quote}We should run tests and also check test artifacts generation are not 
broken.
{quote}
Agree. I will submit CI to run tests.
I don't think we have automated test to check artifacts generation, right?
{quote}Should we create a ticket to simplify the code by removing Windows 
support from the code/pytests? There seems to be a bit of that
{quote}
There are many places that can be removed from the code, so eventually we can 
remove those in the future.

> Remove Windows scripts
> --
>
> Key: CASSANDRA-16171
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16171
> Project: Cassandra
>  Issue Type: Task
>  Components: Packaging
>Reporter: Yuki Morishita
>Assignee: Yuki Morishita
>Priority: Normal
> Fix For: 4.0-rc
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> As per the email thread in cassandra-dev mailing list[1], remove windows 
> scripts from Cassandra 4.0 onwards, due to the lack of maintenance and tests.
> 1: https://www.mail-archive.com/dev@cassandra.apache.org/msg15583.html 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org