[jira] [Commented] (CASSANDRA-15823) Support for networking via identity instead of IP
[
https://issues.apache.org/jira/browse/CASSANDRA-15823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200519#comment-17200519
]
Jeff Jirsa commented on CASSANDRA-15823:
{quote}
If a node comes up with a completely brand new IP that was never part of the
cluster before, then we do not have any issues that I know of. Do you know of
any problems that can happen for that case Jeff Jirsa ?
{quote}
Think this is true, yes
{quote}
While I agree it would be best to have all membership based on UUID's, I think
we need to allow people to have hostnames be the contact point, and have those
re-resolve on every "connect". While I agree "ips are best, dns is the devil",
I have seen bad DNS take down clusters, there are many systems being created
right now where hostnames are the invariant, not ips, and we need Cassandra to
be able to play in those environments.
{quote}
If someone decides to completely redesign all of the internals to make it work
by dns name instead of hostID, I think that would be a pretty horrible mistake.
Yes, working by DNS should work, but things like ownership need to get to the
point that we're referring to the (known unique) host IDs instead of just
slapping a bandaid on top of the IP problem.
> Support for networking via identity instead of IP
> -
>
> Key: CASSANDRA-15823
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15823
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Christopher Bradford
>Priority: Normal
> Labels: kubernetes
> Attachments: consul-mesh-gateways.png,
> istio-multicluster-with-gateways.svg, linkerd-service-mirroring.svg
>
>
> TL;DR: Instead of mapping host ids to IPs, use hostnames. This allows
> resolution to different IP addresses per DC that may then be forwarded to
> nodes on remote networks without requiring node to node IP connectivity for
> cross-dc links.
>
> This approach should not affect existing deployments as those could continue
> to use IPs as the hostname and skip resolution.
>
> With orchestration platforms like Kubernetes and the usage of ephemeral
> containers in environments today we should consider some changes to how we
> handle the tracking of nodes and their network location. Currently we
> maintain a mapping between host ids and IP addresses.
>
> With traditional infrastructure, if a node goes down it, usually, comes back
> up with the same IP. In some environments this contract may be explicit with
> virtual IPs that may move between hosts. In newer deployments, like on
> Kubernetes, this contract is not possible. Pods (analogous to nodes) are
> assigned an IP address at start time. Should the pod be restarted or
> scheduled on a different host there is no guarantee we would have the same
> IP. Cassandra is protected here as we already have logic in place to update
> peers when we come up with the same host id, but a different IP address.
>
> There are ways to get Kubernetes to assign a specific IP per Pod. Most
> recommendations involve the use of a service per pod. Communication with the
> fixed service IP would automatically forward to the associated pod,
> regardless of address. We _could_ use this approach, but it seems like this
> would needlessly create a number of extra resources in our k8s cluster to get
> around the problem. Which, to be fair, doesn't seem like much of a problem
> with the aforementioned mitigations built into C*.
>
> So what is the _actual_ problem? *Cross-region, cross-cloud,
> hybrid-deployment connectivity between pods is a pain.* This can be solved
> with significant investment by those who want to deploy these types of
> topologies. You can definitely configure connectivity between clouds over
> dedicated connections, or VPN tunnels. With a big chunk of time insuring that
> pod to pod connectivity just works even if those pods are managed by separate
> control planes, but that again requires time and talent. There are a number
> of edge cases to support between the ever so slight, but very important,
> differences in cloud vendor networks.
>
> Recently there have been a number of innovations that aid in the deployment
> and operation of these types of applications on Kubernetes. Service meshes
> support distributed microservices running across multiple k8s cluster control
> planes in disparate networks. Instead of directly connecting to IP addresses
> of remote services instead they use a hostname. With this approach, hostname
> traffic may then be routed to a proxy that sends traffic over the WAN
> (sometimes with mTLS) to another proxy pod in the remote cluster which then
> forwards the data along to the correct pod in that network. (See attached
> diagrams)
>
> Which brings us to the point of this ticket.
[jira] [Commented] (CASSANDRA-16127) NullPointerException when calling nodetool enablethrift
[
https://issues.apache.org/jira/browse/CASSANDRA-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200457#comment-17200457
]
David Capwell commented on CASSANDRA-16127:
---
[~yifanc] applied to all feedback
[~e.dimitrova] rebased 3.11 and dtest (the PR branches).
> NullPointerException when calling nodetool enablethrift
> ---
>
> Key: CASSANDRA-16127
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16127
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Thrift
>Reporter: Tibor Repasi
>Assignee: David Capwell
>Priority: Normal
> Fix For: 2.2.x, 3.0.x, 3.11.x
>
>
> Having thrift disabled, it's impossible to enable it again without restarting
> the node:
> {code}
> $ nodetool statusthrift
> not running
> $ nodetool enablethrift
> error: null
> -- StackTrace --
> java.lang.NullPointerException
> at
> org.apache.cassandra.service.StorageService.startRPCServer(StorageService.java:392)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16048) Safely Ignore Compact Storage Tables Where Users Have Defined Clustering and Value Columns
[ https://issues.apache.org/jira/browse/CASSANDRA-16048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Caleb Rackliffe updated CASSANDRA-16048: Reviewers: Caleb Rackliffe, Caleb Rackliffe (was: Caleb Rackliffe) Caleb Rackliffe, Caleb Rackliffe (was: Caleb Rackliffe) Status: Review In Progress (was: Patch Available) > Safely Ignore Compact Storage Tables Where Users Have Defined Clustering and > Value Columns > -- > > Key: CASSANDRA-16048 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16048 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/CQL >Reporter: Jordan West >Assignee: Jordan West >Priority: Normal > Fix For: 4.0-beta > > > Some compact storage tables, specifically those where the user has defined > both at least one clustering and the value column, can be safely handled in > 4.0 because besides the DENSE flag they are not materially different post 3.0 > and there is no visible change to the user facing schema after dropping > compact storage. We can detect this case and allow these tables to silently > drop the DENSE flag while still throwing a start-up error for COMPACT STORAGE > tables that don’t meet the criteria. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15164) Overflowed Partition Cell Histograms Can Prevent Compactions from Executing
[ https://issues.apache.org/jira/browse/CASSANDRA-15164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200364#comment-17200364 ] Caleb Rackliffe commented on CASSANDRA-15164: - The tests look pretty clean, outside hitting CASSANDRA-15313 again. > Overflowed Partition Cell Histograms Can Prevent Compactions from Executing > --- > > Key: CASSANDRA-15164 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15164 > Project: Cassandra > Issue Type: Bug > Components: CQL/Interpreter >Reporter: Ankur Jha >Assignee: Caleb Rackliffe >Priority: Urgent > Labels: compaction, partition > Fix For: 4.0-beta > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Hi, we are running 6 node Cassandra cluster in production with 3 seed node > but from last night one of our seed nodes is continuously throwing an error > like this;- > cassandra.protocol.ServerError: message="java.lang.IllegalStateException: Unable to compute ceiling for max > when histogram overflowed"> > For a cluster to be up and running I Drained this node. > Can somebody help me out with this? > > Any help or lead would be appreciated > > Note : We are using Cassandra version 3.7 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15313) Fix flaky - ChecksummingTransformerTest - org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
[
https://issues.apache.org/jira/browse/CASSANDRA-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200363#comment-17200363
]
Caleb Rackliffe commented on CASSANDRA-15313:
-
Reproduced again here:
https://app.circleci.com/pipelines/github/maedhroz/cassandra/118/workflows/3cecc61c-043b-48c9-ad10-88495df8b6d1/jobs/640
> Fix flaky - ChecksummingTransformerTest -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> ---
>
> Key: CASSANDRA-15313
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15313
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Vinay Chella
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: CASSANDRA-15313-hack.patch
>
>
> During the recent runs, this test appears to be flaky.
> Example failure:
> [https://circleci.com/gh/vinaykumarchella/cassandra/459#tests/containers/94]
> corruptionCausesFailure-compression -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> {code:java}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> at org.quicktheories.impl.Precursor.(Precursor.java:17)
> at
> org.quicktheories.impl.ConcreteDetachedSource.(ConcreteDetachedSource.java:8)
> at
> org.quicktheories.impl.ConcreteDetachedSource.detach(ConcreteDetachedSource.java:23)
> at org.quicktheories.generators.Retry.generate(CodePoints.java:51)
> at
> org.quicktheories.generators.Generate.lambda$intArrays$10(Generate.java:190)
> at
> org.quicktheories.generators.Generate$$Lambda$17/1847008471.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$mix$10(Gen.java:184)
> at org.quicktheories.core.Gen$$Lambda$45/802243390.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$flatMap$5(Gen.java:93)
> at org.quicktheories.core.Gen$$Lambda$48/363509958.generate(Unknown
> Source)
> at
> org.quicktheories.dsl.TheoryBuilder4.lambda$prgnToTuple$12(TheoryBuilder4.java:188)
> at
> org.quicktheories.dsl.TheoryBuilder4$$Lambda$40/2003496028.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.FilteredGenerator.generate(Gen.java:225)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.impl.Core.generate(Core.java:150)
> at org.quicktheories.impl.Core.shrink(Core.java:103)
> at org.quicktheories.impl.Core.run(Core.java:39)
> at org.quicktheories.impl.TheoryRunner.check(TheoryRunner.java:35)
> at org.quicktheories.dsl.TheoryBuilder4.check(TheoryBuilder4.java:150)
> at
> org.quicktheories.dsl.TheoryBuilder4.checkAssert(TheoryBuilder4.java:162)
> at
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure(ChecksummingTransformerTest.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15164) Overflowed Partition Cell Histograms Can Prevent Compactions from Executing
[ https://issues.apache.org/jira/browse/CASSANDRA-15164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200343#comment-17200343 ] Caleb Rackliffe edited comment on CASSANDRA-15164 at 9/22/20, 8:23 PM: --- [~blerer] [~clohfink] I've reworked the patch a bit to handle both of the stats histograms and take into account the feedback [here|https://github.com/apache/cassandra/pull/750#pullrequestreview-493580468]. Short of dynamically growing the number of buckets, this seems like the most reasonable quick fix to make the things that might depend on an overflowed histogram safe. (The worst case is that our estimates won't be as helpful as they would if we had more buckets, etc.) Tests in progress [here|https://app.circleci.com/pipelines/github/maedhroz/cassandra?branch=CASSANDRA-15164]... was (Author: maedhroz): [~blerer] [~clohfink] I've reworked the patch a bit to handle both of the stats histograms and take into account the feedback [here|https://github.com/apache/cassandra/pull/750#pullrequestreview-493580468]. Short of dynamically growing the number of buckets, this seems like the most reasonable quick fix to make the things that might depend on an overflowed histogram safe. (The worst case is that our estimates won't be as helpful as they would if we had more buckets, etc.) > Overflowed Partition Cell Histograms Can Prevent Compactions from Executing > --- > > Key: CASSANDRA-15164 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15164 > Project: Cassandra > Issue Type: Bug > Components: CQL/Interpreter >Reporter: Ankur Jha >Assignee: Caleb Rackliffe >Priority: Urgent > Labels: compaction, partition > Fix For: 4.0-beta > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Hi, we are running 6 node Cassandra cluster in production with 3 seed node > but from last night one of our seed nodes is continuously throwing an error > like this;- > cassandra.protocol.ServerError: message="java.lang.IllegalStateException: Unable to compute ceiling for max > when histogram overflowed"> > For a cluster to be up and running I Drained this node. > Can somebody help me out with this? > > Any help or lead would be appreciated > > Note : We are using Cassandra version 3.7 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15164) Overflowed Partition Cell Histograms Can Prevent Compactions from Executing
[ https://issues.apache.org/jira/browse/CASSANDRA-15164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200343#comment-17200343 ] Caleb Rackliffe commented on CASSANDRA-15164: - [~blerer] [~clohfink] I've reworked the patch a bit to handle both of the stats histograms and take into account the feedback [here|https://github.com/apache/cassandra/pull/750#pullrequestreview-493580468]. Short of dynamically growing the number of buckets, this seems like the most reasonable quick fix to make the things that might depend on an overflowed histogram safe. (The worst case is that our estimates won't be as helpful as they would if we had more buckets, etc.) > Overflowed Partition Cell Histograms Can Prevent Compactions from Executing > --- > > Key: CASSANDRA-15164 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15164 > Project: Cassandra > Issue Type: Bug > Components: CQL/Interpreter >Reporter: Ankur Jha >Assignee: Caleb Rackliffe >Priority: Urgent > Labels: compaction, partition > Fix For: 4.0-beta > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Hi, we are running 6 node Cassandra cluster in production with 3 seed node > but from last night one of our seed nodes is continuously throwing an error > like this;- > cassandra.protocol.ServerError: message="java.lang.IllegalStateException: Unable to compute ceiling for max > when histogram overflowed"> > For a cluster to be up and running I Drained this node. > Can somebody help me out with this? > > Any help or lead would be appreciated > > Note : We are using Cassandra version 3.7 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16127) NullPointerException when calling nodetool enablethrift
[
https://issues.apache.org/jira/browse/CASSANDRA-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200311#comment-17200311
]
Ekaterina Dimitrova commented on CASSANDRA-16127:
-
Hey [~dcapwell],
Thanks for the patch, also improving dtests.
Do you think you can group some of the commits if this doesn't bother [~yifanc]
as a reviewer? Not sure where he is in there.
> NullPointerException when calling nodetool enablethrift
> ---
>
> Key: CASSANDRA-16127
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16127
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Thrift
>Reporter: Tibor Repasi
>Assignee: David Capwell
>Priority: Normal
> Fix For: 2.2.x, 3.0.x, 3.11.x
>
>
> Having thrift disabled, it's impossible to enable it again without restarting
> the node:
> {code}
> $ nodetool statusthrift
> not running
> $ nodetool enablethrift
> error: null
> -- StackTrace --
> java.lang.NullPointerException
> at
> org.apache.cassandra.service.StorageService.startRPCServer(StorageService.java:392)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16127) NullPointerException when calling nodetool enablethrift
[
https://issues.apache.org/jira/browse/CASSANDRA-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200311#comment-17200311
]
Ekaterina Dimitrova edited comment on CASSANDRA-16127 at 9/22/20, 7:01 PM:
---
Hey [~dcapwell],
Thanks for the patch, also improving dtests.
Do you think you can group some of the commits if this doesn't bother [~yifanc]
as a reviewer? Not sure where he is in there and whether this will affect his
tracking of the patch changes.
was (Author: e.dimitrova):
Hey [~dcapwell],
Thanks for the patch, also improving dtests.
Do you think you can group some of the commits if this doesn't bother [~yifanc]
as a reviewer? Not sure where he is in there.
> NullPointerException when calling nodetool enablethrift
> ---
>
> Key: CASSANDRA-16127
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16127
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Thrift
>Reporter: Tibor Repasi
>Assignee: David Capwell
>Priority: Normal
> Fix For: 2.2.x, 3.0.x, 3.11.x
>
>
> Having thrift disabled, it's impossible to enable it again without restarting
> the node:
> {code}
> $ nodetool statusthrift
> not running
> $ nodetool enablethrift
> error: null
> -- StackTrace --
> java.lang.NullPointerException
> at
> org.apache.cassandra.service.StorageService.startRPCServer(StorageService.java:392)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail:
[jira] [Commented] (CASSANDRA-16116) Emit a metric for mutations that are too large in commit logs
[ https://issues.apache.org/jira/browse/CASSANDRA-16116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200268#comment-17200268 ] Chris Lohfink commented on CASSANDRA-16116: --- [java8|https://app.circleci.com/pipelines/github/clohfink/cassandra/9/workflows/27d99211-312d-44b1-948f-93ed91cae70f] [java11|https://app.circleci.com/pipelines/github/clohfink/cassandra/9/workflows/b7488408-670f-409e-af0a-aa7a6c1ea8ff] > Emit a metric for mutations that are too large in commit logs > - > > Key: CASSANDRA-16116 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16116 > Project: Cassandra > Issue Type: Task > Components: Observability/Metrics >Reporter: Lee Tang >Assignee: Lee Tang >Priority: Low > Time Spent: 10m > Remaining Estimate: 0h > > Huge memory allocations can cause GC thrashing and a lot of write timeouts. > Having a metric in the commit log allows us to react faster and discern why > write timeouts are happening. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15823) Support for networking via identity instead of IP
[ https://issues.apache.org/jira/browse/CASSANDRA-15823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200199#comment-17200199 ] Roman Chernobelskiy edited comment on CASSANDRA-15823 at 9/22/20, 4:38 PM: --- One way to address this in kubernetes without Cassandra changes is with a sidecar that encodes the pod names into virtual IP addresses, thereby giving each node a stable ip and then looking up the ip based on the encoded hostname on outgoing connections. was (Author: rchernobelskiy): One way to address this in kubernetes is with a sidecar that encodes the pod names into virtual IP addresses, thereby giving each node a stable ip and then looking up the ip based on the encoded hostname on outgoing connections. > Support for networking via identity instead of IP > - > > Key: CASSANDRA-15823 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15823 > Project: Cassandra > Issue Type: Improvement >Reporter: Christopher Bradford >Priority: Normal > Labels: kubernetes > Attachments: consul-mesh-gateways.png, > istio-multicluster-with-gateways.svg, linkerd-service-mirroring.svg > > > TL;DR: Instead of mapping host ids to IPs, use hostnames. This allows > resolution to different IP addresses per DC that may then be forwarded to > nodes on remote networks without requiring node to node IP connectivity for > cross-dc links. > > This approach should not affect existing deployments as those could continue > to use IPs as the hostname and skip resolution. > > With orchestration platforms like Kubernetes and the usage of ephemeral > containers in environments today we should consider some changes to how we > handle the tracking of nodes and their network location. Currently we > maintain a mapping between host ids and IP addresses. > > With traditional infrastructure, if a node goes down it, usually, comes back > up with the same IP. In some environments this contract may be explicit with > virtual IPs that may move between hosts. In newer deployments, like on > Kubernetes, this contract is not possible. Pods (analogous to nodes) are > assigned an IP address at start time. Should the pod be restarted or > scheduled on a different host there is no guarantee we would have the same > IP. Cassandra is protected here as we already have logic in place to update > peers when we come up with the same host id, but a different IP address. > > There are ways to get Kubernetes to assign a specific IP per Pod. Most > recommendations involve the use of a service per pod. Communication with the > fixed service IP would automatically forward to the associated pod, > regardless of address. We _could_ use this approach, but it seems like this > would needlessly create a number of extra resources in our k8s cluster to get > around the problem. Which, to be fair, doesn't seem like much of a problem > with the aforementioned mitigations built into C*. > > So what is the _actual_ problem? *Cross-region, cross-cloud, > hybrid-deployment connectivity between pods is a pain.* This can be solved > with significant investment by those who want to deploy these types of > topologies. You can definitely configure connectivity between clouds over > dedicated connections, or VPN tunnels. With a big chunk of time insuring that > pod to pod connectivity just works even if those pods are managed by separate > control planes, but that again requires time and talent. There are a number > of edge cases to support between the ever so slight, but very important, > differences in cloud vendor networks. > > Recently there have been a number of innovations that aid in the deployment > and operation of these types of applications on Kubernetes. Service meshes > support distributed microservices running across multiple k8s cluster control > planes in disparate networks. Instead of directly connecting to IP addresses > of remote services instead they use a hostname. With this approach, hostname > traffic may then be routed to a proxy that sends traffic over the WAN > (sometimes with mTLS) to another proxy pod in the remote cluster which then > forwards the data along to the correct pod in that network. (See attached > diagrams) > > Which brings us to the point of this ticket. Instead of mapping host ids to > IPs, use hostnames (and update the underlying address periodically instead of > caching indefinitely). This allows resolution to different IP addresses per > DC (k8s cluster) that may then be forwarded to nodes (pods) on remote > networks (k8s clusters) without requiring node to node (pod to pod) IP > connectivity between them. Traditional deployments can still function like > they do today (even if operators opt to keep using IPs as
[jira] [Commented] (CASSANDRA-15823) Support for networking via identity instead of IP
[ https://issues.apache.org/jira/browse/CASSANDRA-15823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200199#comment-17200199 ] Roman Chernobelskiy commented on CASSANDRA-15823: - One way to address this in kubernetes is with a sidecar that encodes the pod names into virtual IP addresses, thereby giving each node a stable ip and then looking up the ip based on the encoded hostname on outgoing connections. > Support for networking via identity instead of IP > - > > Key: CASSANDRA-15823 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15823 > Project: Cassandra > Issue Type: Improvement >Reporter: Christopher Bradford >Priority: Normal > Labels: kubernetes > Attachments: consul-mesh-gateways.png, > istio-multicluster-with-gateways.svg, linkerd-service-mirroring.svg > > > TL;DR: Instead of mapping host ids to IPs, use hostnames. This allows > resolution to different IP addresses per DC that may then be forwarded to > nodes on remote networks without requiring node to node IP connectivity for > cross-dc links. > > This approach should not affect existing deployments as those could continue > to use IPs as the hostname and skip resolution. > > With orchestration platforms like Kubernetes and the usage of ephemeral > containers in environments today we should consider some changes to how we > handle the tracking of nodes and their network location. Currently we > maintain a mapping between host ids and IP addresses. > > With traditional infrastructure, if a node goes down it, usually, comes back > up with the same IP. In some environments this contract may be explicit with > virtual IPs that may move between hosts. In newer deployments, like on > Kubernetes, this contract is not possible. Pods (analogous to nodes) are > assigned an IP address at start time. Should the pod be restarted or > scheduled on a different host there is no guarantee we would have the same > IP. Cassandra is protected here as we already have logic in place to update > peers when we come up with the same host id, but a different IP address. > > There are ways to get Kubernetes to assign a specific IP per Pod. Most > recommendations involve the use of a service per pod. Communication with the > fixed service IP would automatically forward to the associated pod, > regardless of address. We _could_ use this approach, but it seems like this > would needlessly create a number of extra resources in our k8s cluster to get > around the problem. Which, to be fair, doesn't seem like much of a problem > with the aforementioned mitigations built into C*. > > So what is the _actual_ problem? *Cross-region, cross-cloud, > hybrid-deployment connectivity between pods is a pain.* This can be solved > with significant investment by those who want to deploy these types of > topologies. You can definitely configure connectivity between clouds over > dedicated connections, or VPN tunnels. With a big chunk of time insuring that > pod to pod connectivity just works even if those pods are managed by separate > control planes, but that again requires time and talent. There are a number > of edge cases to support between the ever so slight, but very important, > differences in cloud vendor networks. > > Recently there have been a number of innovations that aid in the deployment > and operation of these types of applications on Kubernetes. Service meshes > support distributed microservices running across multiple k8s cluster control > planes in disparate networks. Instead of directly connecting to IP addresses > of remote services instead they use a hostname. With this approach, hostname > traffic may then be routed to a proxy that sends traffic over the WAN > (sometimes with mTLS) to another proxy pod in the remote cluster which then > forwards the data along to the correct pod in that network. (See attached > diagrams) > > Which brings us to the point of this ticket. Instead of mapping host ids to > IPs, use hostnames (and update the underlying address periodically instead of > caching indefinitely). This allows resolution to different IP addresses per > DC (k8s cluster) that may then be forwarded to nodes (pods) on remote > networks (k8s clusters) without requiring node to node (pod to pod) IP > connectivity between them. Traditional deployments can still function like > they do today (even if operators opt to keep using IPs as identifiers instead > of hostnames). This proxy approach is then enabled like those we see in > service meshes. > > _Notes_ > C* already has the concept of broadcast addresses vs those which are bound on > the node. This approach _could_ be leveraged to provide the behavior we're > looking for, but then the broadcast values would need to
[jira] [Updated] (CASSANDRA-15164) Overflowed Partition Cell Histograms Can Prevent Compactions from Executing
[ https://issues.apache.org/jira/browse/CASSANDRA-15164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Lohfink updated CASSANDRA-15164: -- Reviewers: Benjamin Lerer, Chris Lohfink (was: Benjamin Lerer) > Overflowed Partition Cell Histograms Can Prevent Compactions from Executing > --- > > Key: CASSANDRA-15164 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15164 > Project: Cassandra > Issue Type: Bug > Components: CQL/Interpreter >Reporter: Ankur Jha >Assignee: Caleb Rackliffe >Priority: Urgent > Labels: compaction, partition > Fix For: 4.0-beta > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Hi, we are running 6 node Cassandra cluster in production with 3 seed node > but from last night one of our seed nodes is continuously throwing an error > like this;- > cassandra.protocol.ServerError: message="java.lang.IllegalStateException: Unable to compute ceiling for max > when histogram overflowed"> > For a cluster to be up and running I Drained this node. > Can somebody help me out with this? > > Any help or lead would be appreciated > > Note : We are using Cassandra version 3.7 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15164) Overflowed Partition Cell Histograms Can Prevent Compactions from Executing
[ https://issues.apache.org/jira/browse/CASSANDRA-15164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Caleb Rackliffe updated CASSANDRA-15164: Status: In Progress (was: Patch Available) > Overflowed Partition Cell Histograms Can Prevent Compactions from Executing > --- > > Key: CASSANDRA-15164 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15164 > Project: Cassandra > Issue Type: Bug > Components: CQL/Interpreter >Reporter: Ankur Jha >Assignee: Caleb Rackliffe >Priority: Urgent > Labels: compaction, partition > Fix For: 4.0-beta > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Hi, we are running 6 node Cassandra cluster in production with 3 seed node > but from last night one of our seed nodes is continuously throwing an error > like this;- > cassandra.protocol.ServerError: message="java.lang.IllegalStateException: Unable to compute ceiling for max > when histogram overflowed"> > For a cluster to be up and running I Drained this node. > Can somebody help me out with this? > > Any help or lead would be appreciated > > Note : We are using Cassandra version 3.7 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16135) Separate in-JVM test into smaller packages
[ https://issues.apache.org/jira/browse/CASSANDRA-16135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200189#comment-17200189 ] Alex Petrov commented on CASSANDRA-16135: - Patch: https://github.com/apache/cassandra/pull/755 > Separate in-JVM test into smaller packages > -- > > Key: CASSANDRA-16135 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16135 > Project: Cassandra > Issue Type: Task > Components: Test/dtest/java >Reporter: Alex Petrov >Priority: High > Time Spent: 10m > Remaining Estimate: 0h > > Introduce a structure similar to how tags are organised in Cassandra Jira for > corresponding in-jvm dtests to help people find a right place for their tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16135) Separate in-JVM test into smaller packages
[ https://issues.apache.org/jira/browse/CASSANDRA-16135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Petrov updated CASSANDRA-16135: Authors: Alex Petrov Test and Documentation Plan: run CI Status: Patch Available (was: Open) > Separate in-JVM test into smaller packages > -- > > Key: CASSANDRA-16135 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16135 > Project: Cassandra > Issue Type: Task > Components: Test/dtest/java >Reporter: Alex Petrov >Priority: High > Time Spent: 10m > Remaining Estimate: 0h > > Introduce a structure similar to how tags are organised in Cassandra Jira for > corresponding in-jvm dtests to help people find a right place for their tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16135) Separate in-JVM test into smaller packages
[ https://issues.apache.org/jira/browse/CASSANDRA-16135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Petrov updated CASSANDRA-16135: Change Category: Code Clarity Complexity: Normal Component/s: Test/dtest/java Priority: High (was: Normal) Status: Open (was: Triage Needed) > Separate in-JVM test into smaller packages > -- > > Key: CASSANDRA-16135 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16135 > Project: Cassandra > Issue Type: Task > Components: Test/dtest/java >Reporter: Alex Petrov >Priority: High > Time Spent: 10m > Remaining Estimate: 0h > > Introduce a structure similar to how tags are organised in Cassandra Jira for > corresponding in-jvm dtests to help people find a right place for their tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-16135) Separate in-JVM test into smaller packages
Alex Petrov created CASSANDRA-16135: --- Summary: Separate in-JVM test into smaller packages Key: CASSANDRA-16135 URL: https://issues.apache.org/jira/browse/CASSANDRA-16135 Project: Cassandra Issue Type: Task Reporter: Alex Petrov Introduce a structure similar to how tags are organised in Cassandra Jira for corresponding in-jvm dtests to help people find a right place for their tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15260) Add `allocate_tokens_for_local_replication_factor` yaml option for token allocation
[
https://issues.apache.org/jira/browse/CASSANDRA-15260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeremiah Jordan updated CASSANDRA-15260:
Summary: Add `allocate_tokens_for_local_replication_factor` yaml option for
token allocation (was: Add `allocate_tokens_for_dc_rf` yaml option for token
allocation)
> Add `allocate_tokens_for_local_replication_factor` yaml option for token
> allocation
> ---
>
> Key: CASSANDRA-15260
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15260
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Michael Semb Wever
>Priority: Normal
> Fix For: 4.0, 4.0-alpha2
>
>
> Similar to DSE's option: {{allocate_tokens_for_local_replication_factor}}
> Currently the
> [ReplicationAwareTokenAllocator|https://www.datastax.com/dev/blog/token-allocation-algorithm]
> requires a defined keyspace and a replica factor specified in the current
> datacenter.
> This is problematic in a number of ways. The real keyspace can not be used
> when adding new datacenters as, in practice, all its nodes need to be up and
> running before it has the capacity to replicate data into it. New datacenters
> (or lift-and-shifting a cluster via datacenter migration) therefore has to be
> done using a dummy keyspace that duplicates the replication strategy+factor
> of the real keyspace. This gets even more difficult come version 4.0, as the
> replica factor can not even be defined in new datacenters before those
> datacenters are up and running.
> These issues are removed by avoiding the keyspace definition and lookup, and
> presuming the replica strategy is by datacenter, ie NTS. This can be done
> with the use of an {{allocate_tokens_for_dc_rf}} option.
> It may also be of value considering whether {{allocate_tokens_for_dc_rf=3}}
> becomes the default? as this is the replication factor for the vast majority
> of datacenters in production. I suspect this would be a good improvement over
> the existing randomly generated tokens algorithm.
> Initial patch is available in
> [https://github.com/thelastpickle/cassandra/commit/fc4865b0399570e58f11215565ba17dc4a53da97]
> The patch does not remove the existing {{allocate_tokens_for_keyspace}}
> option, as that provides the codebase for handling different replication
> strategies.
>
> fyi [~blambov] [~jay.zhuang] [~chovatia.jayd...@gmail.com] [~alokamvenki]
> [~alexchueshev]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15823) Support for networking via identity instead of IP
[ https://issues.apache.org/jira/browse/CASSANDRA-15823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-15823: -- Labels: kubernetes (was: ) > Support for networking via identity instead of IP > - > > Key: CASSANDRA-15823 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15823 > Project: Cassandra > Issue Type: Improvement >Reporter: Christopher Bradford >Priority: Normal > Labels: kubernetes > Attachments: consul-mesh-gateways.png, > istio-multicluster-with-gateways.svg, linkerd-service-mirroring.svg > > > TL;DR: Instead of mapping host ids to IPs, use hostnames. This allows > resolution to different IP addresses per DC that may then be forwarded to > nodes on remote networks without requiring node to node IP connectivity for > cross-dc links. > > This approach should not affect existing deployments as those could continue > to use IPs as the hostname and skip resolution. > > With orchestration platforms like Kubernetes and the usage of ephemeral > containers in environments today we should consider some changes to how we > handle the tracking of nodes and their network location. Currently we > maintain a mapping between host ids and IP addresses. > > With traditional infrastructure, if a node goes down it, usually, comes back > up with the same IP. In some environments this contract may be explicit with > virtual IPs that may move between hosts. In newer deployments, like on > Kubernetes, this contract is not possible. Pods (analogous to nodes) are > assigned an IP address at start time. Should the pod be restarted or > scheduled on a different host there is no guarantee we would have the same > IP. Cassandra is protected here as we already have logic in place to update > peers when we come up with the same host id, but a different IP address. > > There are ways to get Kubernetes to assign a specific IP per Pod. Most > recommendations involve the use of a service per pod. Communication with the > fixed service IP would automatically forward to the associated pod, > regardless of address. We _could_ use this approach, but it seems like this > would needlessly create a number of extra resources in our k8s cluster to get > around the problem. Which, to be fair, doesn't seem like much of a problem > with the aforementioned mitigations built into C*. > > So what is the _actual_ problem? *Cross-region, cross-cloud, > hybrid-deployment connectivity between pods is a pain.* This can be solved > with significant investment by those who want to deploy these types of > topologies. You can definitely configure connectivity between clouds over > dedicated connections, or VPN tunnels. With a big chunk of time insuring that > pod to pod connectivity just works even if those pods are managed by separate > control planes, but that again requires time and talent. There are a number > of edge cases to support between the ever so slight, but very important, > differences in cloud vendor networks. > > Recently there have been a number of innovations that aid in the deployment > and operation of these types of applications on Kubernetes. Service meshes > support distributed microservices running across multiple k8s cluster control > planes in disparate networks. Instead of directly connecting to IP addresses > of remote services instead they use a hostname. With this approach, hostname > traffic may then be routed to a proxy that sends traffic over the WAN > (sometimes with mTLS) to another proxy pod in the remote cluster which then > forwards the data along to the correct pod in that network. (See attached > diagrams) > > Which brings us to the point of this ticket. Instead of mapping host ids to > IPs, use hostnames (and update the underlying address periodically instead of > caching indefinitely). This allows resolution to different IP addresses per > DC (k8s cluster) that may then be forwarded to nodes (pods) on remote > networks (k8s clusters) without requiring node to node (pod to pod) IP > connectivity between them. Traditional deployments can still function like > they do today (even if operators opt to keep using IPs as identifiers instead > of hostnames). This proxy approach is then enabled like those we see in > service meshes. > > _Notes_ > C* already has the concept of broadcast addresses vs those which are bound on > the node. This approach _could_ be leveraged to provide the behavior we're > looking for, but then the broadcast values would need to be pre-computed > _*and match*_ across all k8s control planes. By using hostnames the > underlying IP address does not matter and will most likely be different in > each cluster. > > I recognize the title may be a bit misleading as we would
[jira] [Updated] (CASSANDRA-15010) Allow subnet configuration for in-jvm distributed tests
[ https://issues.apache.org/jira/browse/CASSANDRA-15010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Petrov updated CASSANDRA-15010: Resolution: Fixed Status: Resolved (was: Open) This is already implemented. > Allow subnet configuration for in-jvm distributed tests > --- > > Key: CASSANDRA-15010 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15010 > Project: Cassandra > Issue Type: Improvement >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Normal > > Allow subnet configuration for in-jvm distributed tests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14944) Tombstones are not removed if bloom filter is turned off
[
https://issues.apache.org/jira/browse/CASSANDRA-14944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov reassigned CASSANDRA-14944:
---
Assignee: (was: Alex Petrov)
> Tombstones are not removed if bloom filter is turned off
>
>
> Key: CASSANDRA-14944
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14944
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
>Reporter: Vitalii Ishchenko
>Priority: Normal
>
> Well actually they are deleted, but not always even though they should,
> because check is done using Index of overlapped tables
> When purge evaluator is constructed we are checking overlapping tables using
> bloom filter, but when it is disabled we are checking against index, but if
> condition is not properly constructed and we still check bloom filter which
> is AlwaysPresentFilter and every overlapping sstable is used to get
> minTimestamp and tombstones, that have their timestamp >= minTimestamp won't
> be deleted
> {code:java}
> if (sstable.getBloomFilter() instanceof AlwaysPresentFilter &&
> sstable.getPosition(key, SSTableReader.Operator.EQ, false) != null
> || sstable.getBloomFilter().isPresent(key))
> {
> {code}
> Should be something like this
> {code:java}
> boolean mightBePresentInTable = sstable.getBloomFilter() instanceof
> AlwaysPresentFilter ? sstable.getPosition(key, SSTableReader.Operator.EQ,
> false) != null : sstable.getBloomFilter().isPresent(key)
> {code}
> Code pointers in 3.11
> [https://github.com/apache/cassandra/blob/08363afa5354c00a7ecd62fe273c392a678db28a/src/java/org/apache/cassandra/db/compaction/CompactionController.java#L274]
> and 4.0
> [https://github.com/apache/cassandra/blob/11384c3279a66e6c0fb7861e2b188b25e963580f/src/java/org/apache/cassandra/db/compaction/CompactionController.java#L281]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-13105) Multi-index query incorrectly returns 0 rows
[
https://issues.apache.org/jira/browse/CASSANDRA-13105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov reassigned CASSANDRA-13105:
---
Assignee: (was: Alex Petrov)
> Multi-index query incorrectly returns 0 rows
>
>
> Key: CASSANDRA-13105
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13105
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/SASI
> Environment: 3.9.0 on linux & osx
>Reporter: Voytek Jarnot
>Priority: Normal
>
> Setup:
> {code}
> create table test1(id1 text PRIMARY KEY, val1 text, val2 text);
> create custom index test1_idx_val1 on test1(val1) using
> 'org.apache.cassandra.index.sasi.SASIIndex';
> create custom index test1_idx_val2 on test1(val2) using
> 'org.apache.cassandra.index.sasi.SASIIndex';
> insert into test1(id1, val1, val2) values ('1', '1val1', '1val2');
> insert into test1(id1, val1, val2) values ('2', '~~', '2val2');
> {code}
> Queries:
> {code}
> (1) select * from test1 where val1 = '~~';
> (2) select * from test1 where val1 < '~~' allow filtering;
> (3) select * from test1 where val2 = '1val2';
> (4) select * from test1 where val1 < '~~' and val2 = '1val2' allow filtering;
> {code}
> 1, 2, and 3 all work correctly. 4 does not work.
> 2, 3, and 4 should return the same row (id1='1'); 2 and 3 do, 4 returns 0
> rows.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-12674) [SASI] Confusing AND/OR semantics for StandardAnalyzer
[
https://issues.apache.org/jira/browse/CASSANDRA-12674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov reassigned CASSANDRA-12674:
---
Assignee: (was: Alex Petrov)
> [SASI] Confusing AND/OR semantics for StandardAnalyzer
> ---
>
> Key: CASSANDRA-12674
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12674
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/SASI
> Environment: Cassandra 3.7
>Reporter: DuyHai Doan
>Priority: Normal
>
> {code:sql}
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 3.4.2 | Native protocol v4]
> Use HELP for help.
> cqlsh> use test;
> cqlsh:test> CREATE TABLE sasi_bug(id int, clustering int, val text, PRIMARY
> KEY((id), clustering));
> cqlsh:test> CREATE CUSTOM INDEX ON sasi_bug(val) USING
> 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {
> 'mode': 'CONTAINS',
> 'analyzer_class':
> 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer',
> 'analyzed': 'true'};
> //1st example SAME PARTITION KEY
> cqlsh:test> INSERT INTO sasi_bug(id, clustering , val ) VALUES(1, 1,
> 'homeworker');
> cqlsh:test> INSERT INTO sasi_bug(id, clustering , val ) VALUES(1, 2,
> 'hardworker');
> cqlsh:test> SELECT * FROM sasi_bug WHERE val LIKE '%work home%';
> id | clustering | val
> ++
> 1 | 1 | homeworker
> 1 | 2 | hardworker
> (2 rows)
> //2nd example DIFFERENT PARTITION KEY
> cqlsh:test> INSERT INTO sasi_bug(id, clustering, val) VALUES(10, 1,
> 'speedrun');
> cqlsh:test> INSERT INTO sasi_bug(id, clustering, val) VALUES(11, 1,
> 'longrun');
> cqlsh:test> SELECT * FROM sasi_bug WHERE val LIKE '%long run%';
> id | clustering | val
> ++-
> 11 | 1 | longrun
> (1 rows)
> {code}
> In the 1st example, both rows belong to the same partition so SASI returns
> both values. Indeed {{LIKE '%work home%'}} means {{contains 'work' OR
> 'home'}} so the result makes sense
> In the 2nd example, only one row is returned whereas we expect 2 rows because
> {{LIKE '%long run%'}} means {{contains 'long' OR 'run'}} so *speedrun* should
> be returned too.
> So where is the problem ? Explanation:
> When there is only 1 predicate, the root operation type is an *AND*:
> {code:java|title=QueryPlan}
> private Operation analyze()
> {
> try
> {
> Operation.Builder and = new Operation.Builder(OperationType.AND,
> controller);
> controller.getExpressions().forEach(and::add);
> return and.complete();
> }
>...
> }
> {code}
> During the parsing of {{LIKE '%long run%'}}, SASI creates 2 expressions for
> the searched term: {{long}} and {{run}}, which corresponds to an *OR* logic.
> However, this piece of code just ruins the *OR* logic:
> {code:java|title=Operation}
> public Operation complete()
> {
> if (!expressions.isEmpty())
> {
> ListMultimap
> analyzedExpressions = analyzeGroup(controller, op, expressions);
> RangeIterator.Builder range =
> controller.getIndexes(op, analyzedExpressions.values());
> ...
> }
> {code}
> As you can see, we blindly take all the *values* of the MultiMap (which
> contains a single entry for the {{val}} column with 2 expressions) and pass
> it to {{controller.getIndexes(...)}}
> {code:java|title=QueryController}
> public RangeIterator.Builder getIndexes(OperationType op,
> Collection expressions)
> {
> if (resources.containsKey(expressions))
> throw new IllegalArgumentException("Can't process the same
> expressions multiple times.");
> RangeIterator.Builder builder = op == OperationType.OR
> ? RangeUnionIterator. Token>builder()
> :
> RangeIntersectionIterator.builder();
> ...
> }
> {code}
> And because the root operation has *AND* type, the
> {{RangeIntersectionIterator}} will be used on both expressions {{long}} and
> {{run}}.
> So when data belong to different partitions, we have the *AND* logic that
> applies and eliminates _speedrun_
> When data belong to the same partition but different row, the
> {{RangeIntersectionIterator}} returns a single partition and then the rows
> are filtered further by {{operationTree.satisfiedBy}} and the results are
> correct
> {code:java|title=QueryPlan}
> while (currentKeys.hasNext())
> {
> DecoratedKey key = currentKeys.next();
> if (!keyRange.right.isMinimum() &&
> keyRange.right.compareTo(key) < 0)
> return endOfData();
> try
[jira] [Assigned] (CASSANDRA-12734) Materialized View schema file for snapshots created as tables
[ https://issues.apache.org/jira/browse/CASSANDRA-12734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Petrov reassigned CASSANDRA-12734: --- Assignee: (was: Alex Petrov) > Materialized View schema file for snapshots created as tables > - > > Key: CASSANDRA-12734 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12734 > Project: Cassandra > Issue Type: Bug > Components: Feature/Materialized Views, Legacy/Tools >Reporter: Hau Phan >Priority: Normal > Fix For: 3.0.x > > > The materialized view schema file that gets created and stored with the > sstables is created as a table instead of a materialized view. > Can the materialized view be created and added to the corresponding table's > schema file? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16109) Don't adjust nodeCount when setting node id topology in in-jvm dtests
[ https://issues.apache.org/jira/browse/CASSANDRA-16109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199931#comment-17199931 ] Marcus Eriksson edited comment on CASSANDRA-16109 at 9/22/20, 2:20 PM: --- published the 0.0.5 snapshot and ran the jvm dtests, this branch includes updates for CASSANDRA-16101, CASSANDRA-15386 and CASSANDRA-16109: [2.2|https://github.com/krummas/cassandra/commits/dtest0.0.5-snapshot-2.2] [circle|https://app.circleci.com/pipelines/github/krummas/cassandra/524/workflows/9693d0d8-be85-40a1-9ce8-2eee6ac53b83] [3.0|https://github.com/krummas/cassandra/commits/dtest0.0.5-snapshot-3.0] [circle|https://app.circleci.com/pipelines/github/krummas/cassandra/526/workflows/ba3e5ab6-ece7-4a74-a246-29a6b3f0f204] [3.11|https://github.com/krummas/cassandra/commits/dtest0.0.5-snapshot-3.11] [circle|https://app.circleci.com/pipelines/github/krummas/cassandra/527/workflows/510bb3fd-66f0-4732-8c64-32c8274f61c7] [trunk|https://github.com/krummas/cassandra/commits/dtest0.0.5-snapshot-trunk] [circle|https://app.circleci.com/pipelines/github/krummas/cassandra/534/workflows/e6c198ca-9482-46c7-82d1-c91d4cfd9e58] was (Author: krummas): published the 0.0.5 snapshot and ran the jvm dtests, this branch includes updates for CASSANDRA-16101, CASSANDRA-15386 and CASSANDRA-16109: [2.2|https://github.com/krummas/cassandra/commits/dtest0.0.5-snapshot-2.2] [circle|https://app.circleci.com/pipelines/github/krummas/cassandra/524/workflows/9693d0d8-be85-40a1-9ce8-2eee6ac53b83] [3.0|https://github.com/krummas/cassandra/commits/dtest0.0.5-snapshot-3.0] [circle|https://app.circleci.com/pipelines/github/krummas/cassandra/526/workflows/ba3e5ab6-ece7-4a74-a246-29a6b3f0f204] [3.11|https://github.com/krummas/cassandra/commits/dtest0.0.5-snapshot-3.11] [circle|https://app.circleci.com/pipelines/github/krummas/cassandra/527/workflows/510bb3fd-66f0-4732-8c64-32c8274f61c7] [trunk|https://github.com/krummas/cassandra/commits/dtest0.0.5-snapshot-trunk] [circle|https://app.circleci.com/pipelines/github/krummas/cassandra/525/workflows/20b8d7e7-1bb9-414d-bce1-b5857473c6a1] > Don't adjust nodeCount when setting node id topology in in-jvm dtests > - > > Key: CASSANDRA-16109 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16109 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest/java >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Low > Labels: pull-request-available > > We update the node count when setting the node id topology in in-jvm dtests, > this should only happen if node count is smaller than the node id topology, > otherwise bootstrap tests error out. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14151) [TRUNK] TestRepair.test_dead_sync_initiator failed due to ERROR in logs "SSTableTidier ran with no existing data file for an sstable that was not new"
[
https://issues.apache.org/jira/browse/CASSANDRA-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-14151:
Resolution: Cannot Reproduce
Status: Resolved (was: Open)
guess this has been fixed, maybe CASSANDRA-15963
> [TRUNK] TestRepair.test_dead_sync_initiator failed due to ERROR in logs
> "SSTableTidier ran with no existing data file for an sstable that was not new"
> --
>
> Key: CASSANDRA-14151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14151
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Testing
>Reporter: Michael Kjellman
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 3.11.x, 4.x
>
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log,
> stdout-novnodes.txt
>
>
> TestRepair.test_dead_sync_initiator failed due to finding the following
> unexpected error in the node's logs:
> {code}
> ERROR [NonPeriodicTasks:1] 2018-01-06 03:38:50,229 LogTransaction.java:347 -
> SSTableTidier ran with no existing data file for an sstable that was not new
> {code}
> If this is "okay/expected" behavior we should change the log level to
> something different (which will fix the test) or if it's an actual bug use
> this JIRA to fix it. I've attached all of the logs from all 3 instances from
> the dtest run that hit this failure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14151) [TRUNK] TestRepair.test_dead_sync_initiator failed due to ERROR in logs "SSTableTidier ran with no existing data file for an sstable that was not new"
[
https://issues.apache.org/jira/browse/CASSANDRA-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-14151:
Status: Open (was: Patch Available)
> [TRUNK] TestRepair.test_dead_sync_initiator failed due to ERROR in logs
> "SSTableTidier ran with no existing data file for an sstable that was not new"
> --
>
> Key: CASSANDRA-14151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14151
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Testing
>Reporter: Michael Kjellman
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 3.11.x, 4.x
>
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log,
> stdout-novnodes.txt
>
>
> TestRepair.test_dead_sync_initiator failed due to finding the following
> unexpected error in the node's logs:
> {code}
> ERROR [NonPeriodicTasks:1] 2018-01-06 03:38:50,229 LogTransaction.java:347 -
> SSTableTidier ran with no existing data file for an sstable that was not new
> {code}
> If this is "okay/expected" behavior we should change the log level to
> something different (which will fix the test) or if it's an actual bug use
> this JIRA to fix it. I've attached all of the logs from all 3 instances from
> the dtest run that hit this failure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15348) Harry: generator library and extensible framework for fuzz testing Apache Cassandra
[ https://issues.apache.org/jira/browse/CASSANDRA-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200093#comment-17200093 ] Alex Petrov edited comment on CASSANDRA-15348 at 9/22/20, 1:54 PM: --- Committed in [1d7f66e2d5b39702ff218cd36e0b9043b0d47cf1|https://github.com/apache/cassandra-harry/commit/1d7f66e2d5b39702ff218cd36e0b9043b0d47cf1] was (Author: ifesdjeen): Committed in [1d7f66e2d5b39702ff218cd36e0b9043b0d47cf1|https://github.com/apache/cassandra-harry/commit/1d7f66e2d5b39702ff218cd36e0b9043b0d47cf1 > Harry: generator library and extensible framework for fuzz testing Apache > Cassandra > --- > > Key: CASSANDRA-15348 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15348 > Project: Cassandra > Issue Type: New Feature >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Normal > Fix For: 4.0-beta > > > h2. Description: > This ticket introduces Harry, a component for fuzz testing and verification > of the Apache Cassandra clusters at scale. > h2. Motivation: > Current testing tooling largely tests for common- and edge-cases, and most of > the tests use predefined datasets. Property-based tests can help explore a > broader range of states, but often require either a complex model or a large > state to test against. > h2. What problems Harry solves: > Harry allows to run tests that are able to validate state of both dense nodes > (to test local read-write path) and large clusters (to test distributed > read-write path), and do it efficiently. Main goals, and what sets it apart > from the other testing tools is: > * The state required for verification should remain as compact as possible. > * The verification process itself should be as performant as possible. > * Ideally, we'd want a way to verify database state while _continuing_ > running state change queries against it. > h2. What Harry does: > To achieve this, Harry defines a model that holds the state of the database, > generators that produce reproducible, pseudo-random schemas, mutations, and > queries, and a validator that asserts the correctness of the model following > execution of generated traffic. > h2. Harry consists of multiple reusable components: > * Generator library: how to create a library of invertible, order-preserving > generators for simple and composite data types. > * Model and checker: how to use the properties of generators to validate the > output of an eventually-consistent database in a linear time. > * Runner library: how to create a scheme for reproducible runs, despite the > concurrent nature of database and fuzzer itself. > h2. Short and somewhat approximate description of how Harry achieves this: > Generation and validation define strict mathematical relations between the > generated values and pseudorandom numbers they were generated from. Using > these properties, we can store minimal state and check if these properties > hold during validation. > Since Cassandra stores data in rows, we should be able to "inflate" data to > insert a row into the database from a single number we call _descriptor_. > Each value in the row read from the database can be "deflated" back to the > descriptor it was generated from. This way, to precisely verify the state of > the row, we only need to know the descriptor it was generated from and a > timestamp at which it was inserted. > Similarly, keys for the inserted row can be "inflated" from a single 64-bit > integer, and then "deflated" back to it. To efficiently search for keys, > while allowing range scans, our generation scheme preserves the order of the > original 64-bit integer. Every pair of keys generated from two 64-bit > integers would sort the same way as these integers. > This way, in order to validate a state of the range of rows queried from the > database, it is sufficient to "deflate" its key and data values, use deflated > 64-bit key representation to find all descriptors these rows were generated > from, and ensure that the given sequence of descriptors could have resulted > in the state that database has responded with. > Using this scheme, we keep a minimum possible amount of data per row, can > efficiently generate the data, and backtrack values to the numbers they were > generated from. Most of the time, we operate on 64-bit integer values and > only use "inflated" objects when running queries against database state, > minimizing the amount of required memory. > h2. Name: > Harry (verb). > According to Marriam-Webster: > * to torment by or as if by constant attack > * persistently carry out attacks on (an enemy or an enemy's territory) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CASSANDRA-15348) Harry: generator library and extensible framework for fuzz testing Apache Cassandra
[ https://issues.apache.org/jira/browse/CASSANDRA-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Petrov updated CASSANDRA-15348: Resolution: Fixed Status: Resolved (was: Triage Needed) > Harry: generator library and extensible framework for fuzz testing Apache > Cassandra > --- > > Key: CASSANDRA-15348 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15348 > Project: Cassandra > Issue Type: New Feature >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Normal > Fix For: 4.0-beta > > > h2. Description: > This ticket introduces Harry, a component for fuzz testing and verification > of the Apache Cassandra clusters at scale. > h2. Motivation: > Current testing tooling largely tests for common- and edge-cases, and most of > the tests use predefined datasets. Property-based tests can help explore a > broader range of states, but often require either a complex model or a large > state to test against. > h2. What problems Harry solves: > Harry allows to run tests that are able to validate state of both dense nodes > (to test local read-write path) and large clusters (to test distributed > read-write path), and do it efficiently. Main goals, and what sets it apart > from the other testing tools is: > * The state required for verification should remain as compact as possible. > * The verification process itself should be as performant as possible. > * Ideally, we'd want a way to verify database state while _continuing_ > running state change queries against it. > h2. What Harry does: > To achieve this, Harry defines a model that holds the state of the database, > generators that produce reproducible, pseudo-random schemas, mutations, and > queries, and a validator that asserts the correctness of the model following > execution of generated traffic. > h2. Harry consists of multiple reusable components: > * Generator library: how to create a library of invertible, order-preserving > generators for simple and composite data types. > * Model and checker: how to use the properties of generators to validate the > output of an eventually-consistent database in a linear time. > * Runner library: how to create a scheme for reproducible runs, despite the > concurrent nature of database and fuzzer itself. > h2. Short and somewhat approximate description of how Harry achieves this: > Generation and validation define strict mathematical relations between the > generated values and pseudorandom numbers they were generated from. Using > these properties, we can store minimal state and check if these properties > hold during validation. > Since Cassandra stores data in rows, we should be able to "inflate" data to > insert a row into the database from a single number we call _descriptor_. > Each value in the row read from the database can be "deflated" back to the > descriptor it was generated from. This way, to precisely verify the state of > the row, we only need to know the descriptor it was generated from and a > timestamp at which it was inserted. > Similarly, keys for the inserted row can be "inflated" from a single 64-bit > integer, and then "deflated" back to it. To efficiently search for keys, > while allowing range scans, our generation scheme preserves the order of the > original 64-bit integer. Every pair of keys generated from two 64-bit > integers would sort the same way as these integers. > This way, in order to validate a state of the range of rows queried from the > database, it is sufficient to "deflate" its key and data values, use deflated > 64-bit key representation to find all descriptors these rows were generated > from, and ensure that the given sequence of descriptors could have resulted > in the state that database has responded with. > Using this scheme, we keep a minimum possible amount of data per row, can > efficiently generate the data, and backtrack values to the numbers they were > generated from. Most of the time, we operate on 64-bit integer values and > only use "inflated" objects when running queries against database state, > minimizing the amount of required memory. > h2. Name: > Harry (verb). > According to Marriam-Webster: > * to torment by or as if by constant attack > * persistently carry out attacks on (an enemy or an enemy's territory) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15348) Harry: generator library and extensible framework for fuzz testing Apache Cassandra
[ https://issues.apache.org/jira/browse/CASSANDRA-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200093#comment-17200093 ] Alex Petrov commented on CASSANDRA-15348: - Committed in [1d7f66e2d5b39702ff218cd36e0b9043b0d47cf1 |https://github.com/apache/cassandra-harry/commit/1d7f66e2d5b39702ff218cd36e0b9043b0d47cf1 > Harry: generator library and extensible framework for fuzz testing Apache > Cassandra > --- > > Key: CASSANDRA-15348 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15348 > Project: Cassandra > Issue Type: New Feature >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Normal > Fix For: 4.0-beta > > > h2. Description: > This ticket introduces Harry, a component for fuzz testing and verification > of the Apache Cassandra clusters at scale. > h2. Motivation: > Current testing tooling largely tests for common- and edge-cases, and most of > the tests use predefined datasets. Property-based tests can help explore a > broader range of states, but often require either a complex model or a large > state to test against. > h2. What problems Harry solves: > Harry allows to run tests that are able to validate state of both dense nodes > (to test local read-write path) and large clusters (to test distributed > read-write path), and do it efficiently. Main goals, and what sets it apart > from the other testing tools is: > * The state required for verification should remain as compact as possible. > * The verification process itself should be as performant as possible. > * Ideally, we'd want a way to verify database state while _continuing_ > running state change queries against it. > h2. What Harry does: > To achieve this, Harry defines a model that holds the state of the database, > generators that produce reproducible, pseudo-random schemas, mutations, and > queries, and a validator that asserts the correctness of the model following > execution of generated traffic. > h2. Harry consists of multiple reusable components: > * Generator library: how to create a library of invertible, order-preserving > generators for simple and composite data types. > * Model and checker: how to use the properties of generators to validate the > output of an eventually-consistent database in a linear time. > * Runner library: how to create a scheme for reproducible runs, despite the > concurrent nature of database and fuzzer itself. > h2. Short and somewhat approximate description of how Harry achieves this: > Generation and validation define strict mathematical relations between the > generated values and pseudorandom numbers they were generated from. Using > these properties, we can store minimal state and check if these properties > hold during validation. > Since Cassandra stores data in rows, we should be able to "inflate" data to > insert a row into the database from a single number we call _descriptor_. > Each value in the row read from the database can be "deflated" back to the > descriptor it was generated from. This way, to precisely verify the state of > the row, we only need to know the descriptor it was generated from and a > timestamp at which it was inserted. > Similarly, keys for the inserted row can be "inflated" from a single 64-bit > integer, and then "deflated" back to it. To efficiently search for keys, > while allowing range scans, our generation scheme preserves the order of the > original 64-bit integer. Every pair of keys generated from two 64-bit > integers would sort the same way as these integers. > This way, in order to validate a state of the range of rows queried from the > database, it is sufficient to "deflate" its key and data values, use deflated > 64-bit key representation to find all descriptors these rows were generated > from, and ensure that the given sequence of descriptors could have resulted > in the state that database has responded with. > Using this scheme, we keep a minimum possible amount of data per row, can > efficiently generate the data, and backtrack values to the numbers they were > generated from. Most of the time, we operate on 64-bit integer values and > only use "inflated" objects when running queries against database state, > minimizing the amount of required memory. > h2. Name: > Harry (verb). > According to Marriam-Webster: > * to torment by or as if by constant attack > * persistently carry out attacks on (an enemy or an enemy's territory) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15348) Harry: generator library and extensible framework for fuzz testing Apache Cassandra
[ https://issues.apache.org/jira/browse/CASSANDRA-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200093#comment-17200093 ] Alex Petrov edited comment on CASSANDRA-15348 at 9/22/20, 1:53 PM: --- Committed in [1d7f66e2d5b39702ff218cd36e0b9043b0d47cf1|https://github.com/apache/cassandra-harry/commit/1d7f66e2d5b39702ff218cd36e0b9043b0d47cf1 was (Author: ifesdjeen): Committed in [1d7f66e2d5b39702ff218cd36e0b9043b0d47cf1 |https://github.com/apache/cassandra-harry/commit/1d7f66e2d5b39702ff218cd36e0b9043b0d47cf1 > Harry: generator library and extensible framework for fuzz testing Apache > Cassandra > --- > > Key: CASSANDRA-15348 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15348 > Project: Cassandra > Issue Type: New Feature >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Normal > Fix For: 4.0-beta > > > h2. Description: > This ticket introduces Harry, a component for fuzz testing and verification > of the Apache Cassandra clusters at scale. > h2. Motivation: > Current testing tooling largely tests for common- and edge-cases, and most of > the tests use predefined datasets. Property-based tests can help explore a > broader range of states, but often require either a complex model or a large > state to test against. > h2. What problems Harry solves: > Harry allows to run tests that are able to validate state of both dense nodes > (to test local read-write path) and large clusters (to test distributed > read-write path), and do it efficiently. Main goals, and what sets it apart > from the other testing tools is: > * The state required for verification should remain as compact as possible. > * The verification process itself should be as performant as possible. > * Ideally, we'd want a way to verify database state while _continuing_ > running state change queries against it. > h2. What Harry does: > To achieve this, Harry defines a model that holds the state of the database, > generators that produce reproducible, pseudo-random schemas, mutations, and > queries, and a validator that asserts the correctness of the model following > execution of generated traffic. > h2. Harry consists of multiple reusable components: > * Generator library: how to create a library of invertible, order-preserving > generators for simple and composite data types. > * Model and checker: how to use the properties of generators to validate the > output of an eventually-consistent database in a linear time. > * Runner library: how to create a scheme for reproducible runs, despite the > concurrent nature of database and fuzzer itself. > h2. Short and somewhat approximate description of how Harry achieves this: > Generation and validation define strict mathematical relations between the > generated values and pseudorandom numbers they were generated from. Using > these properties, we can store minimal state and check if these properties > hold during validation. > Since Cassandra stores data in rows, we should be able to "inflate" data to > insert a row into the database from a single number we call _descriptor_. > Each value in the row read from the database can be "deflated" back to the > descriptor it was generated from. This way, to precisely verify the state of > the row, we only need to know the descriptor it was generated from and a > timestamp at which it was inserted. > Similarly, keys for the inserted row can be "inflated" from a single 64-bit > integer, and then "deflated" back to it. To efficiently search for keys, > while allowing range scans, our generation scheme preserves the order of the > original 64-bit integer. Every pair of keys generated from two 64-bit > integers would sort the same way as these integers. > This way, in order to validate a state of the range of rows queried from the > database, it is sufficient to "deflate" its key and data values, use deflated > 64-bit key representation to find all descriptors these rows were generated > from, and ensure that the given sequence of descriptors could have resulted > in the state that database has responded with. > Using this scheme, we keep a minimum possible amount of data per row, can > efficiently generate the data, and backtrack values to the numbers they were > generated from. Most of the time, we operate on 64-bit integer values and > only use "inflated" objects when running queries against database state, > minimizing the amount of required memory. > h2. Name: > Harry (verb). > According to Marriam-Webster: > * to torment by or as if by constant attack > * persistently carry out attacks on (an enemy or an enemy's territory) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[cassandra-harry] branch master updated: Update README.md
This is an automated email from the ASF dual-hosted git repository. ifesdjeen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/cassandra-harry.git The following commit(s) were added to refs/heads/master by this push: new 6adedb0 Update README.md 6adedb0 is described below commit 6adedb0c1da9b07a3dcf259959bd01ebd6d94c79 Author: Scott Hirleman AuthorDate: Mon Sep 21 14:22:03 2020 -0700 Update README.md In What's Missing section: Removed the repeated bullet of * Partition deletions are not implemented Removed extra "o" from * Exhaustive checker should use more precise information from data tracker, not just watermarks --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index 69a12b1..ec93b22 100644 --- a/README.md +++ b/README.md @@ -499,7 +499,6 @@ Harry is by no means feature-complete. Main things that are missing are: * Runner and scheduler are rather rudimentary and require significant rework and proper scheduling * TTL is not supported * Some SELECT queries are not supported: `LIMIT`, `IN`, `GROUP BY`, token range queries - * Partition deletions are not implemented * Pagination is not implemented Some things, even though are implemented, can be improved or optimized: @@ -510,7 +509,7 @@ Some things, even though are implemented, can be improved or optimized: off-heap data structure * Exhaustive checker can be significantly optimized * Harry shouldn't rely on java-driver for query generation - * Exhaustive checker shouold use more precise information from data tracker, not + * Exhaustive checker should use more precise information from data tracker, not just watermarks * Decision-making about _when_ we visit partitions and/or rows should be improved - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15029) Fix flaky CancelCompactionsTest
[ https://issues.apache.org/jira/browse/CASSANDRA-15029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15029: Status: Open (was: Patch Available) > Fix flaky CancelCompactionsTest > --- > > Key: CASSANDRA-15029 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15029 > Project: Cassandra > Issue Type: Bug >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 4.x > > > Have seen CancelCompactionsTest be flaky locally, fix it -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15029) Fix flaky CancelCompactionsTest
[ https://issues.apache.org/jira/browse/CASSANDRA-15029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15029: Resolution: Duplicate Status: Resolved (was: Open) > Fix flaky CancelCompactionsTest > --- > > Key: CASSANDRA-15029 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15029 > Project: Cassandra > Issue Type: Bug >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 4.x > > > Have seen CancelCompactionsTest be flaky locally, fix it -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15164) Overflowed Partition Cell Histograms Can Prevent Compactions from Executing
[ https://issues.apache.org/jira/browse/CASSANDRA-15164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Lerer updated CASSANDRA-15164: --- Reviewers: Benjamin Lerer > Overflowed Partition Cell Histograms Can Prevent Compactions from Executing > --- > > Key: CASSANDRA-15164 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15164 > Project: Cassandra > Issue Type: Bug > Components: CQL/Interpreter >Reporter: Ankur Jha >Assignee: Caleb Rackliffe >Priority: Urgent > Labels: compaction, partition > Fix For: 4.0-beta > > Time Spent: 1h > Remaining Estimate: 0h > > Hi, we are running 6 node Cassandra cluster in production with 3 seed node > but from last night one of our seed nodes is continuously throwing an error > like this;- > cassandra.protocol.ServerError: message="java.lang.IllegalStateException: Unable to compute ceiling for max > when histogram overflowed"> > For a cluster to be up and running I Drained this node. > Can somebody help me out with this? > > Any help or lead would be appreciated > > Note : We are using Cassandra version 3.7 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-12126) CAS Reads Inconsistencies
[ https://issues.apache.org/jira/browse/CASSANDRA-12126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Lerer updated CASSANDRA-12126: --- Reviewers: Benjamin Lerer, Benjamin Lerer (was: Benjamin Lerer) Benjamin Lerer, Benjamin Lerer (was: Benjamin Lerer) Status: Review In Progress (was: Patch Available) > CAS Reads Inconsistencies > -- > > Key: CASSANDRA-12126 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12126 > Project: Cassandra > Issue Type: Bug > Components: Feature/Lightweight Transactions, Legacy/Coordination >Reporter: Sankalp Kohli >Assignee: Sylvain Lebresne >Priority: Normal > Labels: LWT, pull-request-available > Fix For: 3.0.x, 3.11.x, 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > While looking at the CAS code in Cassandra, I found a potential issue with > CAS Reads. Here is how it can happen with RF=3 > 1) You issue a CAS Write and it fails in the propose phase. A machine replies > true to a propose and saves the commit in accepted filed. The other two > machines B and C does not get to the accept phase. > Current state is that machine A has this commit in paxos table as accepted > but not committed and B and C does not. > 2) Issue a CAS Read and it goes to only B and C. You wont be able to read the > value written in step 1. This step is as if nothing is inflight. > 3) Issue another CAS Read and it goes to A and B. Now we will discover that > there is something inflight from A and will propose and commit it with the > current ballot. Now we can read the value written in step 1 as part of this > CAS read. > If we skip step 3 and instead run step 4, we will never learn about value > written in step 1. > 4. Issue a CAS Write and it involves only B and C. This will succeed and > commit a different value than step 1. Step 1 value will never be seen again > and was never seen before. > If you read the Lamport “paxos made simple” paper and read section 2.3. It > talks about this issue which is how learners can find out if majority of the > acceptors have accepted the proposal. > In step 3, it is correct that we propose the value again since we dont know > if it was accepted by majority of acceptors. When we ask majority of > acceptors, and more than one acceptors but not majority has something in > flight, we have no way of knowing if it is accepted by majority of acceptors. > So this behavior is correct. > However we need to fix step 2, since it caused reads to not be linearizable > with respect to writes and other reads. In this case, we know that majority > of acceptors have no inflight commit which means we have majority that > nothing was accepted by majority. I think we should run a propose step here > with empty commit and that will cause write written in step 1 to not be > visible ever after. > With this fix, we will either see data written in step 1 on next serial read > or will never see it which is what we want. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16102) Build target for shaded in-JVM dtest jar
[ https://issues.apache.org/jira/browse/CASSANDRA-16102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199951#comment-17199951 ] Alex Petrov commented on CASSANDRA-16102: - Thank you for taking a look [~dcapwell]. I've re-triggered no-vnode once again. Committed to trunk with [fb49ab2b12bf813697971b41fe47ac11f4a240c0|https://github.com/apache/cassandra/commit/fb49ab2b12bf813697971b41fe47ac11f4a240c0] > Build target for shaded in-JVM dtest jar > > > Key: CASSANDRA-16102 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16102 > Project: Cassandra > Issue Type: Task > Components: Build >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: High > Fix For: 4.0-beta > > > Several small changes that are required as a prerequisite for releasing > [Harry|https://issues.apache.org/jira/browse/CASSANDRA-15348]: > 1. Update snakeYaml in Cassandra > 2. Add a shade maven target and packaging script for shaded dtest artifacts > This patch is only a bridge for everyone to test out Harry with trunk/4.0, > before we're ready to build in-JVM dtest jars for every version, and stop > depending on java-driver (and, subsequently, on Netty and Guava) in Harry. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16102) Build target for shaded in-JVM dtest jar
[ https://issues.apache.org/jira/browse/CASSANDRA-16102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Petrov updated CASSANDRA-16102: Status: Resolved (was: Open) > Build target for shaded in-JVM dtest jar > > > Key: CASSANDRA-16102 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16102 > Project: Cassandra > Issue Type: Task > Components: Build >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: High > Fix For: 4.0-beta > > > Several small changes that are required as a prerequisite for releasing > [Harry|https://issues.apache.org/jira/browse/CASSANDRA-15348]: > 1. Update snakeYaml in Cassandra > 2. Add a shade maven target and packaging script for shaded dtest artifacts > This patch is only a bridge for everyone to test out Harry with trunk/4.0, > before we're ready to build in-JVM dtest jars for every version, and stop > depending on java-driver (and, subsequently, on Netty and Guava) in Harry. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Fix test failure caused by CASSANDRA-16102
This is an automated email from the ASF dual-hosted git repository.
ifesdjeen pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new fb49ab2 Fix test failure caused by CASSANDRA-16102
fb49ab2 is described below
commit fb49ab2b12bf813697971b41fe47ac11f4a240c0
Author: Alex Petrov
AuthorDate: Sun Sep 20 13:24:22 2020 +0300
Fix test failure caused by CASSANDRA-16102
Patch by Alex Petrov; reviewed by David Capwell for CASSANDRA-16102
---
.../apache/cassandra/config/YamlConfigurationLoader.java | 14 +++---
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/src/java/org/apache/cassandra/config/YamlConfigurationLoader.java
b/src/java/org/apache/cassandra/config/YamlConfigurationLoader.java
index 08dfd4c..ac9201c 100644
--- a/src/java/org/apache/cassandra/config/YamlConfigurationLoader.java
+++ b/src/java/org/apache/cassandra/config/YamlConfigurationLoader.java
@@ -17,7 +17,6 @@
*/
package org.apache.cassandra.config;
-import java.beans.IntrospectionException;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
@@ -45,6 +44,7 @@ import org.apache.cassandra.exceptions.ConfigurationException;
import org.yaml.snakeyaml.TypeDescription;
import org.yaml.snakeyaml.Yaml;
import org.yaml.snakeyaml.constructor.Constructor;
+import org.yaml.snakeyaml.constructor.CustomClassLoaderConstructor;
import org.yaml.snakeyaml.error.YAMLException;
import org.yaml.snakeyaml.introspector.MissingProperty;
import org.yaml.snakeyaml.introspector.Property;
@@ -119,7 +119,8 @@ public class YamlConfigurationLoader implements
ConfigurationLoader
throw new AssertionError(e);
}
-Constructor constructor = new CustomConstructor(Config.class);
+
+Constructor constructor = new CustomConstructor(Config.class,
Yaml.class.getClassLoader());
PropertiesChecker propertiesChecker = new PropertiesChecker();
constructor.setPropertyUtils(propertiesChecker);
Yaml yaml = new Yaml(constructor);
@@ -134,11 +135,11 @@ public class YamlConfigurationLoader implements
ConfigurationLoader
}
}
-static class CustomConstructor extends Constructor
+static class CustomConstructor extends CustomClassLoaderConstructor
{
-CustomConstructor(Class theRoot)
+CustomConstructor(Class theRoot, ClassLoader classLoader)
{
-super(theRoot);
+super(theRoot, classLoader);
TypeDescription seedDesc = new
TypeDescription(ParameterizedClass.class);
seedDesc.putMapPropertyType("parameters", String.class,
String.class);
@@ -148,7 +149,6 @@ public class YamlConfigurationLoader implements
ConfigurationLoader
@Override
protected List createDefaultList(int initSize)
{
-
return Lists.newCopyOnWriteArrayList();
}
@@ -165,7 +165,7 @@ public class YamlConfigurationLoader implements
ConfigurationLoader
}
}
-private Config loadConfig(Yaml yaml, byte[] configBytes)
+private static Config loadConfig(Yaml yaml, byte[] configBytes)
{
Config config = yaml.loadAs(new ByteArrayInputStream(configBytes),
Config.class);
// If the configuration file is empty yaml will return null. In this
case we should use the default
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15938) Fix support for adding UDT fields to clustering keys
[ https://issues.apache.org/jira/browse/CASSANDRA-15938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Petrov updated CASSANDRA-15938: Since Version: 2.2.18 Source Control Link: https://github.com/apache/cassandra/commit/2f0eb6f799f32c6f01d1f8384d48910c34ff6a98 Resolution: Fixed Status: Resolved (was: Ready to Commit) > Fix support for adding UDT fields to clustering keys > > > Key: CASSANDRA-15938 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15938 > Project: Cassandra > Issue Type: Bug > Components: Feature/UDT >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Normal > Fix For: 2.2.x, 3.0.x, 3.11.x, 4.0-beta > > > Adding UDT fields to clustering keys is broken in all versions, however > slightly differently. > In 4.0, there will be a brief moment while schema changes are propagated > during which we won’t be able to decode and compare byte sequences. > Unfortunately, it is unclear what we should do in such cases, since we can’t > just come up with a comparator, and we can’t ignore non-null trailing values, > since this will lead to cases where compare for tuples `a;1` and `a;2` would > return 0, effectively making them equal, and we don’t know how to compare > unknown trailing values. Probably we should reject such query since we can’t > sort correctly, but we should make the error message more descriptive than > just "Index 1 out of bounds for length 1”. The only problem is that we get > this exception only on flush right now, so data already propagates to the > node by that time. > In 3.0, the problem is a bit worse than that, since in 3.0 we do not ignore > trailing nulls, so some of the values, written before `ALTER TYPE .. ADD` > become inaccessible. Both old values, and the new ones should always be > accessible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16102) Build target for shaded in-JVM dtest jar
[ https://issues.apache.org/jira/browse/CASSANDRA-16102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Petrov updated CASSANDRA-16102: Status: Open (was: Resolved) > Build target for shaded in-JVM dtest jar > > > Key: CASSANDRA-16102 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16102 > Project: Cassandra > Issue Type: Task > Components: Build >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: High > Fix For: 4.0-beta > > > Several small changes that are required as a prerequisite for releasing > [Harry|https://issues.apache.org/jira/browse/CASSANDRA-15348]: > 1. Update snakeYaml in Cassandra > 2. Add a shade maven target and packaging script for shaded dtest artifacts > This patch is only a bridge for everyone to test out Harry with trunk/4.0, > before we're ready to build in-JVM dtest jars for every version, and stop > depending on java-driver (and, subsequently, on Netty and Guava) in Harry. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15938) Fix support for adding UDT fields to clustering keys
[ https://issues.apache.org/jira/browse/CASSANDRA-15938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Petrov updated CASSANDRA-15938: Status: Ready to Commit (was: Review In Progress) > Fix support for adding UDT fields to clustering keys > > > Key: CASSANDRA-15938 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15938 > Project: Cassandra > Issue Type: Bug > Components: Feature/UDT >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Normal > Fix For: 2.2.x, 3.0.x, 3.11.x, 4.0-beta > > > Adding UDT fields to clustering keys is broken in all versions, however > slightly differently. > In 4.0, there will be a brief moment while schema changes are propagated > during which we won’t be able to decode and compare byte sequences. > Unfortunately, it is unclear what we should do in such cases, since we can’t > just come up with a comparator, and we can’t ignore non-null trailing values, > since this will lead to cases where compare for tuples `a;1` and `a;2` would > return 0, effectively making them equal, and we don’t know how to compare > unknown trailing values. Probably we should reject such query since we can’t > sort correctly, but we should make the error message more descriptive than > just "Index 1 out of bounds for length 1”. The only problem is that we get > this exception only on flush right now, so data already propagates to the > node by that time. > In 3.0, the problem is a bit worse than that, since in 3.0 we do not ignore > trailing nulls, so some of the values, written before `ALTER TYPE .. ADD` > become inaccessible. Both old values, and the new ones should always be > accessible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15938) Fix support for adding UDT fields to clustering keys
[ https://issues.apache.org/jira/browse/CASSANDRA-15938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199950#comment-17199950 ] Alex Petrov commented on CASSANDRA-15938: - [~maedhroz] thank you for checking! we could do that, but I have left it as-is for now. Main motivation for not changing it is withouot storing all potential duplicates, we'd only say how many duplicates were different to the first clustering we've seen, which is only partially useful. That said, if you already have duplicates, you probably should take a close look at your sstables anyways. I've committed the change to 2.2 with [2f0eb6f799f32c6f01d1f8384d48910c34ff6a98|https://github.com/apache/cassandra/commit/2f0eb6f799f32c6f01d1f8384d48910c34ff6a98] and merged it up to [3.0|https://github.com/apache/cassandra/commit/2cde7a7e148e333c3cac7a015322db5f7ccbe6c5], [3.11|https://github.com/apache/cassandra/commit/492009ae9b48839cc97540ecb1cf5217c1f30118] and [trunk|https://github.com/apache/cassandra/commit/53234e744db432666dc27625f87258ce6aab2d23] with corresponding changes. > Fix support for adding UDT fields to clustering keys > > > Key: CASSANDRA-15938 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15938 > Project: Cassandra > Issue Type: Bug > Components: Feature/UDT >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Normal > Fix For: 2.2.x, 3.0.x, 3.11.x, 4.0-beta > > > Adding UDT fields to clustering keys is broken in all versions, however > slightly differently. > In 4.0, there will be a brief moment while schema changes are propagated > during which we won’t be able to decode and compare byte sequences. > Unfortunately, it is unclear what we should do in such cases, since we can’t > just come up with a comparator, and we can’t ignore non-null trailing values, > since this will lead to cases where compare for tuples `a;1` and `a;2` would > return 0, effectively making them equal, and we don’t know how to compare > unknown trailing values. Probably we should reject such query since we can’t > sort correctly, but we should make the error message more descriptive than > just "Index 1 out of bounds for length 1”. The only problem is that we get > this exception only on flush right now, so data already propagates to the > node by that time. > In 3.0, the problem is a bit worse than that, since in 3.0 we do not ignore > trailing nulls, so some of the values, written before `ALTER TYPE .. ADD` > become inaccessible. Both old values, and the new ones should always be > accessible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk
This is an automated email from the ASF dual-hosted git repository.
ifesdjeen pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 53234e744db432666dc27625f87258ce6aab2d23
Merge: 0dc5bd5 492009a
Author: Alex Petrov
AuthorDate: Tue Sep 22 11:36:42 2020 +0200
Merge branch 'cassandra-3.11' into trunk
.../org/apache/cassandra/db/marshal/TupleType.java | 33 ++---
.../db/transform/DuplicateRowChecker.java | 12 +-
.../cassandra/distributed/test/FrozenUDTTest.java | 153 +
.../operations/TuplesWithNullsComparisonTest.java | 78 +++
.../db/compaction/CompactionIteratorTest.java | 2 +-
.../db/transform/DuplicateRowCheckerTest.java | 1 -
6 files changed, 258 insertions(+), 21 deletions(-)
diff --cc src/java/org/apache/cassandra/db/marshal/TupleType.java
index 8a2f7ee,b87964e..83fbb25
--- a/src/java/org/apache/cassandra/db/marshal/TupleType.java
+++ b/src/java/org/apache/cassandra/db/marshal/TupleType.java
@@@ -138,20 -113,15 +138,20 @@@ public class TupleType extends Abstract
return types;
}
-public int compareCustom(ByteBuffer o1, ByteBuffer o2)
+public boolean isTuple()
{
-if (!o1.hasRemaining() || !o2.hasRemaining())
-return o1.hasRemaining() ? 1 : o2.hasRemaining() ? -1 : 0;
+return true;
+}
-ByteBuffer bb1 = o1.duplicate();
-ByteBuffer bb2 = o2.duplicate();
+public int compareCustom(VL left, ValueAccessor accessorL,
VR right, ValueAccessor accessorR)
+{
+if (accessorL.isEmpty(left) || accessorR.isEmpty(right))
+return Boolean.compare(accessorR.isEmpty(right),
accessorL.isEmpty(left));
+
+int offsetL = 0;
+int offsetR = 0;
- for (int i = 0; !accessorL.isEmptyFromOffset(left, offsetL) &&
!accessorR.isEmptyFromOffset(right, offsetR); i++)
-for (int i = 0; bb1.remaining() > 0 && bb2.remaining() > 0 && i <
types.size(); i++)
++for (int i = 0; !accessorL.isEmptyFromOffset(left, offsetL) &&
!accessorR.isEmptyFromOffset(right, offsetR) && i < types.size(); i++)
{
AbstractType comparator = types.get(i);
@@@ -179,24 -145,26 +179,25 @@@
return cmp;
}
- // handle trailing nulls
- while (!accessorL.isEmptyFromOffset(left, offsetL))
- {
- int size = accessorL.getInt(left, offsetL);
- offsetL += TypeSizes.INT_SIZE;
- if (size > 0) // non-null
- return 1;
- }
-if (bb1.remaining() == 0 && bb2.remaining() == 0)
++if (allRemainingComponentsAreNull(left, accessorL, offsetL) &&
allRemainingComponentsAreNull(right, accessorR, offsetR))
+ return 0;
+
-if (bb1.remaining() == 0)
-return allRemainingComponentsAreNull(bb2) ? 0 : -1;
++if (accessorL.isEmptyFromOffset(left, offsetL))
++return allRemainingComponentsAreNull(right, accessorR, offsetR) ?
0 : -1;
+
-// bb1.remaining() > 0 && bb2.remaining() == 0
-return allRemainingComponentsAreNull(bb1) ? 0 : 1;
++return allRemainingComponentsAreNull(left, accessorL, offsetL) ? 0 :
1;
+ }
- while (!accessorR.isEmptyFromOffset(right, offsetR))
-// checks if all remaining components are null (e.g., their size is -1)
-private static boolean allRemainingComponentsAreNull(ByteBuffer bb)
++private boolean allRemainingComponentsAreNull(T v, ValueAccessor
accessor, int offset)
+ {
-while (bb.hasRemaining())
++while (!accessor.isEmptyFromOffset(v, offset))
{
- int size = accessorR.getInt(right, offsetR);
- offsetR += TypeSizes.INT_SIZE;
- if (size > 0) // non-null
- return -1;
-int size = bb.getInt();
++int size = accessor.getInt(v, offset);
++offset += TypeSizes.INT_SIZE;
+ if (size >= 0)
+ return false;
}
-
- return 0;
+ return true;
}
/**
diff --cc src/java/org/apache/cassandra/db/transform/DuplicateRowChecker.java
index 16fabcf,57aa0ae..4347192
--- a/src/java/org/apache/cassandra/db/transform/DuplicateRowChecker.java
+++ b/src/java/org/apache/cassandra/db/transform/DuplicateRowChecker.java
@@@ -39,12 -39,13 +39,13 @@@ public class DuplicateRowChecker extend
{
private static final Logger logger =
LoggerFactory.getLogger(DuplicateRowChecker.class);
-Clustering previous = null;
+Clustering previous = null;
int duplicatesDetected = 0;
+ boolean hadNonEqualDuplicates = false;
final String stage;
-final List replicas;
-final CFMetaData metadata;
+final List replicas;
+final TableMetadata metadata;
final DecoratedKey key;
final boolean snapshotOnDuplicate;
@@@ -88,10 -93,12 +93,11
[cassandra] branch trunk updated (0dc5bd5 -> 53234e7)
This is an automated email from the ASF dual-hosted git repository. ifesdjeen pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 0dc5bd5 Changes required for Harry build new 2f0eb6f Fix support for adding UDT fields to clustering keys new 2cde7a7 Merge branch 'cassandra-2.2' into cassandra-3.0 new 492009a Merge branch 'cassandra-3.0' into cassandra-3.11 new 53234e7 Merge branch 'cassandra-3.11' into trunk The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/cassandra/db/marshal/TupleType.java | 33 ++--- .../db/transform/DuplicateRowChecker.java | 12 +- .../cassandra/distributed/test/FrozenUDTTest.java | 153 + .../operations/TuplesWithNullsComparisonTest.java | 78 +++ .../db/compaction/CompactionIteratorTest.java | 2 +- .../db/transform/DuplicateRowCheckerTest.java | 1 - 6 files changed, 258 insertions(+), 21 deletions(-) create mode 100644 test/distributed/org/apache/cassandra/distributed/test/FrozenUDTTest.java create mode 100644 test/unit/org/apache/cassandra/cql3/validation/operations/TuplesWithNullsComparisonTest.java - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.0 updated (277d839 -> 2cde7a7)
This is an automated email from the ASF dual-hosted git repository. ifesdjeen pushed a change to branch cassandra-3.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 277d839 Use IF NOT EXISTS for index and UDT create statements in snapshot schema files new 2f0eb6f Fix support for adding UDT fields to clustering keys new 2cde7a7 Merge branch 'cassandra-2.2' into cassandra-3.0 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/cassandra/db/marshal/TupleType.java | 21 ++--- .../cassandra/db/transform/DuplicateRowChecker.java | 12 +--- 2 files changed, 27 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-2.2 updated: Fix support for adding UDT fields to clustering keys
This is an automated email from the ASF dual-hosted git repository.
ifesdjeen pushed a commit to branch cassandra-2.2
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-2.2 by this push:
new 2f0eb6f Fix support for adding UDT fields to clustering keys
2f0eb6f is described below
commit 2f0eb6f799f32c6f01d1f8384d48910c34ff6a98
Author: Alex Petrov
AuthorDate: Fri Jul 24 16:53:43 2020 +0200
Fix support for adding UDT fields to clustering keys
Patch by Alex Petrov; reviewed by Caleb Rackliffe for CASSANDRA-15938
---
.../org/apache/cassandra/db/marshal/TupleType.java | 21 ++-
.../cassandra/distributed/test/FrozenUDTTest.java | 153 +
.../operations/TuplesWithNullsComparisonTest.java | 78 +++
3 files changed, 249 insertions(+), 3 deletions(-)
diff --git a/src/java/org/apache/cassandra/db/marshal/TupleType.java
b/src/java/org/apache/cassandra/db/marshal/TupleType.java
index f3600ef..e62d069 100644
--- a/src/java/org/apache/cassandra/db/marshal/TupleType.java
+++ b/src/java/org/apache/cassandra/db/marshal/TupleType.java
@@ -100,7 +100,7 @@ public class TupleType extends AbstractType
ByteBuffer bb1 = o1.duplicate();
ByteBuffer bb2 = o2.duplicate();
-for (int i = 0; bb1.remaining() > 0 && bb2.remaining() > 0; i++)
+for (int i = 0; bb1.remaining() > 0 && bb2.remaining() > 0 && i <
types.size(); i++)
{
AbstractType comparator = types.get(i);
@@ -124,11 +124,26 @@ public class TupleType extends AbstractType
return cmp;
}
+if (bb1.remaining() == 0 && bb2.remaining() == 0)
+return 0;
+
if (bb1.remaining() == 0)
-return bb2.remaining() == 0 ? 0 : -1;
+return allRemainingComponentsAreNull(bb2) ? 0 : -1;
// bb1.remaining() > 0 && bb2.remaining() == 0
-return 1;
+return allRemainingComponentsAreNull(bb1) ? 0 : 1;
+}
+
+// checks if all remaining components are null (e.g., their size is -1)
+private static boolean allRemainingComponentsAreNull(ByteBuffer bb)
+{
+while (bb.hasRemaining())
+{
+int size = bb.getInt();
+if (size >= 0)
+return false;
+}
+return true;
}
/**
diff --git
a/test/distributed/org/apache/cassandra/distributed/test/FrozenUDTTest.java
b/test/distributed/org/apache/cassandra/distributed/test/FrozenUDTTest.java
new file mode 100644
index 000..2a45b86
--- /dev/null
+++ b/test/distributed/org/apache/cassandra/distributed/test/FrozenUDTTest.java
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.distributed.test;
+
+import java.io.IOException;
+import java.util.concurrent.ExecutionException;
+
+import org.junit.Test;
+
+import org.apache.cassandra.distributed.Cluster;
+import org.apache.cassandra.distributed.api.ConsistencyLevel;
+import org.apache.cassandra.service.StorageService;
+
+import static org.apache.cassandra.distributed.shared.AssertUtils.assertRows;
+import static org.apache.cassandra.distributed.shared.AssertUtils.row;
+
+public class FrozenUDTTest extends TestBaseImpl
+{
+@Test
+public void testAddAndUDTField() throws IOException
+{
+try (Cluster cluster = init(Cluster.build(1).start()))
+{
+cluster.schemaChange("create type " + KEYSPACE + ".a (foo text)");
+cluster.schemaChange("create table " + KEYSPACE + ".x (id int, ck
frozen, i int, primary key (id, ck))");
+for (int i = 0; i < 10; i++)
+cluster.coordinator(1).execute("insert into " + KEYSPACE + ".x
(id, ck, i) VALUES (?, " + json(i) + ", ? )", ConsistencyLevel.ALL, i, i);
+
+for (int i = 0; i < 10; i++)
+assertRows(cluster.coordinator(1).execute("select i from " +
KEYSPACE + ".x WHERE id = ? and ck = " + json(i), ConsistencyLevel.ALL, i),
+ row(i));
+
+cluster.schemaChange("alter type " + KEYSPACE + ".a add bar text");
+
+for (int i = 5; i < 15; i++)
+
[cassandra] 01/01: Merge branch 'cassandra-3.0' into cassandra-3.11
This is an automated email from the ASF dual-hosted git repository.
ifesdjeen pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 492009ae9b48839cc97540ecb1cf5217c1f30118
Merge: cbb7644 2cde7a7
Author: Alex Petrov
AuthorDate: Tue Sep 22 11:34:34 2020 +0200
Merge branch 'cassandra-3.0' into cassandra-3.11
.../org/apache/cassandra/db/marshal/TupleType.java | 32 +++--
.../db/transform/DuplicateRowChecker.java | 13 +-
.../cassandra/distributed/test/FrozenUDTTest.java | 153 +
.../operations/TuplesWithNullsComparisonTest.java | 78 +++
.../db/compaction/CompactionIteratorTest.java | 2 +-
5 files changed, 260 insertions(+), 18 deletions(-)
diff --cc src/java/org/apache/cassandra/db/transform/DuplicateRowChecker.java
index 7a6f7f9,621821b..57aa0ae
--- a/src/java/org/apache/cassandra/db/transform/DuplicateRowChecker.java
+++ b/src/java/org/apache/cassandra/db/transform/DuplicateRowChecker.java
@@@ -88,10 -93,11 +93,12 @@@ public class DuplicateRowChecker extend
{
if (duplicatesDetected > 0)
{
- logger.warn("Detected {} duplicate rows for {} during {}",
+ logger.warn("Detected {} duplicate rows for {} during {}.{}",
duplicatesDetected,
metadata.getKeyValidator().getString(key.getKey()),
- stage);
+ stage,
+ hadNonEqualDuplicates ? " Some duplicates had
different byte representation." : "");
++
if (snapshotOnDuplicate)
DiagnosticSnapshotService.duplicateRows(metadata, replicas);
}
diff --cc
test/distributed/org/apache/cassandra/distributed/test/FrozenUDTTest.java
index 000,000..2a45b86
new file mode 100644
--- /dev/null
+++ b/test/distributed/org/apache/cassandra/distributed/test/FrozenUDTTest.java
@@@ -1,0 -1,0 +1,153 @@@
++/*
++ * Licensed to the Apache Software Foundation (ASF) under one
++ * or more contributor license agreements. See the NOTICE file
++ * distributed with this work for additional information
++ * regarding copyright ownership. The ASF licenses this file
++ * to you under the Apache License, Version 2.0 (the
++ * "License"); you may not use this file except in compliance
++ * with the License. You may obtain a copy of the License at
++ *
++ * http://www.apache.org/licenses/LICENSE-2.0
++ *
++ * Unless required by applicable law or agreed to in writing, software
++ * distributed under the License is distributed on an "AS IS" BASIS,
++ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
++ * See the License for the specific language governing permissions and
++ * limitations under the License.
++ */
++
++package org.apache.cassandra.distributed.test;
++
++import java.io.IOException;
++import java.util.concurrent.ExecutionException;
++
++import org.junit.Test;
++
++import org.apache.cassandra.distributed.Cluster;
++import org.apache.cassandra.distributed.api.ConsistencyLevel;
++import org.apache.cassandra.service.StorageService;
++
++import static org.apache.cassandra.distributed.shared.AssertUtils.assertRows;
++import static org.apache.cassandra.distributed.shared.AssertUtils.row;
++
++public class FrozenUDTTest extends TestBaseImpl
++{
++@Test
++public void testAddAndUDTField() throws IOException
++{
++try (Cluster cluster = init(Cluster.build(1).start()))
++{
++cluster.schemaChange("create type " + KEYSPACE + ".a (foo text)");
++cluster.schemaChange("create table " + KEYSPACE + ".x (id int, ck
frozen, i int, primary key (id, ck))");
++for (int i = 0; i < 10; i++)
++cluster.coordinator(1).execute("insert into " + KEYSPACE +
".x (id, ck, i) VALUES (?, " + json(i) + ", ? )", ConsistencyLevel.ALL, i, i);
++
++for (int i = 0; i < 10; i++)
++assertRows(cluster.coordinator(1).execute("select i from " +
KEYSPACE + ".x WHERE id = ? and ck = " + json(i), ConsistencyLevel.ALL, i),
++ row(i));
++
++cluster.schemaChange("alter type " + KEYSPACE + ".a add bar
text");
++
++for (int i = 5; i < 15; i++)
++cluster.coordinator(1).execute("insert into " + KEYSPACE +
".x (id, ck, i) VALUES (?, " + json(i) + ", ? )", ConsistencyLevel.ALL, i, i);
++cluster.forEach(i -> i.flush(KEYSPACE));
++
++for (int i = 5; i < 15; i++)
++assertRows(cluster.coordinator(1).execute("select i from " +
KEYSPACE + ".x WHERE id = ? and ck = " + json(i), ConsistencyLevel.ALL, i),
++ row(i));
++}
++}
++
++@Test
++public void testEmptyValue() throws IOException
++{
++try (Cluster cluster = init(Cluster.build(1).start()))
++{
++cluster.schemaChange("create type " + KEYSPACE +
[cassandra] branch cassandra-3.11 updated (cbb7644 -> 492009a)
This is an automated email from the ASF dual-hosted git repository. ifesdjeen pushed a change to branch cassandra-3.11 in repository https://gitbox.apache.org/repos/asf/cassandra.git. from cbb7644 Merge branch cassandra-3.0 into cassandra-3.11 new 2f0eb6f Fix support for adding UDT fields to clustering keys new 2cde7a7 Merge branch 'cassandra-2.2' into cassandra-3.0 new 492009a Merge branch 'cassandra-3.0' into cassandra-3.11 The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/cassandra/db/marshal/TupleType.java | 32 +++-- .../db/transform/DuplicateRowChecker.java | 13 +- .../cassandra/distributed/test/FrozenUDTTest.java | 153 + .../operations/TuplesWithNullsComparisonTest.java | 78 +++ .../db/compaction/CompactionIteratorTest.java | 2 +- 5 files changed, 260 insertions(+), 18 deletions(-) create mode 100644 test/distributed/org/apache/cassandra/distributed/test/FrozenUDTTest.java create mode 100644 test/unit/org/apache/cassandra/cql3/validation/operations/TuplesWithNullsComparisonTest.java - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-2.2' into cassandra-3.0
This is an automated email from the ASF dual-hosted git repository.
ifesdjeen pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 2cde7a7e148e333c3cac7a015322db5f7ccbe6c5
Merge: 277d839 2f0eb6f
Author: Alex Petrov
AuthorDate: Tue Sep 22 11:29:04 2020 +0200
Merge branch 'cassandra-2.2' into cassandra-3.0
.../org/apache/cassandra/db/marshal/TupleType.java | 21 ++---
.../cassandra/db/transform/DuplicateRowChecker.java | 12 +---
2 files changed, 27 insertions(+), 6 deletions(-)
diff --cc src/java/org/apache/cassandra/db/transform/DuplicateRowChecker.java
index 7a6f7f9,000..621821b
mode 100644,00..100644
--- a/src/java/org/apache/cassandra/db/transform/DuplicateRowChecker.java
+++ b/src/java/org/apache/cassandra/db/transform/DuplicateRowChecker.java
@@@ -1,139 -1,0 +1,145 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.transform;
+
+import java.net.InetAddress;
+import java.util.Collections;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.CFMetaData;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.*;
+import org.apache.cassandra.db.compaction.OperationType;
+import org.apache.cassandra.db.partitions.PartitionIterator;
+import org.apache.cassandra.db.partitions.UnfilteredPartitionIterator;
+import org.apache.cassandra.db.rows.*;
+import org.apache.cassandra.utils.DiagnosticSnapshotService;
+import org.apache.cassandra.utils.FBUtilities;
+
+public class DuplicateRowChecker extends Transformation>
+{
+private static final Logger logger =
LoggerFactory.getLogger(DuplicateRowChecker.class);
+
+Clustering previous = null;
+int duplicatesDetected = 0;
++boolean hadNonEqualDuplicates = false;
+
+final String stage;
+final List replicas;
+final CFMetaData metadata;
+final DecoratedKey key;
+final boolean snapshotOnDuplicate;
+
+DuplicateRowChecker(final DecoratedKey key,
+final CFMetaData metadata,
+final String stage,
+final boolean snapshotOnDuplicate,
+final List replicas)
+{
+this.key = key;
+this.metadata = metadata;
+this.stage = stage;
+this.snapshotOnDuplicate = snapshotOnDuplicate;
+this.replicas = replicas;
+}
+
+protected DeletionTime applyToDeletion(DeletionTime deletionTime)
+{
+return deletionTime;
+}
+
+protected RangeTombstoneMarker applyToMarker(RangeTombstoneMarker marker)
+{
+return marker;
+}
+
+protected Row applyToStatic(Row row)
+{
+return row;
+}
+
+protected Row applyToRow(Row row)
+{
- if (null != previous && row.clustering().equals(previous))
++if (null != previous && metadata.comparator.compare(row.clustering(),
previous) == 0)
++{
+duplicatesDetected++;
++hadNonEqualDuplicates |= !row.clustering().equals(previous);
++}
++
+previous = row.clustering();
+return row;
+}
+
+protected void onPartitionClose()
+{
+if (duplicatesDetected > 0)
+{
- logger.warn("Detected {} duplicate rows for {} during {}",
++logger.warn("Detected {} duplicate rows for {} during {}.{}",
+duplicatesDetected,
+metadata.getKeyValidator().getString(key.getKey()),
- stage);
++stage,
++hadNonEqualDuplicates ? " Some duplicates had
different byte representation." : "");
+if (snapshotOnDuplicate)
+DiagnosticSnapshotService.duplicateRows(metadata, replicas);
+}
+duplicatesDetected = 0;
+previous = null;
+super.onPartitionClose();
+}
+
+public static UnfilteredPartitionIterator duringCompaction(final
UnfilteredPartitionIterator iterator, OperationType type)
[jira] [Commented] (CASSANDRA-16109) Don't adjust nodeCount when setting node id topology in in-jvm dtests
[ https://issues.apache.org/jira/browse/CASSANDRA-16109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199931#comment-17199931 ] Marcus Eriksson commented on CASSANDRA-16109: - published the 0.0.5 snapshot and ran the jvm dtests, this branch includes updates for CASSANDRA-16101, CASSANDRA-15386 and CASSANDRA-16109: [2.2|https://github.com/krummas/cassandra/commits/dtest0.0.5-snapshot-2.2] [circle|https://app.circleci.com/pipelines/github/krummas/cassandra/524/workflows/9693d0d8-be85-40a1-9ce8-2eee6ac53b83] [3.0|https://github.com/krummas/cassandra/commits/dtest0.0.5-snapshot-3.0] [circle|https://app.circleci.com/pipelines/github/krummas/cassandra/526/workflows/ba3e5ab6-ece7-4a74-a246-29a6b3f0f204] [3.11|https://github.com/krummas/cassandra/commits/dtest0.0.5-snapshot-3.11] [circle|https://app.circleci.com/pipelines/github/krummas/cassandra/527/workflows/510bb3fd-66f0-4732-8c64-32c8274f61c7] [trunk|https://github.com/krummas/cassandra/commits/dtest0.0.5-snapshot-trunk] [circle|https://app.circleci.com/pipelines/github/krummas/cassandra/525/workflows/20b8d7e7-1bb9-414d-bce1-b5857473c6a1] > Don't adjust nodeCount when setting node id topology in in-jvm dtests > - > > Key: CASSANDRA-16109 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16109 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest/java >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Low > Labels: pull-request-available > > We update the node count when setting the node id topology in in-jvm dtests, > this should only happen if node count is smaller than the node id topology, > otherwise bootstrap tests error out. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16069) Loss of functionality around null clustering when dropping compact storage
[
https://issues.apache.org/jira/browse/CASSANDRA-16069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199923#comment-17199923
]
Alex Petrov commented on CASSANDRA-16069:
-
I think this problem is quite similar to treating cell deletions as row
deletions in compact storage. Main problem here is that if we go with {{2}},
I'm not sure why allow to drop compact storage at all. In this case it'll just
be that compact tables are going to still be special-cased. Or we'll have to
have two versions of drop compact storage: one that preserves behaviour
(basically, (2)), and one that doesn't (what we have already).
I generally don't mind (1), but I don't see huge value in allowing implicit
nulls, so I'd rather just document it.
> Loss of functionality around null clustering when dropping compact storage
> --
>
> Key: CASSANDRA-16069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16069
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/CQL
>Reporter: Sylvain Lebresne
>Priority: Normal
>
> For backward compatibility reasons[1], it is allowed to insert rows where
> some of the clustering columns are {{null}} for compact tables. That support
> is a tad limited/inconsistent[2] but essentially you can do:
> {noformat}
> cqlsh:ks> CREATE TABLE t (k int, c1 int, c2 int, v int, PRIMARY KEY (k, c1,
> c2)) WITH COMPACT STORAGE;
> cqlsh:ks> INSERT INTO t(k, c1, v) VALUES (1, 1, 1);
> cqlsh:ks> SELECT * FROM t;
> k | c1 | c2 | v
> ---++--+---
> 1 | 1 | null | 1
> (1 rows)
> cqlsh:ks> UPDATE t SET v = 2 WHERE k = 1 AND c1 = 1;
> cqlsh:ks> SELECT * FROM t;
> k | c1 | c2 | v
> ---++--+---
> 1 | 1 | null | 2
> (1 rows)
> {noformat}
> This is not allowed on non-compact tables however:
> {noformat}
> cqlsh:ks> CREATE TABLE t2 (k int, c1 int, c2 int, v int, PRIMARY KEY (k, c1,
> c2));
> cqlsh:ks> INSERT INTO t2(k, c1, v) VALUES (1, 1, 1);
> InvalidRequest: Error from server: code=2200 [Invalid query] message="Some
> clustering keys are missing: c2"
> cqlsh:ks> UPDATE t2 SET v = 2 WHERE k = 1 AND c1 = 1;
> InvalidRequest: Error from server: code=2200 [Invalid query] message="Some
> clustering keys are missing: c2"
> {noformat}
> Which means that a user with a compact table that rely on this will not be
> able to use {{DROP COMPACT STORAGE}}.
> Which is a problem for the 4.0 upgrade story. Problem to which we need an
> answer.
>
>
> [1]: the underlying {{CompositeType}} used by such tables allows to provide
> only a prefix of components, so thrift users could have used such
> functionality. We thus had to support it in CQL, or those users wouldn't have
> been able to upgrade to CQL easily.
> [2]: building on the example above, the value for {{c2}} is essentially
> {{null}}, yet none of the following is currently allowed:
> {noformat}
> cqlsh:ks> INSERT INTO t(k, c1, c2, v) VALUES (1, 1, null, 1);
> InvalidRequest: Error from server: code=2200 [Invalid query] message="Invalid
> null value in condition for column c2"
> cqlsh:ks> UPDATE t SET v = 2 WHERE k = 1 AND c1 = 1 AND c2 = null;
> InvalidRequest: Error from server: code=2200 [Invalid query] message="Invalid
> null value in condition for column c2"
> cqlsh:ks> SELECT * FROM c WHERE k = 1 AND c1 = 1 AND c2 = null;
> InvalidRequest: Error from server: code=2200 [Invalid query] message="Invalid
> null value in condition for column c2"
> {noformat}
> Not only is that unintuitive/inconsistent, but the {{SELECT}} one means there
> is no way to select only the row. You can skip specifying {{c2}} in the
> {{SELECT}}, but this become a slice selection essentially, as shown below:
> {noformat}
> cqlsh:ks> INSERT INTO ct(k, c1, c2, v) VALUES (1, 1, 1, 1);
> cqlsh:ks> SELECT * FROM ct WHERE k = 1 AND c1 = 1;
> k | c1 | c2 | v
> ---++--+---
> 1 | 1 | null | 1
> 1 | 1 |1 | 1
> (2 rows)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-in-jvm-dtest-api] branch master updated: CASSANDRA-16109: if user has not set nodeCount, use the node id topology size (#20)
This is an automated email from the ASF dual-hosted git repository.
marcuse pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-in-jvm-dtest-api.git
The following commit(s) were added to refs/heads/master by this push:
new b0054ea CASSANDRA-16109: if user has not set nodeCount, use the node
id topology size (#20)
b0054ea is described below
commit b0054ea44f1311d938aaa22f775f4930a8a1419c
Author: Marcus Eriksson
AuthorDate: Tue Sep 22 10:22:54 2020 +0200
CASSANDRA-16109: if user has not set nodeCount, use the node id topology
size (#20)
---
.../org/apache/cassandra/distributed/shared/AbstractBuilder.java | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git
a/src/main/java/org/apache/cassandra/distributed/shared/AbstractBuilder.java
b/src/main/java/org/apache/cassandra/distributed/shared/AbstractBuilder.java
index 47f2d48..95e7e5d 100644
--- a/src/main/java/org/apache/cassandra/distributed/shared/AbstractBuilder.java
+++ b/src/main/java/org/apache/cassandra/distributed/shared/AbstractBuilder.java
@@ -292,7 +292,10 @@ public abstract class AbstractBuilder nodeCount)
{
-System.out.printf("nodeIdTopology configured for %d nodes
while nodeCount is %d%n", nodeIdTopology.size(), nodeCount);
+if (nodeCount == 0)
+nodeCount = nodeIdTopology.size();
+else
+System.out.printf("nodeIdTopology configured for %d nodes
while nodeCount is %d%n", nodeIdTopology.size(), nodeCount);
}
}
else
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16121) Circleci should run cqlshlib tests as well
[ https://issues.apache.org/jira/browse/CASSANDRA-16121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199912#comment-17199912 ] Berenguer Blasi commented on CASSANDRA-16121: - I added that new executor and updated the files. I will now run small, medium and high to make sure the 3 still work and then we should be able to merge. > Circleci should run cqlshlib tests as well > -- > > Key: CASSANDRA-16121 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16121 > Project: Cassandra > Issue Type: Bug > Components: CI, Test/unit >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > > Currently circleci is not running cqlshlib tests. This resulted in some bugs > not being caught before committing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16109) Don't adjust nodeCount when setting node id topology in in-jvm dtests
[ https://issues.apache.org/jira/browse/CASSANDRA-16109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CASSANDRA-16109: --- Labels: pull-request-available (was: ) > Don't adjust nodeCount when setting node id topology in in-jvm dtests > - > > Key: CASSANDRA-16109 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16109 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest/java >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Low > Labels: pull-request-available > > We update the node count when setting the node id topology in in-jvm dtests, > this should only happen if node count is smaller than the node id topology, > otherwise bootstrap tests error out. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
