[jira] [Commented] (CASSANDRA-14641) Nodetool toppartitions raises java.lang.IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CASSANDRA-14641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583397#comment-16583397
]
Chris Lohfink commented on CASSANDRA-14641:
---
ahh I suspect this has something to do with {{WITH COMPACT STORAGE}}
> Nodetool toppartitions raises java.lang.IndexOutOfBoundsException
> -
>
> Key: CASSANDRA-14641
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14641
> Project: Cassandra
> Issue Type: Bug
> Environment: Linux 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
> 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Irina Truong
>Priority: Minor
>
> I get this exception in most cases when using nodetool.toppartitions command:
> {noformat}
> irina@host:/usr/local/cassandra/bin$ ./nodetool toppartitions casterisk
> events_2018_08_13 1000
> error: index (4) must be less than size (4)
> -- StackTrace --
> java.lang.IndexOutOfBoundsException: index (4) must be less than size (4)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:310)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:292)
> at
> com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:65)
> at
> org.apache.cassandra.db.marshal.CompositeType.getAndAppendComparator(CompositeType.java:148)
> at
> org.apache.cassandra.db.marshal.AbstractCompositeType.getString(AbstractCompositeType.java:174)
> at
> org.apache.cassandra.db.ColumnFamilyStore.finishLocalSampling(ColumnFamilyStore.java:1588)
> at sun.reflect.GeneratedMethodAccessor1899.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
> at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:324)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Occasionally, nodetool will return a proper response. But in most cases, no.
> Cassandra version 3.0.8.
> table schema:
> {noformat}
> cqlsh:casterisk> describe table
[jira] [Commented] (CASSANDRA-14647) Reading cardinality from Statistics.db failed
[
https://issues.apache.org/jira/browse/CASSANDRA-14647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583361#comment-16583361
]
Marcus Eriksson commented on CASSANDRA-14647:
-
[~nezdali] could you post logs?
> Reading cardinality from Statistics.db failed
> -
>
> Key: CASSANDRA-14647
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14647
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: Clients are doing only writes with Local One, cluster
> consist of 3 regions with RF3.
> Storage is configured wth jbod/XFS on 10 x 1Tb disks
> IOPS limit for each disk 500 (total 5000 iops)
> Bandwith for each disk 60mb/s (600 total)
> OS is Debian linux.
>Reporter: Vitali Djatsuk
>Priority: Major
> Fix For: 3.0.x
>
> Attachments: cassandra_compaction_pending_tasks_7days.png
>
>
> There is some issue with sstable metadata which is visible in system.log, the
> messages says:
> {noformat}
> WARN [Thread-6] 2018-07-25 07:12:47,928 SSTableReader.java:249 - Reading
> cardinality from Statistics.db failed for
> /opt/data/disk5/data/keyspace/table/mc-big-Data.db.{noformat}
> Although there is no such file.
> The message has appeared after i've changed the compaction strategy from
> SizeTiered to Leveled. Compaction strategy has been changed region by region
> (total 3 regions) and it has coincided with the double client write traffic
> increase.
> I have tried to run nodetool scrub to rebuilt the sstable, but that does not
> fix the issue.
> So very hard to define the steps to reproduce, probably it will be:
> # run stress tool with write traffic
> # under load change compaction strategy from SireTiered to Leveled for the
> bunch of hosts
> # add more write traffic
> Reading the code it is said that if this metadata is broken, then "estimating
> the keys will be done using index summary".
>
> [https://github.com/apache/cassandra/blob/cassandra-3.0.17/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L247]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13458) Diag. Events: Add unit testing support
[ https://issues.apache.org/jira/browse/CASSANDRA-13458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526227#comment-16526227 ] mck edited comment on CASSANDRA-13458 at 8/17/18 3:48 AM: -- (so far) this is a +1 from me. was (Author: michaelsembwever): this is a +1 from me. > Diag. Events: Add unit testing support > -- > > Key: CASSANDRA-13458 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13458 > Project: Cassandra > Issue Type: Sub-task > Components: Testing >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > > Diagnostic events will improve unit testing by > * providing test execution control instances based on CompletableFutures (see > [PendingRangeCalculatorServiceTest.java|https://github.com/spodkowinski/cassandra/blob/WIP-13458/test/unit/org/apache/cassandra/gms/PendingRangeCalculatorServiceTest.java]) > > * validate state and behavior by allowing you to inspect generated events > (see > [HintsServiceEventsTest.java|https://github.com/spodkowinski/cassandra/blob/WIP-13458/test/unit/org/apache/cassandra/hints/HintsServiceEventsTest.java]) > See included > [testing.rst|https://github.com/spodkowinski/cassandra/blob/WIP-13458/doc/source/development/testing.rst#diagnostic-events-40] > draft for more details. Let me know if this would be useful for you as a > developer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13457) Diag. Events: Add base classes
[ https://issues.apache.org/jira/browse/CASSANDRA-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mck updated CASSANDRA-13457: Reviewers: Jason Brown > Diag. Events: Add base classes > -- > > Key: CASSANDRA-13457 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13457 > Project: Cassandra > Issue Type: Sub-task > Components: Core, Observability >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > > Base ticket for adding classes that will allow you to implement and subscribe > to events. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13457) Diag. Events: Add base classes
[ https://issues.apache.org/jira/browse/CASSANDRA-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583319#comment-16583319 ] mck commented on CASSANDRA-13457: - My +1 stands. > Diag. Events: Add base classes > -- > > Key: CASSANDRA-13457 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13457 > Project: Cassandra > Issue Type: Sub-task > Components: Core, Observability >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > > Base ticket for adding classes that will allow you to implement and subscribe > to events. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14635) Support table level configuration of monotonic reads
[
https://issues.apache.org/jira/browse/CASSANDRA-14635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Blake Eggleston updated CASSANDRA-14635:
Fix Version/s: 4.0
Status: Patch Available (was: Open)
[trunk|https://github.com/bdeggleston/cassandra/tree/C14635]
[dtests|https://github.com/bdeggleston/cassandra-dtest/tree/C14635]
Initial implementation ready for review. This adds the {{read_repair}} option
to table schema, with the options {{blocking}} (default), {{async}}, and
{{none}}.
> Support table level configuration of monotonic reads
>
>
> Key: CASSANDRA-14635
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14635
> Project: Cassandra
> Issue Type: Improvement
> Components: Coordination
>Reporter: Ariel Weisberg
>Assignee: Blake Eggleston
>Priority: Major
> Fix For: 4.0
>
>
> In CASSANDRA-10726 it was discussed that allowing expert users to forgo
> monotonic reads might be desirable. It is practical to control monotonicity
> of reads at a fine grained level because it involves changing the behavior of
> read repair on a per read basis.
> Per CASSANDRA-14593 we already don't preserve update atomicity down to the
> column level. You could read the key out of a row and read repair the key,
> pass the key to another process which attempts to read the value, but finds
> the value is null because read repair only repairs the data (including
> columns) that is part of the read. IMO it's a stretch to say that reads are
> monotonic. It is technically correct, the best kind of correct, but far from
> as useful as it should be.
> An initial implementation could make read repair asynchronous or forgo read
> repair entirely. This would improve the throughput and latency of reads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14647) Reading cardinality from Statistics.db failed
[
https://issues.apache.org/jira/browse/CASSANDRA-14647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583121#comment-16583121
]
Dinesh Joshi commented on CASSANDRA-14647:
--
Hey Vitali, thanks for the report. [~krummas] any idea what is going on here?
Would changing compaction strategies cause this issue?
> Reading cardinality from Statistics.db failed
> -
>
> Key: CASSANDRA-14647
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14647
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: Clients are doing only writes with Local One, cluster
> consist of 3 regions with RF3.
> Storage is configured wth jbod/XFS on 10 x 1Tb disks
> IOPS limit for each disk 500 (total 5000 iops)
> Bandwith for each disk 60mb/s (600 total)
> OS is Debian linux.
>Reporter: Vitali Djatsuk
>Priority: Major
> Fix For: 3.0.x
>
> Attachments: cassandra_compaction_pending_tasks_7days.png
>
>
> There is some issue with sstable metadata which is visible in system.log, the
> messages says:
> {noformat}
> WARN [Thread-6] 2018-07-25 07:12:47,928 SSTableReader.java:249 - Reading
> cardinality from Statistics.db failed for
> /opt/data/disk5/data/keyspace/table/mc-big-Data.db.{noformat}
> Although there is no such file.
> The message has appeared after i've changed the compaction strategy from
> SizeTiered to Leveled. Compaction strategy has been changed region by region
> (total 3 regions) and it has coincided with the double client write traffic
> increase.
> I have tried to run nodetool scrub to rebuilt the sstable, but that does not
> fix the issue.
> So very hard to define the steps to reproduce, probably it will be:
> # run stress tool with write traffic
> # under load change compaction strategy from SireTiered to Leveled for the
> bunch of hosts
> # add more write traffic
> Reading the code it is said that if this metadata is broken, then "estimating
> the keys will be done using index summary".
>
> [https://github.com/apache/cassandra/blob/cassandra-3.0.17/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L247]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14647) Reading cardinality from Statistics.db failed
[
https://issues.apache.org/jira/browse/CASSANDRA-14647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Joshi updated CASSANDRA-14647:
-
Description:
There is some issue with sstable metadata which is visible in system.log, the
messages says:
{noformat}
WARN [Thread-6] 2018-07-25 07:12:47,928 SSTableReader.java:249 - Reading
cardinality from Statistics.db failed for
/opt/data/disk5/data/keyspace/table/mc-big-Data.db.{noformat}
Although there is no such file.
The message has appeared after i've changed the compaction strategy from
SizeTiered to Leveled. Compaction strategy has been changed region by region
(total 3 regions) and it has coincided with the double client write traffic
increase.
I have tried to run nodetool scrub to rebuilt the sstable, but that does not
fix the issue.
So very hard to define the steps to reproduce, probably it will be:
# run stress tool with write traffic
# under load change compaction strategy from SireTiered to Leveled for the
bunch of hosts
# add more write traffic
Reading the code it is said that if this metadata is broken, then "estimating
the keys will be done using index summary".
[https://github.com/apache/cassandra/blob/cassandra-3.0.17/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L247]
was:
There is some issue with sstable metadata which is visible in system.log, the
messages says:
WARN [Thread-6] 2018-07-25 07:12:47,928 SSTableReader.java:249 - Reading
cardinality from Statistics.db failed for
/opt/data/disk5/data/keyspace/table/mc-big-Data.db. Although there is no such
file.
The message has appeared after i've changed the compaction strategy from
SizeTiered to Leveled. Compaction strategy has been changed region by region
(total 3 regions) and it has coincided with the double client write traffic
increase.
I have tried to run nodetool scrub to rebuilt the sstable, but that does not
fix the issue.
So very hard to define the steps to reproduce, probably it will be:
1) run stress tool with write traffic
2) under load change compaction strategy from SireTiered to Leveled for the
bunch of hosts
3) add more write traffic
Reading the code it is said that if this metadata is broken, then "estimating
the keys will be done using index summary".
[https://github.com/apache/cassandra/blob/cassandra-3.0.17/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L247]
> Reading cardinality from Statistics.db failed
> -
>
> Key: CASSANDRA-14647
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14647
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: Clients are doing only writes with Local One, cluster
> consist of 3 regions with RF3.
> Storage is configured wth jbod/XFS on 10 x 1Tb disks
> IOPS limit for each disk 500 (total 5000 iops)
> Bandwith for each disk 60mb/s (600 total)
> OS is Debian linux.
>Reporter: Vitali Djatsuk
>Priority: Major
> Fix For: 3.0.x
>
> Attachments: cassandra_compaction_pending_tasks_7days.png
>
>
> There is some issue with sstable metadata which is visible in system.log, the
> messages says:
> {noformat}
> WARN [Thread-6] 2018-07-25 07:12:47,928 SSTableReader.java:249 - Reading
> cardinality from Statistics.db failed for
> /opt/data/disk5/data/keyspace/table/mc-big-Data.db.{noformat}
> Although there is no such file.
> The message has appeared after i've changed the compaction strategy from
> SizeTiered to Leveled. Compaction strategy has been changed region by region
> (total 3 regions) and it has coincided with the double client write traffic
> increase.
> I have tried to run nodetool scrub to rebuilt the sstable, but that does not
> fix the issue.
> So very hard to define the steps to reproduce, probably it will be:
> # run stress tool with write traffic
> # under load change compaction strategy from SireTiered to Leveled for the
> bunch of hosts
> # add more write traffic
> Reading the code it is said that if this metadata is broken, then "estimating
> the keys will be done using index summary".
>
> [https://github.com/apache/cassandra/blob/cassandra-3.0.17/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L247]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14574) Incomplete handling of exceptions when decoding incoming messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Joshi updated CASSANDRA-14574:
-
Status: Ready to Commit (was: Patch Available)
> Incomplete handling of exceptions when decoding incoming messages
> --
>
> Key: CASSANDRA-14574
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14574
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
>Reporter: Aleksey Yeschenko
>Assignee: Jason Brown
>Priority: Major
> Fix For: 4.0
>
>
> {{MessageInHandler.decode()}} occasionally reads the payload incorrectly,
> passing the full message to {{MessageIn.read()}} instead of just the payload
> bytes.
> You can see the stack trace in the logs from this [CI
> run|https://circleci.com/gh/iamaleksey/cassandra/437#tests/containers/38].
> {code}
> Caused by: java.lang.AssertionError: null
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:351)
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:371)
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:335)
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:158)
> at
> org.apache.cassandra.net.async.MessageInHandler.decode(MessageInHandler.java:132)
> {code}
> Reconstructed, truncated stream passed to {{MessageIn.read()}}:
> {{000b000743414c5f42414301002a01e1a5c9b089fd11e8b517436ee124300704005d10fc50ec}}
> You can clearly see parameters in there encoded before the payload:
> {{[43414c5f424143 - CAL_BAC] [01 - ONE_BYTE] [002a - 42, payload size] 01 e1
> a5 c9 b0 89 fd 11 e8 b5 17 43 6e e1 24 30 07 04 00 00 00 1d 10 fc 50 ec}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14574) Incomplete handling of exceptions when decoding incoming messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583117#comment-16583117
]
Dinesh Joshi commented on CASSANDRA-14574:
--
The dtest looks good! I'm +1 on the patch.
> Incomplete handling of exceptions when decoding incoming messages
> --
>
> Key: CASSANDRA-14574
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14574
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
>Reporter: Aleksey Yeschenko
>Assignee: Jason Brown
>Priority: Major
> Fix For: 4.0
>
>
> {{MessageInHandler.decode()}} occasionally reads the payload incorrectly,
> passing the full message to {{MessageIn.read()}} instead of just the payload
> bytes.
> You can see the stack trace in the logs from this [CI
> run|https://circleci.com/gh/iamaleksey/cassandra/437#tests/containers/38].
> {code}
> Caused by: java.lang.AssertionError: null
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:351)
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:371)
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:335)
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:158)
> at
> org.apache.cassandra.net.async.MessageInHandler.decode(MessageInHandler.java:132)
> {code}
> Reconstructed, truncated stream passed to {{MessageIn.read()}}:
> {{000b000743414c5f42414301002a01e1a5c9b089fd11e8b517436ee124300704005d10fc50ec}}
> You can clearly see parameters in there encoded before the payload:
> {{[43414c5f424143 - CAL_BAC] [01 - ONE_BYTE] [002a - 42, payload size] 01 e1
> a5 c9 b0 89 fd 11 e8 b5 17 43 6e e1 24 30 07 04 00 00 00 1d 10 fc 50 ec}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14652) Extend IAuthenticator to accept peer SSL certificates
[ https://issues.apache.org/jira/browse/CASSANDRA-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583109#comment-16583109 ] Dinesh Joshi commented on CASSANDRA-14652: -- ||trunk|| |[branch|https://github.com/dineshjoshi/cassandra/tree/14652-trunk]| |[utests dtests|https://circleci.com/gh/dineshjoshi/workflows/cassandra/tree/14652-trunk]| || > Extend IAuthenticator to accept peer SSL certificates > - > > Key: CASSANDRA-14652 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14652 > Project: Cassandra > Issue Type: Improvement > Components: Auth >Reporter: Dinesh Joshi >Assignee: Dinesh Joshi >Priority: Major > Fix For: 4.0 > > > This patch will extend the IAuthenticator interface to accept peer's SSL > certificates. This will allow the Authenticator implementations to perform > additional checks from the client, if so desired. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14652) Extend IAuthenticator to accept peer SSL certificates
[ https://issues.apache.org/jira/browse/CASSANDRA-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Joshi updated CASSANDRA-14652: - Fix Version/s: 4.0 Status: Patch Available (was: Open) > Extend IAuthenticator to accept peer SSL certificates > - > > Key: CASSANDRA-14652 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14652 > Project: Cassandra > Issue Type: Improvement > Components: Auth >Reporter: Dinesh Joshi >Assignee: Dinesh Joshi >Priority: Major > Fix For: 4.0 > > > This patch will extend the IAuthenticator interface to accept peer's SSL > certificates. This will allow the Authenticator implementations to perform > additional checks from the client, if so desired. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14652) Extend IAuthenticator to accept peer SSL certificates
Dinesh Joshi created CASSANDRA-14652: Summary: Extend IAuthenticator to accept peer SSL certificates Key: CASSANDRA-14652 URL: https://issues.apache.org/jira/browse/CASSANDRA-14652 Project: Cassandra Issue Type: Improvement Components: Auth Reporter: Dinesh Joshi Assignee: Dinesh Joshi This patch will extend the IAuthenticator interface to accept peer's SSL certificates. This will allow the Authenticator implementations to perform additional checks from the client, if so desired. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14649) Index summaries fail when their size gets > 2G and use more space than necessary
[
https://issues.apache.org/jira/browse/CASSANDRA-14649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582965#comment-16582965
]
Jeff Jirsa commented on CASSANDRA-14649:
Safe to relate to CASSANDRA-12014 (haven't read the patch, just sounds
familiar)?
> Index summaries fail when their size gets > 2G and use more space than
> necessary
>
>
> Key: CASSANDRA-14649
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14649
> Project: Cassandra
> Issue Type: Bug
>Reporter: Branimir Lambov
>Assignee: Branimir Lambov
>Priority: Major
>
> After building a summary, {{IndexSummaryBuilder}} tries to trim the memory
> writers by calling {{SafeMemoryWriter.setCapacity(capacity())}}. Instead of
> trimming, this ends up allocating at least as much extra space and failing
> the {{Buffer.position()}} call when the size is greater than
> {{Integer.MAX_VALUE}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-11990) Address rows rather than partitions in SASI
[ https://issues.apache.org/jira/browse/CASSANDRA-11990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jordan West updated CASSANDRA-11990: Status: Patch Available (was: Open) I've taken another stab at this. There are some small optimizations I need to make before this merges and more tests to write but I wanted to get this up for review since its not a small patch. [branch|https://github.com/jrwest/cassandra/tree/11990-trunk] [tests|https://circleci.com/gh/jrwest/cassandra/tree/11990-trunk] cc/ [~ifesdjeen] > Address rows rather than partitions in SASI > --- > > Key: CASSANDRA-11990 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11990 > Project: Cassandra > Issue Type: Improvement > Components: CQL, sasi >Reporter: Alex Petrov >Assignee: Jordan West >Priority: Major > Fix For: 4.x > > Attachments: perf.pdf, size_comparison.png > > > Currently, the lookup in SASI index would return the key position of the > partition. After the partition lookup, the rows are iterated and the > operators are applied in order to filter out ones that do not match. > bq. TokenTree which accepts variable size keys (such would enable different > partitioners, collections support, primary key indexing etc.), -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14651) No longer possible to specify cassandra_dir via pytest.ini on cassandra-dtest
[
https://issues.apache.org/jira/browse/CASSANDRA-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582878#comment-16582878
]
Paulo Motta edited comment on CASSANDRA-14651 at 8/16/18 6:05 PM:
--
bq. I believe it was CASSANDRA-14134 that intentionally removed ini support.
oh right, sorry for the confusion.
one downside of supporting {{pytest.ini}} as currently is, is that if you make
any custom changes it will show up in the git diff.. I wonder if we should turn
that into a template file instead.
btw, just ninja-ed adding {{.pytest_cache}} on .gitignore to prevent it from
being added by mistake to the repo.
was (Author: pauloricardomg):
bq. I believe it was CASSANDRA-14134 that intentionally removed ini support.
oh right, sorry for the confusion.
one downside of supporting {{pytest.ini}} as currently is, is that if you make
any custom changes it will show up in the git diff.. I wonder if we should turn
that into a template file instead.
> No longer possible to specify cassandra_dir via pytest.ini on cassandra-dtest
> -
>
> Key: CASSANDRA-14651
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14651
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Trivial
>
> It seems like ability to specify {{cassandra_dir}} via {{pytest.init}}, as
> [stated on the
> doc|https://github.com/apache/cassandra-dtest/blame/master/README.md#L79] was
> lost after CASSANDRA-14449. We should either get it back or remove it from
> the doc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14651) No longer possible to specify cassandra_dir via pytest.ini on cassandra-dtest
[
https://issues.apache.org/jira/browse/CASSANDRA-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582878#comment-16582878
]
Paulo Motta commented on CASSANDRA-14651:
-
bq. I believe it was CASSANDRA-14134 that intentionally removed ini support.
oh right, sorry for the confusion.
one downside of supporting {{pytest.ini}} as currently is, is that if you make
any custom changes it will show up in the git diff.. I wonder if we should turn
that into a template file instead.
> No longer possible to specify cassandra_dir via pytest.ini on cassandra-dtest
> -
>
> Key: CASSANDRA-14651
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14651
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Trivial
>
> It seems like ability to specify {{cassandra_dir}} via {{pytest.init}}, as
> [stated on the
> doc|https://github.com/apache/cassandra-dtest/blame/master/README.md#L79] was
> lost after CASSANDRA-14449. We should either get it back or remove it from
> the doc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14651) No longer possible to specify cassandra_dir via pytest.ini on cassandra-dtest
[
https://issues.apache.org/jira/browse/CASSANDRA-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West updated CASSANDRA-14651:
Reviewer: Jordan West
> No longer possible to specify cassandra_dir via pytest.ini on cassandra-dtest
> -
>
> Key: CASSANDRA-14651
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14651
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Trivial
>
> It seems like ability to specify {{cassandra_dir}} via {{pytest.init}}, as
> [stated on the
> doc|https://github.com/apache/cassandra-dtest/blame/master/README.md#L79] was
> lost after CASSANDRA-14449. We should either get it back or remove it from
> the doc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14651) No longer possible to specify cassandra_dir via pytest.ini on cassandra-dtest
[
https://issues.apache.org/jira/browse/CASSANDRA-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582874#comment-16582874
]
Jordan West commented on CASSANDRA-14651:
-
[~pauloricardomg] I believe it was CASSANDRA-14134 that intentionally removed
{{ini}} support. That said, I'm not personally against it, especially since its
documented behavior. I will review.
> No longer possible to specify cassandra_dir via pytest.ini on cassandra-dtest
> -
>
> Key: CASSANDRA-14651
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14651
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Trivial
>
> It seems like ability to specify {{cassandra_dir}} via {{pytest.init}}, as
> [stated on the
> doc|https://github.com/apache/cassandra-dtest/blame/master/README.md#L79] was
> lost after CASSANDRA-14449. We should either get it back or remove it from
> the doc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Resolved] (CASSANDRA-14646) built_views entries are not removed after dropping keyspace
[
https://issues.apache.org/jira/browse/CASSANDRA-14646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paulo Motta resolved CASSANDRA-14646.
-
Resolution: Fixed
Fix Version/s: (was: 4.x)
4.0
LGTM, commited dtest as {{e426ce1daa1d52983b8823388e568e5097254c0c}} to master
and patch as {{d3e6891ec33a6e65cf383ad346c452293cfe50ec}} to trunk.
ftr, dtest looks good on private CI and also double checked locally.
Thanks!
> built_views entries are not removed after dropping keyspace
> ---
>
> Key: CASSANDRA-14646
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14646
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata, Materialized Views
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Major
> Fix For: 4.0
>
>
> If we restore view schema after dropping keyspace, view build won't be
> triggered because it was marked as SUCCESS in {{built_views}} table.
> | patch | CI |
> | [trunk|https://github.com/jasonstack/cassandra/commits/mv_drop_ks] |
> [utest|https://circleci.com/gh/jasonstack/cassandra/739] |
> | [dtest|https://github.com/apache/cassandra-dtest/pull/36]|
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
cassandra-dtest git commit: Add test for CASSANDRA-14646: built_views entries are not removed when dropping keyspace
Repository: cassandra-dtest
Updated Branches:
refs/heads/master bd419a7ae -> e426ce1da
Add test for CASSANDRA-14646: built_views entries are not removed when dropping
keyspace
Project: http://git-wip-us.apache.org/repos/asf/cassandra-dtest/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra-dtest/commit/e426ce1d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra-dtest/tree/e426ce1d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra-dtest/diff/e426ce1d
Branch: refs/heads/master
Commit: e426ce1daa1d52983b8823388e568e5097254c0c
Parents: bd419a7
Author: Zhao Yang
Authored: Tue Aug 14 16:31:12 2018 +0800
Committer: Paulo Motta
Committed: Thu Aug 16 12:01:10 2018 -0300
--
materialized_views_test.py | 76 +
1 file changed, 76 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/cassandra-dtest/blob/e426ce1d/materialized_views_test.py
--
diff --git a/materialized_views_test.py b/materialized_views_test.py
index f843dc8..717d799 100644
--- a/materialized_views_test.py
+++ b/materialized_views_test.py
@@ -169,6 +169,81 @@ class TestMaterializedViews(Tester):
result = list(node_session.execute("SELECT count(*) FROM
system.batches;"))
assert result[0].count == 0
+def _assert_view_meta(self, session, views, exists=True, nodes=2):
+if exists:
+assert_one(session, "SELECT COUNT(*) FROM system.built_views",
[views])
+if self.cluster.version() >= '3.11':
+assert_one(session, "SELECT COUNT(*) FROM
system_distributed.view_build_status", [views * nodes])
+else:
+assert_none(session, "SELECT * FROM system.built_views")
+if self.cluster.version() >= '3.11':
+assert_none(session, "SELECT * FROM
system_distributed.view_build_status")
+assert_none(session, "SELECT * FROM
{}".format(self._build_progress_table()))
+
+def test_view_metadata_cleanup(self):
+"""
+drop keyspace or view should clear built_views and view_build_status
+"""
+session = self.prepare(rf=2, nodes=2)
+
+def populate_data(session, rows):
+logger.debug("populate base data")
+for v in range(rows):
+session.execute("INSERT INTO t(k,c,a,b,e,f)
VALUES({v},{v},{v},{v},{v},{v})".format(v=v))
+
+def verify_data(session, rows, views):
+logger.debug("verify view data")
+for v in range(rows):
+for view in range(views):
+assert_one(session, "SELECT * FROM mv{} WHERE k={v} AND
c={v}".format(view, v=v), [v, v, v, v, v, v])
+
+def create_keyspace(session, ks="ks1", rf=2):
+create_ks(session, ks, rf)
+
+def create_table(session):
+logger.debug("create base table")
+session.execute("CREATE TABLE t (k int, c int, a int, b int, e
int, f int, primary key(k, c))")
+
+def create_views(session, views, keyspace="ks1"):
+logger.debug("create view")
+for view in range(views):
+session.execute("CREATE MATERIALIZED VIEW mv{} AS SELECT *
FROM t "
+"WHERE k IS NOT NULL AND c IS NOT NULL PRIMARY
KEY (c,k)".format(view))
+for view in range(views):
+self._wait_for_view(keyspace, "mv{}".format(view))
+
+def drop_keyspace(session, keyspace="ks1"):
+logger.debug("drop keyspace {}".format(keyspace))
+session.execute("DROP KEYSPACE IF EXISTS {}".format(keyspace))
+
+def drop_views(session, views):
+logger.debug("drop all views")
+for view in range(views):
+session.execute("DROP MATERIALIZED VIEW IF EXISTS
mv{}".format(view))
+
+rows = 100
+views = 5
+
+create_keyspace(session)
+create_table(session)
+populate_data(session, rows)
+create_views(session, views)
+verify_data(session, rows, views)
+
+self._assert_view_meta(session, views)
+drop_keyspace(session)
+self._assert_view_meta(session, views, exists=False)
+
+create_keyspace(session)
+create_table(session)
+populate_data(session, rows)
+create_views(session, views)
+verify_data(session, rows, views)
+
+self._assert_view_meta(session, views)
+drop_views(session, views)
+self._assert_view_meta(session, views, exists=False)
+
def test_create(self):
"""Test the materialized view creation"""
session = self.prepare(user_table=True)
@@ -1099,6 +1174,7 @@ class TestMaterializedViews(Tester):
except InvalidRequest:
failed = True
cassandra git commit: Clear view system metadata when dropping keyspace
Repository: cassandra
Updated Branches:
refs/heads/trunk 76c1b5557 -> d3e6891ec
Clear view system metadata when dropping keyspace
Patch by Zhao Yang; Reviewed by Paulo Motta for CASSANDRA-14646
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d3e6891e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d3e6891e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d3e6891e
Branch: refs/heads/trunk
Commit: d3e6891ec33a6e65cf383ad346c452293cfe50ec
Parents: 76c1b55
Author: Zhao Yang
Authored: Thu Aug 16 13:53:30 2018 +0800
Committer: Paulo Motta
Committed: Thu Aug 16 11:56:27 2018 -0300
--
CHANGES.txt | 1 +
.../apache/cassandra/db/view/ViewManager.java | 24 ++--
.../org/apache/cassandra/schema/Schema.java | 4 ++--
3 files changed, 10 insertions(+), 19 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/d3e6891e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4cd4cc5..d8aca56 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
4.0
+ * Clear view system metadata when dropping keyspace (CASSANDRA-14646)
* Allocate ReentrantLock on-demand in java11 AtomicBTreePartitionerBase
(CASSANDRA-14637)
* Make all existing virtual tables use LocalPartitioner (CASSANDRA-14640)
* Revert 4.0 GC alg back to CMS (CASANDRA-14636)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/d3e6891e/src/java/org/apache/cassandra/db/view/ViewManager.java
--
diff --git a/src/java/org/apache/cassandra/db/view/ViewManager.java
b/src/java/org/apache/cassandra/db/view/ViewManager.java
index 0d565ae..000477d 100644
--- a/src/java/org/apache/cassandra/db/view/ViewManager.java
+++ b/src/java/org/apache/cassandra/db/view/ViewManager.java
@@ -104,12 +104,6 @@ public class ViewManager
newViewsByName.put(definition.name(), definition);
}
-for (String viewName : viewsByName.keySet())
-{
-if (!newViewsByName.containsKey(viewName))
-removeView(viewName);
-}
-
for (Map.Entry entry : newViewsByName.entrySet())
{
if (!viewsByName.containsKey(entry.getKey()))
@@ -157,28 +151,24 @@ public class ViewManager
viewsByName.put(definition.name(), view);
}
-public void removeView(String name)
+/**
+ * Stops the building of the specified view, no-op if it isn't building.
+ *
+ * @param name the name of the view
+ */
+public void dropView(String name)
{
View view = viewsByName.remove(name);
if (view == null)
return;
+view.stopBuild();
forTable(view.getDefinition().baseTableId).removeByName(name);
SystemKeyspace.setViewRemoved(keyspace.getName(), view.name);
SystemDistributedKeyspace.setViewRemoved(keyspace.getName(),
view.name);
}
-/**
- * Stops the building of the specified view, no-op if it isn't building.
- *
- * @param name the name of the view
- */
-public void stopBuild(String name)
-{
-viewsByName.get(name).stopBuild();
-}
-
public View getByName(String name)
{
return viewsByName.get(name);
http://git-wip-us.apache.org/repos/asf/cassandra/blob/d3e6891e/src/java/org/apache/cassandra/schema/Schema.java
--
diff --git a/src/java/org/apache/cassandra/schema/Schema.java
b/src/java/org/apache/cassandra/schema/Schema.java
index fc09c24..e1353cd 100644
--- a/src/java/org/apache/cassandra/schema/Schema.java
+++ b/src/java/org/apache/cassandra/schema/Schema.java
@@ -660,7 +660,7 @@ public final class Schema
delta.tables.altered.forEach(diff -> alterTable(diff.after));
delta.views.altered.forEach(diff -> alterView(diff.after));
-// deal with all removed, added, and altered views
+// deal with all added, and altered views
Keyspace.open(delta.after.name).viewManager.reload(true);
// notify on everything dropped
@@ -720,7 +720,7 @@ public final class Schema
private void dropView(ViewMetadata metadata)
{
-
Keyspace.open(metadata.keyspace()).viewManager.stopBuild(metadata.name());
+
Keyspace.open(metadata.keyspace()).viewManager.dropView(metadata.name());
dropTable(metadata.metadata);
}
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14651) No longer possible to specify cassandra_dir via pytest.ini on cassandra-dtest
[
https://issues.apache.org/jira/browse/CASSANDRA-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paulo Motta updated CASSANDRA-14651:
Status: Patch Available (was: Open)
trivial patch adding back ability to specify cassandra dir via cassandra_dir
pytest.ini property:
[branch|http://www.github.com/pauloricardomg/cassandra-dtest/tree/CASSANDRA-14651]
mind taking a look [~aweisberg] [~jrwest] ?
> No longer possible to specify cassandra_dir via pytest.ini on cassandra-dtest
> -
>
> Key: CASSANDRA-14651
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14651
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Trivial
>
> It seems like ability to specify {{cassandra_dir}} via {{pytest.init}}, as
> [stated on the
> doc|https://github.com/apache/cassandra-dtest/blame/master/README.md#L79] was
> lost after CASSANDRA-14449. We should either get it back or remove it from
> the doc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14651) No longer possible to specify cassandra_dir via pytest.ini on cassandra-dtest
Paulo Motta created CASSANDRA-14651:
---
Summary: No longer possible to specify cassandra_dir via
pytest.ini on cassandra-dtest
Key: CASSANDRA-14651
URL: https://issues.apache.org/jira/browse/CASSANDRA-14651
Project: Cassandra
Issue Type: Bug
Components: Testing
Reporter: Paulo Motta
Assignee: Paulo Motta
It seems like ability to specify {{cassandra_dir}} via {{pytest.init}}, as
[stated on the
doc|https://github.com/apache/cassandra-dtest/blame/master/README.md#L79] was
lost after CASSANDRA-14449. We should either get it back or remove it from the
doc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14649) Index summaries fail when their size gets > 2G and use more space than necessary
[
https://issues.apache.org/jira/browse/CASSANDRA-14649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582769#comment-16582769
]
Benedict commented on CASSANDRA-14649:
--
You're right, I was just assuming {{length()}} was inherited from
{{DataOutputBuffer}}
I realise {{ensureCapacity}} was a terrible suggestion, but {{ensureHeadroom}}
might be clearer than {{expandToFit}} since it might be that you're trying to
fit N total, not N extra.
But no strong feeling, and +1 either way.
> Index summaries fail when their size gets > 2G and use more space than
> necessary
>
>
> Key: CASSANDRA-14649
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14649
> Project: Cassandra
> Issue Type: Bug
>Reporter: Branimir Lambov
>Assignee: Branimir Lambov
>Priority: Major
>
> After building a summary, {{IndexSummaryBuilder}} tries to trim the memory
> writers by calling {{SafeMemoryWriter.setCapacity(capacity())}}. Instead of
> trimming, this ends up allocating at least as much extra space and failing
> the {{Buffer.position()}} call when the size is greater than
> {{Integer.MAX_VALUE}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14408) Transient Replication: Incremental & Validation repair handling of transient replicas
[ https://issues.apache.org/jira/browse/CASSANDRA-14408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ariel Weisberg updated CASSANDRA-14408: --- Reviewers: Ariel Weisberg, Benedict, Marcus Eriksson Reviewer: Marcus Eriksson > Transient Replication: Incremental & Validation repair handling of transient > replicas > - > > Key: CASSANDRA-14408 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14408 > Project: Cassandra > Issue Type: Sub-task > Components: Repair >Reporter: Ariel Weisberg >Assignee: Blake Eggleston >Priority: Major > Fix For: 4.0 > > > At transient replicas anti-compaction shouldn't output any data for transient > ranges as the data will be dropped after repair. > Transient replicas should also never have data streamed to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13304) Add checksumming to the native protocol
[
https://issues.apache.org/jira/browse/CASSANDRA-13304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582689#comment-16582689
]
Benedict commented on CASSANDRA-13304:
--
It's worth bearing in mind that the TLS variant may be offloaded to the NIC,
and throughput on servers with this capability should be unaffected.
This actually possibly is a downside, however, as this may be a correlated
bit-flip risk.
> Add checksumming to the native protocol
> ---
>
> Key: CASSANDRA-13304
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13304
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Sam Tunnicliffe
>Priority: Blocker
> Labels: client-impacting
> Fix For: 4.x
>
> Attachments: 13304_v1.diff, boxplot-read-throughput.png,
> boxplot-write-throughput.png
>
>
> The native binary transport implementation doesn't include checksums. This
> makes it highly susceptible to silently inserting corrupted data either due
> to hardware issues causing bit flips on the sender/client side, C*/receiver
> side, or network in between.
> Attaching an implementation that makes checksum'ing mandatory (assuming both
> client and server know about a protocol version that supports checksums) --
> and also adds checksumming to clients that request compression.
> The serialized format looks something like this:
> {noformat}
> * 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
> * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Number of Compressed Chunks | Compressed Length (e1)/
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * / Compressed Length cont. (e1) |Uncompressed Length (e1) /
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length cont. (e1)| CRC32 Checksum of Lengths (e1)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Checksum of Lengths cont. (e1)|Compressed Bytes (e1)+//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e1) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (e2)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (e2) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e2) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (en)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (en) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (en) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> {noformat}
> The first pass here adds checksums only to the actual contents of the frame
> body itself (and doesn't actually checksum lengths and headers). While it
> would be great to fully add checksuming across the entire protocol, the
> proposed implementation will ensure we at least catch corrupted data and
> likely protect ourselves pretty well anyways.
> I didn't go to the trouble of implementing a Snappy Checksum'ed Compressor
> implementation as it's been deprecated for a while -- is really slow and
> crappy compared to LZ4 -- and we should do everything in our power to make
> sure no one in the community is still using it. I left it in (for obvious
> backwards compatibility aspects) old for clients that don't know about the
> new protocol.
> The current protocol has a 256MB (max) frame body -- where the serialized
> contents are simply written in to the frame body.
> If the client sends a compression option in the startup, we will install a
> FrameCompressor inline. Unfortunately, we went with a
[jira] [Commented] (CASSANDRA-14648) CircleCI dtest runs should (by default) depend upon successful unit tests
[ https://issues.apache.org/jira/browse/CASSANDRA-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582672#comment-16582672 ] Benedict commented on CASSANDRA-14648: -- Yes, that's also a good point. We could make the test serial. If the tests were set to run serially (which can be configured in the CircleCI yaml), it should create a test graph - and you should be able to click on the bits of the graph and tell it to run those parts. I've not as-yet tried this for a sequence of not-run pieces, but I am happy to confirm if it works. > CircleCI dtest runs should (by default) depend upon successful unit tests > - > > Key: CASSANDRA-14648 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14648 > Project: Cassandra > Issue Type: Improvement >Reporter: Benedict >Assignee: Benedict >Priority: Major > > Unit tests are very quick to run, and if they fail to pass there’s probably > no value in running dtests - particularly if we are honouring our > expectations of never committing code that breaks either unit or dtests. > When sharing CircleCI resources between multiple branches (or multiple > users), it is wasteful to have two dtest runs kicked off for every incomplete > branch that is pushed to GitHub for safe keeping. So I think a better > default CircleCI config file would only run the dtests after a successful > unit test run, and those who want to modify this behaviour can do so > consciously by editing the config file for themselves. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8969) Add indication in cassandra.yaml that rpc timeouts going too high will cause memory build up
[ https://issues.apache.org/jira/browse/CASSANDRA-8969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582668#comment-16582668 ] Jeremy Hanna commented on CASSANDRA-8969: - Any more thoughts on this change to make sure people don't unwittingly cause heap problems by setting the timeout extremely high? > Add indication in cassandra.yaml that rpc timeouts going too high will cause > memory build up > > > Key: CASSANDRA-8969 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8969 > Project: Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Jeremy Hanna >Assignee: Jeremy Hanna >Priority: Minor > Labels: lhf > Fix For: 3.11.x > > Attachments: 8969.txt > > > It would be helpful to communicate that setting the rpc timeouts too high may > cause memory problems on the server as it can become overloaded and has to > retain the in flight requests in memory. I'll get this done but just adding > the ticket as a placeholder for memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14648) CircleCI dtest runs should (by default) depend upon successful unit tests
[ https://issues.apache.org/jira/browse/CASSANDRA-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582660#comment-16582660 ] Ariel Weisberg commented on CASSANDRA-14648: Also another big source of redundancy is vnodes and novnodes. I would like to run just one until everything is passing. > CircleCI dtest runs should (by default) depend upon successful unit tests > - > > Key: CASSANDRA-14648 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14648 > Project: Cassandra > Issue Type: Improvement >Reporter: Benedict >Assignee: Benedict >Priority: Major > > Unit tests are very quick to run, and if they fail to pass there’s probably > no value in running dtests - particularly if we are honouring our > expectations of never committing code that breaks either unit or dtests. > When sharing CircleCI resources between multiple branches (or multiple > users), it is wasteful to have two dtest runs kicked off for every incomplete > branch that is pushed to GitHub for safe keeping. So I think a better > default CircleCI config file would only run the dtests after a successful > unit test run, and those who want to modify this behaviour can do so > consciously by editing the config file for themselves. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14635) Support table level configuration of monotonic reads
[ https://issues.apache.org/jira/browse/CASSANDRA-14635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-14635: -- Reviewer: Aleksey Yeschenko > Support table level configuration of monotonic reads > > > Key: CASSANDRA-14635 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14635 > Project: Cassandra > Issue Type: Improvement > Components: Coordination >Reporter: Ariel Weisberg >Assignee: Blake Eggleston >Priority: Major > > In CASSANDRA-10726 it was discussed that allowing expert users to forgo > monotonic reads might be desirable. It is practical to control monotonicity > of reads at a fine grained level because it involves changing the behavior of > read repair on a per read basis. > Per CASSANDRA-14593 we already don't preserve update atomicity down to the > column level. You could read the key out of a row and read repair the key, > pass the key to another process which attempts to read the value, but finds > the value is null because read repair only repairs the data (including > columns) that is part of the read. IMO it's a stretch to say that reads are > monotonic. It is technically correct, the best kind of correct, but far from > as useful as it should be. > An initial implementation could make read repair asynchronous or forgo read > repair entirely. This would improve the throughput and latency of reads. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14648) CircleCI dtest runs should (by default) depend upon successful unit tests
[ https://issues.apache.org/jira/browse/CASSANDRA-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582658#comment-16582658 ] Ariel Weisberg commented on CASSANDRA-14648: bq. Well, that isn't a very good citizen approach IMO. For starters, we shouldn't have flakey tests, and this would encourage us to fix them. It's also trivial to manually kick off the dtests if the failed unit tests are inspected to be OK. That assumes a failed unit tests means dtests are also going to just implode and not bear information and IME it's rarely true. bq. I'd personally prefer we only ran dtests on a manual request, as they should only be needed a fraction of the frequency I think they're requested. This is an orthogonal concern and I do agree with it. I usually end up killing a lot of CircleCI runs spawned automatically due to git activity. Have you been having trouble getting your stuff run? I feel like I have never waited for containers. Is it actually possible to get the dtests to run manually if the code in your branch is checked and says "don't run the dtests"? Does it even create the jobs for that commit for you? > CircleCI dtest runs should (by default) depend upon successful unit tests > - > > Key: CASSANDRA-14648 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14648 > Project: Cassandra > Issue Type: Improvement >Reporter: Benedict >Assignee: Benedict >Priority: Major > > Unit tests are very quick to run, and if they fail to pass there’s probably > no value in running dtests - particularly if we are honouring our > expectations of never committing code that breaks either unit or dtests. > When sharing CircleCI resources between multiple branches (or multiple > users), it is wasteful to have two dtest runs kicked off for every incomplete > branch that is pushed to GitHub for safe keeping. So I think a better > default CircleCI config file would only run the dtests after a successful > unit test run, and those who want to modify this behaviour can do so > consciously by editing the config file for themselves. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14646) built_views entries are not removed after dropping keyspace
[
https://issues.apache.org/jira/browse/CASSANDRA-14646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paulo Motta updated CASSANDRA-14646:
Reviewer: Paulo Motta
> built_views entries are not removed after dropping keyspace
> ---
>
> Key: CASSANDRA-14646
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14646
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata, Materialized Views
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Major
> Fix For: 4.x
>
>
> If we restore view schema after dropping keyspace, view build won't be
> triggered because it was marked as SUCCESS in {{built_views}} table.
> | patch | CI |
> | [trunk|https://github.com/jasonstack/cassandra/commits/mv_drop_ks] |
> [utest|https://circleci.com/gh/jasonstack/cassandra/739] |
> | [dtest|https://github.com/apache/cassandra-dtest/pull/36]|
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13304) Add checksumming to the native protocol
[
https://issues.apache.org/jira/browse/CASSANDRA-13304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582626#comment-16582626
]
Ariel Weisberg commented on CASSANDRA-13304:
bq. Is that their expectation? Or that the checksum is reliable?
I don't think we should use xxhash I think we should use CRC32. The hardware
and API support for it are excellent and it provides strong enough guarantees
for transmission errors. Sure it's not a good hash function and it's collision
prone, but we aren't using it as a hash function.
bq. I ran a quick SHA1 benchmark to be sure, but it looks like it's able to
hash ~1GB/s per thread on my test machine, which should be fast enough for it
to go unnoticeable
That's not fast. That's glacially slow. And it's also not the performance you
get when you have to warm up the hash function each time you checksum a small
message. Startup time matters for hash functions used to hash many small items.
Or am I wrong in assuming you were hashing a relatively large amount of data in
that benchmark? 10s of kilobytes or more hashed in a tight loop is enough to
show unrealistically fast performance.
Also are you sure the implementation you are using matches the one that TLS
uses? There is hardware support for SHA-1 I believe so it might be faster.
IMO to really know if this matters someone has to measure implementations with
Cassandra stress.
There is also more to this than just hash functions. Using TLS means going
through SSLEngine or whatever we use to interface with SSL. Netty might have
its own SSL thing right now that isn't just a wrapper around the Java thing.
That's not necessarily free either. I don't think we should be changing
defaults without measuring impact against a cluster with at least RF=3:3 (2 DC
cluster).
> Add checksumming to the native protocol
> ---
>
> Key: CASSANDRA-13304
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13304
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Sam Tunnicliffe
>Priority: Blocker
> Labels: client-impacting
> Fix For: 4.x
>
> Attachments: 13304_v1.diff, boxplot-read-throughput.png,
> boxplot-write-throughput.png
>
>
> The native binary transport implementation doesn't include checksums. This
> makes it highly susceptible to silently inserting corrupted data either due
> to hardware issues causing bit flips on the sender/client side, C*/receiver
> side, or network in between.
> Attaching an implementation that makes checksum'ing mandatory (assuming both
> client and server know about a protocol version that supports checksums) --
> and also adds checksumming to clients that request compression.
> The serialized format looks something like this:
> {noformat}
> * 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
> * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Number of Compressed Chunks | Compressed Length (e1)/
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * / Compressed Length cont. (e1) |Uncompressed Length (e1) /
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length cont. (e1)| CRC32 Checksum of Lengths (e1)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Checksum of Lengths cont. (e1)|Compressed Bytes (e1)+//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e1) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (e2)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (e2) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e2) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (en)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (en) |
> *
[jira] [Updated] (CASSANDRA-14638) Column result order can change in 'SELECT *' results when upgrading from 2.1 to 3.0 causing response corruption for queries using prepared statements when static col
[
https://issues.apache.org/jira/browse/CASSANDRA-14638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Yeschenko updated CASSANDRA-14638:
--
Resolution: Fixed
Fix Version/s: (was: 4.0.x)
(was: 3.11.x)
(was: 3.0.x)
4.0
3.11.4
3.0.18
Status: Resolved (was: Patch Available)
> Column result order can change in 'SELECT *' results when upgrading from 2.1
> to 3.0 causing response corruption for queries using prepared statements when
> static columns are used
> --
>
> Key: CASSANDRA-14638
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14638
> Project: Cassandra
> Issue Type: Bug
> Environment: Single C* node ccm cluster upgraded from C* 2.1.20 to
> 3.0.17
>Reporter: Andy Tolbert
>Assignee: Aleksey Yeschenko
>Priority: Major
> Fix For: 3.0.18, 3.11.4, 4.0
>
>
> When performing an upgrade from C* 2.1.20 to 3.0.17 I observed that the order
> of columns returned from a 'SELECT *' query changes, particularly when static
> columns are involved.
> This may not seem like that much of a problem, however if using Prepared
> Statements, any clients that remain connected during the upgrade may
> encounter issues consuming results from these queries, as data is reordered
> and the client not aware of it. The result definition is sent in the
> original prepared statement response, so if order changes the client has no
> way of knowing (until C* 4.0 via CASSANDRA-10786) without re-preparing, which
> is non-trivial as most client drivers cache prepared statements.
> This could lead to reading the wrong values for columns, which could result
> in some kind of deserialization exception or if the data types of the
> switched columns are compatible, the wrong values. This happens even if the
> client attempts to retrieve a column value by name (i.e. row.getInt("colx")).
> Unfortunately I don't think there is an easy fix for this. If the order was
> changed back to the previous format, you risk issues for users upgrading from
> older 3.0 version. I think it would be nice to add a note in the NEWS file
> in the 3.0 upgrade section that describes this issue, and how to work around
> it (specify all column names of interest explicitly in query).
> Example schema and code to reproduce:
>
> {noformat}
> create keyspace ks with replication = {'class': 'SimpleStrategy',
> 'replication_factor': 1};
> create table ks.tbl (p0 text,
> p1 text,
> m map static,
> t text,
> u text static,
> primary key (p0, p1)
> );
> insert into ks.tbl (p0, p1, m, t, u) values ('p0', 'p1', { 'm0' : 'm1' },
> 't', 'u');{noformat}
>
> When querying with 2.1 you'll observe the following order via cqlsh:
> {noformat}
> p0 | p1 | m| u | t
> ++--+---+---
> p0 | p1 | {'m0': 'm1'} | u | t{noformat}
>
> With 3.0, observe that u and m are transposed:
>
> {noformat}
> p0 | p1 | u | m | t
> ++---+--+---
> p0 | p1 | u | {'m0': 'm1'} | t{noformat}
>
>
> {code:java}
> import com.datastax.driver.core.BoundStatement;
> import com.datastax.driver.core.Cluster;
> import com.datastax.driver.core.ColumnDefinitions;
> import com.datastax.driver.core.PreparedStatement;
> import com.datastax.driver.core.ResultSet;
> import com.datastax.driver.core.Row;
> import com.datastax.driver.core.Session;
> import com.google.common.util.concurrent.Uninterruptibles;
> import java.util.concurrent.TimeUnit;
> public class LiveUpgradeTest {
> public static void main(String args[]) {
> Cluster cluster = Cluster.builder().addContactPoints("127.0.0.1").build();
> try {
> Session session = cluster.connect();
> PreparedStatement p = session.prepare("SELECT * from ks.tbl");
> BoundStatement bs = p.bind();
> // continually query every 30 seconds
> while (true) {
> try {
> ResultSet r = session.execute(bs);
> Row row = r.one();
> int i = 0;
> // iterate over the result metadata in order printing the
> // index, name, type, and length of the first row of data.
> for (ColumnDefinitions.Definition d : r.getColumnDefinitions()) {
> System.out.println(
> i++
> + ": "
> + d.getName()
> + " -> "
> + d.getType()
> + " -> val = "
> + row.getBytesUnsafe(d.getName()).array().length);
> }
> } catch (Throwable t) {
> t.printStackTrace();
[jira] [Commented] (CASSANDRA-14638) Column result order can change in 'SELECT *' results when upgrading from 2.1 to 3.0 causing response corruption for queries using prepared statements when static c
[
https://issues.apache.org/jira/browse/CASSANDRA-14638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582623#comment-16582623
]
Aleksey Yeschenko commented on CASSANDRA-14638:
---
Cheers, committed as
[236c47e65ce95dcc7c8c75706715b5a2a88fd237|https://github.com/apache/cassandra/commit/236c47e65ce95dcc7c8c75706715b5a2a88fd237]
to 3.0 and merged up to 3.11 and trunk.
> Column result order can change in 'SELECT *' results when upgrading from 2.1
> to 3.0 causing response corruption for queries using prepared statements when
> static columns are used
> --
>
> Key: CASSANDRA-14638
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14638
> Project: Cassandra
> Issue Type: Bug
> Environment: Single C* node ccm cluster upgraded from C* 2.1.20 to
> 3.0.17
>Reporter: Andy Tolbert
>Assignee: Aleksey Yeschenko
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
>
> When performing an upgrade from C* 2.1.20 to 3.0.17 I observed that the order
> of columns returned from a 'SELECT *' query changes, particularly when static
> columns are involved.
> This may not seem like that much of a problem, however if using Prepared
> Statements, any clients that remain connected during the upgrade may
> encounter issues consuming results from these queries, as data is reordered
> and the client not aware of it. The result definition is sent in the
> original prepared statement response, so if order changes the client has no
> way of knowing (until C* 4.0 via CASSANDRA-10786) without re-preparing, which
> is non-trivial as most client drivers cache prepared statements.
> This could lead to reading the wrong values for columns, which could result
> in some kind of deserialization exception or if the data types of the
> switched columns are compatible, the wrong values. This happens even if the
> client attempts to retrieve a column value by name (i.e. row.getInt("colx")).
> Unfortunately I don't think there is an easy fix for this. If the order was
> changed back to the previous format, you risk issues for users upgrading from
> older 3.0 version. I think it would be nice to add a note in the NEWS file
> in the 3.0 upgrade section that describes this issue, and how to work around
> it (specify all column names of interest explicitly in query).
> Example schema and code to reproduce:
>
> {noformat}
> create keyspace ks with replication = {'class': 'SimpleStrategy',
> 'replication_factor': 1};
> create table ks.tbl (p0 text,
> p1 text,
> m map static,
> t text,
> u text static,
> primary key (p0, p1)
> );
> insert into ks.tbl (p0, p1, m, t, u) values ('p0', 'p1', { 'm0' : 'm1' },
> 't', 'u');{noformat}
>
> When querying with 2.1 you'll observe the following order via cqlsh:
> {noformat}
> p0 | p1 | m| u | t
> ++--+---+---
> p0 | p1 | {'m0': 'm1'} | u | t{noformat}
>
> With 3.0, observe that u and m are transposed:
>
> {noformat}
> p0 | p1 | u | m | t
> ++---+--+---
> p0 | p1 | u | {'m0': 'm1'} | t{noformat}
>
>
> {code:java}
> import com.datastax.driver.core.BoundStatement;
> import com.datastax.driver.core.Cluster;
> import com.datastax.driver.core.ColumnDefinitions;
> import com.datastax.driver.core.PreparedStatement;
> import com.datastax.driver.core.ResultSet;
> import com.datastax.driver.core.Row;
> import com.datastax.driver.core.Session;
> import com.google.common.util.concurrent.Uninterruptibles;
> import java.util.concurrent.TimeUnit;
> public class LiveUpgradeTest {
> public static void main(String args[]) {
> Cluster cluster = Cluster.builder().addContactPoints("127.0.0.1").build();
> try {
> Session session = cluster.connect();
> PreparedStatement p = session.prepare("SELECT * from ks.tbl");
> BoundStatement bs = p.bind();
> // continually query every 30 seconds
> while (true) {
> try {
> ResultSet r = session.execute(bs);
> Row row = r.one();
> int i = 0;
> // iterate over the result metadata in order printing the
> // index, name, type, and length of the first row of data.
> for (ColumnDefinitions.Definition d : r.getColumnDefinitions()) {
> System.out.println(
> i++
> + ": "
> + d.getName()
> + " -> "
> + d.getType()
> + " -> val = "
> + row.getBytesUnsafe(d.getName()).array().length);
> }
> } catch (Throwable t) {
> t.printStackTrace();
> } finally {
>
[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/65a46820 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/65a46820 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/65a46820 Branch: refs/heads/trunk Commit: 65a46820bb6d28bc6e359807699e8c6ae562e0c4 Parents: 4f4c390 236c47e Author: Aleksey Yeshchenko Authored: Thu Aug 16 15:44:12 2018 +0100 Committer: Aleksey Yeshchenko Committed: Thu Aug 16 15:44:12 2018 +0100 -- CHANGES.txt | 1 + NEWS.txt| 10 +++ doc/cql3/CQL.textile| 2 +- src/java/org/apache/cassandra/db/Columns.java | 22 +-- .../org/apache/cassandra/db/ColumnsTest.java| 69 +++- 5 files changed, 95 insertions(+), 9 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/65a46820/CHANGES.txt -- diff --cc CHANGES.txt index e2d1681,eccece2..6464667 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,5 -1,5 +1,6 @@@ -3.0.18 +3.11.4 +Merged from 3.0: + * Fix static column order for SELECT * wildcard queries (CASSANDRA-14638) * sstableloader should use discovered broadcast address to connect intra-cluster (CASSANDRA-14522) * Fix reading columns with non-UTF names from schema (CASSANDRA-14468) Merged from 2.2: http://git-wip-us.apache.org/repos/asf/cassandra/blob/65a46820/NEWS.txt -- diff --cc NEWS.txt index 578e426,29864bc..ea53c36 --- a/NEWS.txt +++ b/NEWS.txt @@@ -42,7 -42,17 +42,17 @@@ restore snapshots created with the prev 'sstableloader' tool. You can upgrade the file format of your snapshots using the provided 'sstableupgrade' tool. -3.0.18 ++3.11.4 + == + + Upgrading + - + - The order of static columns in SELECT * has been fixed to match that of 2.0 and 2.1 - they are now sorted + alphabetically again, by their name, just like regular columns are. If you use prepared statements and + SELECT * queries, and have both simple and collection static columns in those tables, and are upgrading from an + earlier 3.0 version, then you might be affected by this change. Please see CASSANDRA-14638 for details. + -3.0.17 +3.11.3 = Upgrading http://git-wip-us.apache.org/repos/asf/cassandra/blob/65a46820/doc/cql3/CQL.textile -- diff --cc doc/cql3/CQL.textile index f32e30c,5f35d57..61b9d63 --- a/doc/cql3/CQL.textile +++ b/doc/cql3/CQL.textile @@@ -1115,9 -1079,9 +1115,9 @@@ The @SELECT@ statements reads one or mo h4(#selectSelection). @@ - The @@ determines which columns needs to be queried and returned in the result-set. It consists of either the comma-separated list of or the wildcard character (@*@) to select all the columns defined for the table. + The @@ determines which columns needs to be queried and returned in the result-set. It consists of either the comma-separated list of or the wildcard character (@*@) to select all the columns defined for the table. Please note that for wildcard @SELECT@ queries the order of columns returned is not specified and is not guaranteed to be stable between Cassandra versions. -A @@ is either a column name to retrieve or a @@ of one or more @@s. The function allowed are the same as for @@ and are described in the "function section":#functions. In addition to these generic functions, the @WRITETIME@ (resp. @TTL@) function allows to select the timestamp of when the column was inserted (resp. the time to live (in seconds) for the column (or null if the column has no expiration set)). +A @@ is either a column name to retrieve or a @@ of one or more @@s. The function allowed are the same as for @@ and are described in the "function section":#functions. In addition to these generic functions, the @WRITETIME@ (resp. @TTL@) function allows to select the timestamp of when the column was inserted (resp. the time to live (in seconds) for the column (or null if the column has no expiration set)) and the "@CAST@":#castFun function can be used to convert one data type to another. Any @@ can be aliased using @AS@ keyword (see examples). Please note that @@ and @@ clause should refer to the columns by their original names and not by their aliases. http://git-wip-us.apache.org/repos/asf/cassandra/blob/65a46820/src/java/org/apache/cassandra/db/Columns.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/65a46820/test/unit/org/apache/cassandra/db/ColumnsTest.java
[1/6] cassandra git commit: Fix static column order for SELECT * wildcard queries
Repository: cassandra
Updated Branches:
refs/heads/cassandra-3.0 3d48cbc74 -> 236c47e65
refs/heads/cassandra-3.11 4f4c390bc -> 65a46820b
refs/heads/trunk 07b0aca6d -> 76c1b5557
Fix static column order for SELECT * wildcard queries
patch by Aleksey Yeschenko; reviewed by Benedict Elliott Smith for
CASSANDRA-14638
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/236c47e6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/236c47e6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/236c47e6
Branch: refs/heads/cassandra-3.0
Commit: 236c47e65ce95dcc7c8c75706715b5a2a88fd237
Parents: 3d48cbc
Author: Aleksey Yeshchenko
Authored: Thu Aug 16 12:45:55 2018 +0100
Committer: Aleksey Yeshchenko
Committed: Thu Aug 16 15:43:51 2018 +0100
--
CHANGES.txt | 1 +
NEWS.txt| 10 +++
doc/cql3/CQL.textile| 2 +-
src/java/org/apache/cassandra/db/Columns.java | 22 +--
.../org/apache/cassandra/db/ColumnsTest.java| 69 +++-
5 files changed, 95 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/236c47e6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 7e4d88c..eccece2 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.0.18
+ * Fix static column order for SELECT * wildcard queries (CASSANDRA-14638)
* sstableloader should use discovered broadcast address to connect
intra-cluster (CASSANDRA-14522)
* Fix reading columns with non-UTF names from schema (CASSANDRA-14468)
Merged from 2.2:
http://git-wip-us.apache.org/repos/asf/cassandra/blob/236c47e6/NEWS.txt
--
diff --git a/NEWS.txt b/NEWS.txt
index 234154e..29864bc 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -42,6 +42,16 @@ restore snapshots created with the previous major version
using the
'sstableloader' tool. You can upgrade the file format of your snapshots
using the provided 'sstableupgrade' tool.
+3.0.18
+==
+
+Upgrading
+-
+- The order of static columns in SELECT * has been fixed to match that of
2.0 and 2.1 - they are now sorted
+ alphabetically again, by their name, just like regular columns are. If
you use prepared statements and
+ SELECT * queries, and have both simple and collection static columns in
those tables, and are upgrading from an
+ earlier 3.0 version, then you might be affected by this change. Please
see CASSANDRA-14638 for details.
+
3.0.17
=
http://git-wip-us.apache.org/repos/asf/cassandra/blob/236c47e6/doc/cql3/CQL.textile
--
diff --git a/doc/cql3/CQL.textile b/doc/cql3/CQL.textile
index cc2b9aa..5f35d57 100644
--- a/doc/cql3/CQL.textile
+++ b/doc/cql3/CQL.textile
@@ -1079,7 +1079,7 @@ The @SELECT@ statements reads one or more columns for one
or more rows in a tabl
h4(#selectSelection). @@
-The @@ determines which columns needs to be queried and
returned in the result-set. It consists of either the comma-separated list of
or the wildcard character (@*@) to select all the columns defined
for the table.
+The @@ determines which columns needs to be queried and
returned in the result-set. It consists of either the comma-separated list of
or the wildcard character (@*@) to select all the columns defined
for the table. Please note that for wildcard @SELECT@ queries the order of
columns returned is not specified and is not guaranteed to be stable between
Cassandra versions.
A @@ is either a column name to retrieve or a @@ of one or
more @@s. The function allowed are the same as for @@ and are
described in the "function section":#functions. In addition to these generic
functions, the @WRITETIME@ (resp. @TTL@) function allows to select the
timestamp of when the column was inserted (resp. the time to live (in seconds)
for the column (or null if the column has no expiration set)).
http://git-wip-us.apache.org/repos/asf/cassandra/blob/236c47e6/src/java/org/apache/cassandra/db/Columns.java
--
diff --git a/src/java/org/apache/cassandra/db/Columns.java
b/src/java/org/apache/cassandra/db/Columns.java
index eb4f761..45ce91e 100644
--- a/src/java/org/apache/cassandra/db/Columns.java
+++ b/src/java/org/apache/cassandra/db/Columns.java
@@ -50,7 +50,16 @@ public class Columns extends
AbstractCollection implements Col
{
public static final Serializer serializer = new Serializer();
public static final Columns NONE = new Columns(BTree.empty(), 0);
-public static final ColumnDefinition
[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk
Merge branch 'cassandra-3.11' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/76c1b555 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/76c1b555 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/76c1b555 Branch: refs/heads/trunk Commit: 76c1b555708fd91e126e9d86920e81793b599d16 Parents: 07b0aca 65a4682 Author: Aleksey Yeshchenko Authored: Thu Aug 16 15:45:53 2018 +0100 Committer: Aleksey Yeshchenko Committed: Thu Aug 16 15:47:09 2018 +0100 -- CHANGES.txt | 2 + NEWS.txt| 10 +++ doc/cql3/CQL.textile| 2 +- src/java/org/apache/cassandra/db/Columns.java | 22 +-- .../org/apache/cassandra/db/ColumnsTest.java| 69 +++- 5 files changed, 96 insertions(+), 9 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/76c1b555/CHANGES.txt -- diff --cc CHANGES.txt index 67e85f6,6464667..4cd4cc5 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,282 -1,7 +1,284 @@@ +4.0 + * Allocate ReentrantLock on-demand in java11 AtomicBTreePartitionerBase (CASSANDRA-14637) + * Make all existing virtual tables use LocalPartitioner (CASSANDRA-14640) + * Revert 4.0 GC alg back to CMS (CASANDRA-14636) + * Remove hardcoded java11 jvm args in idea workspace files (CASSANDRA-14627) + * Update netty to 4.1.128 (CASSANDRA-14633) + * Add a virtual table to expose thread pools (CASSANDRA-14523) + * Add a virtual table to expose caches (CASSANDRA-14538) + * Fix toDate function for timestamp arguments (CASSANDRA-14502) + * Revert running dtests by default in circleci (CASSANDRA-14614) + * Stream entire SSTables when possible (CASSANDRA-14556) + * Add experimental support for Java 11 (CASSANDRA-9608) + * Make PeriodicCommitLogService.blockWhenSyncLagsNanos configurable (CASSANDRA-14580) + * Improve logging in MessageInHandler's constructor (CASSANDRA-14576) + * Set broadcast address in internode messaging handshake (CASSANDRA-14579) + * Wait for schema agreement prior to building MVs (CASSANDRA-14571) + * Make all DDL statements idempotent and not dependent on global state (CASSANDRA-13426) + * Bump the hints messaging version to match the current one (CASSANDRA-14536) + * OffsetAwareConfigurationLoader doesn't set ssl storage port causing bind errors in CircleCI (CASSANDRA-14546) + * Report why native_transport_port fails to bind (CASSANDRA-14544) + * Optimize internode messaging protocol (CASSANDRA-14485) + * Internode messaging handshake sends wrong messaging version number (CASSANDRA-14540) + * Add a virtual table to expose active client connections (CASSANDRA-14458) + * Clean up and refactor client metrics (CASSANDRA-14524) + * Nodetool import row cache invalidation races with adding sstables to tracker (CASSANDRA-14529) + * Fix assertions in LWTs after TableMetadata was made immutable (CASSANDRA-14356) + * Abort compactions quicker (CASSANDRA-14397) + * Support light-weight transactions in cassandra-stress (CASSANDRA-13529) + * Make AsyncOneResponse use the correct timeout (CASSANDRA-14509) + * Add option to sanity check tombstones on reads/compactions (CASSANDRA-14467) + * Add a virtual table to expose all running sstable tasks (CASSANDRA-14457) + * Let nodetool import take a list of directories (CASSANDRA-14442) + * Avoid unneeded memory allocations / cpu for disabled log levels (CASSANDRA-14488) + * Implement virtual keyspace interface (CASSANDRA-7622) + * nodetool import cleanup and improvements (CASSANDRA-14417) + * Bump jackson version to >= 2.9.5 (CASSANDRA-14427) + * Allow nodetool toppartitions without specifying table (CASSANDRA-14360) + * Audit logging for database activity (CASSANDRA-12151) + * Clean up build artifacts in docs container (CASSANDRA-14432) + * Minor network authz improvements (Cassandra-14413) + * Automatic sstable upgrades (CASSANDRA-14197) + * Replace deprecated junit.framework.Assert usages with org.junit.Assert (CASSANDRA-14431) + * Cassandra-stress throws NPE if insert section isn't specified in user profile (CASSSANDRA-14426) + * List clients by protocol versions `nodetool clientstats --by-protocol` (CASSANDRA-14335) + * Improve LatencyMetrics performance by reducing write path processing (CASSANDRA-14281) + * Add network authz (CASSANDRA-13985) + * Use the correct IP/Port for Streaming when localAddress is left unbound (CASSANDRA-14389) + * nodetool listsnapshots is missing local system keyspace snapshots (CASSANDRA-14381) + * Remove StreamCoordinator.streamExecutor thread pool (CASSANDRA-14402) + * Rename nodetool --with-port to --print-port to disambiguate from --port (CASSANDRA-14392)
[2/6] cassandra git commit: Fix static column order for SELECT * wildcard queries
Fix static column order for SELECT * wildcard queries
patch by Aleksey Yeschenko; reviewed by Benedict Elliott Smith for
CASSANDRA-14638
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/236c47e6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/236c47e6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/236c47e6
Branch: refs/heads/cassandra-3.11
Commit: 236c47e65ce95dcc7c8c75706715b5a2a88fd237
Parents: 3d48cbc
Author: Aleksey Yeshchenko
Authored: Thu Aug 16 12:45:55 2018 +0100
Committer: Aleksey Yeshchenko
Committed: Thu Aug 16 15:43:51 2018 +0100
--
CHANGES.txt | 1 +
NEWS.txt| 10 +++
doc/cql3/CQL.textile| 2 +-
src/java/org/apache/cassandra/db/Columns.java | 22 +--
.../org/apache/cassandra/db/ColumnsTest.java| 69 +++-
5 files changed, 95 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/236c47e6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 7e4d88c..eccece2 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.0.18
+ * Fix static column order for SELECT * wildcard queries (CASSANDRA-14638)
* sstableloader should use discovered broadcast address to connect
intra-cluster (CASSANDRA-14522)
* Fix reading columns with non-UTF names from schema (CASSANDRA-14468)
Merged from 2.2:
http://git-wip-us.apache.org/repos/asf/cassandra/blob/236c47e6/NEWS.txt
--
diff --git a/NEWS.txt b/NEWS.txt
index 234154e..29864bc 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -42,6 +42,16 @@ restore snapshots created with the previous major version
using the
'sstableloader' tool. You can upgrade the file format of your snapshots
using the provided 'sstableupgrade' tool.
+3.0.18
+==
+
+Upgrading
+-
+- The order of static columns in SELECT * has been fixed to match that of
2.0 and 2.1 - they are now sorted
+ alphabetically again, by their name, just like regular columns are. If
you use prepared statements and
+ SELECT * queries, and have both simple and collection static columns in
those tables, and are upgrading from an
+ earlier 3.0 version, then you might be affected by this change. Please
see CASSANDRA-14638 for details.
+
3.0.17
=
http://git-wip-us.apache.org/repos/asf/cassandra/blob/236c47e6/doc/cql3/CQL.textile
--
diff --git a/doc/cql3/CQL.textile b/doc/cql3/CQL.textile
index cc2b9aa..5f35d57 100644
--- a/doc/cql3/CQL.textile
+++ b/doc/cql3/CQL.textile
@@ -1079,7 +1079,7 @@ The @SELECT@ statements reads one or more columns for one
or more rows in a tabl
h4(#selectSelection). @@
-The @@ determines which columns needs to be queried and
returned in the result-set. It consists of either the comma-separated list of
or the wildcard character (@*@) to select all the columns defined
for the table.
+The @@ determines which columns needs to be queried and
returned in the result-set. It consists of either the comma-separated list of
or the wildcard character (@*@) to select all the columns defined
for the table. Please note that for wildcard @SELECT@ queries the order of
columns returned is not specified and is not guaranteed to be stable between
Cassandra versions.
A @@ is either a column name to retrieve or a @@ of one or
more @@s. The function allowed are the same as for @@ and are
described in the "function section":#functions. In addition to these generic
functions, the @WRITETIME@ (resp. @TTL@) function allows to select the
timestamp of when the column was inserted (resp. the time to live (in seconds)
for the column (or null if the column has no expiration set)).
http://git-wip-us.apache.org/repos/asf/cassandra/blob/236c47e6/src/java/org/apache/cassandra/db/Columns.java
--
diff --git a/src/java/org/apache/cassandra/db/Columns.java
b/src/java/org/apache/cassandra/db/Columns.java
index eb4f761..45ce91e 100644
--- a/src/java/org/apache/cassandra/db/Columns.java
+++ b/src/java/org/apache/cassandra/db/Columns.java
@@ -50,7 +50,16 @@ public class Columns extends
AbstractCollection implements Col
{
public static final Serializer serializer = new Serializer();
public static final Columns NONE = new Columns(BTree.empty(), 0);
-public static final ColumnDefinition FIRST_COMPLEX =
+
+private static final ColumnDefinition FIRST_COMPLEX_STATIC =
+new ColumnDefinition("",
+ "",
+
[3/6] cassandra git commit: Fix static column order for SELECT * wildcard queries
Fix static column order for SELECT * wildcard queries
patch by Aleksey Yeschenko; reviewed by Benedict Elliott Smith for
CASSANDRA-14638
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/236c47e6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/236c47e6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/236c47e6
Branch: refs/heads/trunk
Commit: 236c47e65ce95dcc7c8c75706715b5a2a88fd237
Parents: 3d48cbc
Author: Aleksey Yeshchenko
Authored: Thu Aug 16 12:45:55 2018 +0100
Committer: Aleksey Yeshchenko
Committed: Thu Aug 16 15:43:51 2018 +0100
--
CHANGES.txt | 1 +
NEWS.txt| 10 +++
doc/cql3/CQL.textile| 2 +-
src/java/org/apache/cassandra/db/Columns.java | 22 +--
.../org/apache/cassandra/db/ColumnsTest.java| 69 +++-
5 files changed, 95 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/236c47e6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 7e4d88c..eccece2 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.0.18
+ * Fix static column order for SELECT * wildcard queries (CASSANDRA-14638)
* sstableloader should use discovered broadcast address to connect
intra-cluster (CASSANDRA-14522)
* Fix reading columns with non-UTF names from schema (CASSANDRA-14468)
Merged from 2.2:
http://git-wip-us.apache.org/repos/asf/cassandra/blob/236c47e6/NEWS.txt
--
diff --git a/NEWS.txt b/NEWS.txt
index 234154e..29864bc 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -42,6 +42,16 @@ restore snapshots created with the previous major version
using the
'sstableloader' tool. You can upgrade the file format of your snapshots
using the provided 'sstableupgrade' tool.
+3.0.18
+==
+
+Upgrading
+-
+- The order of static columns in SELECT * has been fixed to match that of
2.0 and 2.1 - they are now sorted
+ alphabetically again, by their name, just like regular columns are. If
you use prepared statements and
+ SELECT * queries, and have both simple and collection static columns in
those tables, and are upgrading from an
+ earlier 3.0 version, then you might be affected by this change. Please
see CASSANDRA-14638 for details.
+
3.0.17
=
http://git-wip-us.apache.org/repos/asf/cassandra/blob/236c47e6/doc/cql3/CQL.textile
--
diff --git a/doc/cql3/CQL.textile b/doc/cql3/CQL.textile
index cc2b9aa..5f35d57 100644
--- a/doc/cql3/CQL.textile
+++ b/doc/cql3/CQL.textile
@@ -1079,7 +1079,7 @@ The @SELECT@ statements reads one or more columns for one
or more rows in a tabl
h4(#selectSelection). @@
-The @@ determines which columns needs to be queried and
returned in the result-set. It consists of either the comma-separated list of
or the wildcard character (@*@) to select all the columns defined
for the table.
+The @@ determines which columns needs to be queried and
returned in the result-set. It consists of either the comma-separated list of
or the wildcard character (@*@) to select all the columns defined
for the table. Please note that for wildcard @SELECT@ queries the order of
columns returned is not specified and is not guaranteed to be stable between
Cassandra versions.
A @@ is either a column name to retrieve or a @@ of one or
more @@s. The function allowed are the same as for @@ and are
described in the "function section":#functions. In addition to these generic
functions, the @WRITETIME@ (resp. @TTL@) function allows to select the
timestamp of when the column was inserted (resp. the time to live (in seconds)
for the column (or null if the column has no expiration set)).
http://git-wip-us.apache.org/repos/asf/cassandra/blob/236c47e6/src/java/org/apache/cassandra/db/Columns.java
--
diff --git a/src/java/org/apache/cassandra/db/Columns.java
b/src/java/org/apache/cassandra/db/Columns.java
index eb4f761..45ce91e 100644
--- a/src/java/org/apache/cassandra/db/Columns.java
+++ b/src/java/org/apache/cassandra/db/Columns.java
@@ -50,7 +50,16 @@ public class Columns extends
AbstractCollection implements Col
{
public static final Serializer serializer = new Serializer();
public static final Columns NONE = new Columns(BTree.empty(), 0);
-public static final ColumnDefinition FIRST_COMPLEX =
+
+private static final ColumnDefinition FIRST_COMPLEX_STATIC =
+new ColumnDefinition("",
+ "",
+
[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/65a46820 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/65a46820 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/65a46820 Branch: refs/heads/cassandra-3.11 Commit: 65a46820bb6d28bc6e359807699e8c6ae562e0c4 Parents: 4f4c390 236c47e Author: Aleksey Yeshchenko Authored: Thu Aug 16 15:44:12 2018 +0100 Committer: Aleksey Yeshchenko Committed: Thu Aug 16 15:44:12 2018 +0100 -- CHANGES.txt | 1 + NEWS.txt| 10 +++ doc/cql3/CQL.textile| 2 +- src/java/org/apache/cassandra/db/Columns.java | 22 +-- .../org/apache/cassandra/db/ColumnsTest.java| 69 +++- 5 files changed, 95 insertions(+), 9 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/65a46820/CHANGES.txt -- diff --cc CHANGES.txt index e2d1681,eccece2..6464667 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,5 -1,5 +1,6 @@@ -3.0.18 +3.11.4 +Merged from 3.0: + * Fix static column order for SELECT * wildcard queries (CASSANDRA-14638) * sstableloader should use discovered broadcast address to connect intra-cluster (CASSANDRA-14522) * Fix reading columns with non-UTF names from schema (CASSANDRA-14468) Merged from 2.2: http://git-wip-us.apache.org/repos/asf/cassandra/blob/65a46820/NEWS.txt -- diff --cc NEWS.txt index 578e426,29864bc..ea53c36 --- a/NEWS.txt +++ b/NEWS.txt @@@ -42,7 -42,17 +42,17 @@@ restore snapshots created with the prev 'sstableloader' tool. You can upgrade the file format of your snapshots using the provided 'sstableupgrade' tool. -3.0.18 ++3.11.4 + == + + Upgrading + - + - The order of static columns in SELECT * has been fixed to match that of 2.0 and 2.1 - they are now sorted + alphabetically again, by their name, just like regular columns are. If you use prepared statements and + SELECT * queries, and have both simple and collection static columns in those tables, and are upgrading from an + earlier 3.0 version, then you might be affected by this change. Please see CASSANDRA-14638 for details. + -3.0.17 +3.11.3 = Upgrading http://git-wip-us.apache.org/repos/asf/cassandra/blob/65a46820/doc/cql3/CQL.textile -- diff --cc doc/cql3/CQL.textile index f32e30c,5f35d57..61b9d63 --- a/doc/cql3/CQL.textile +++ b/doc/cql3/CQL.textile @@@ -1115,9 -1079,9 +1115,9 @@@ The @SELECT@ statements reads one or mo h4(#selectSelection). @@ - The @@ determines which columns needs to be queried and returned in the result-set. It consists of either the comma-separated list of or the wildcard character (@*@) to select all the columns defined for the table. + The @@ determines which columns needs to be queried and returned in the result-set. It consists of either the comma-separated list of or the wildcard character (@*@) to select all the columns defined for the table. Please note that for wildcard @SELECT@ queries the order of columns returned is not specified and is not guaranteed to be stable between Cassandra versions. -A @@ is either a column name to retrieve or a @@ of one or more @@s. The function allowed are the same as for @@ and are described in the "function section":#functions. In addition to these generic functions, the @WRITETIME@ (resp. @TTL@) function allows to select the timestamp of when the column was inserted (resp. the time to live (in seconds) for the column (or null if the column has no expiration set)). +A @@ is either a column name to retrieve or a @@ of one or more @@s. The function allowed are the same as for @@ and are described in the "function section":#functions. In addition to these generic functions, the @WRITETIME@ (resp. @TTL@) function allows to select the timestamp of when the column was inserted (resp. the time to live (in seconds) for the column (or null if the column has no expiration set)) and the "@CAST@":#castFun function can be used to convert one data type to another. Any @@ can be aliased using @AS@ keyword (see examples). Please note that @@ and @@ clause should refer to the columns by their original names and not by their aliases. http://git-wip-us.apache.org/repos/asf/cassandra/blob/65a46820/src/java/org/apache/cassandra/db/Columns.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/65a46820/test/unit/org/apache/cassandra/db/ColumnsTest.java
[jira] [Updated] (CASSANDRA-14649) Index summaries fail when their size gets > 2G and use more space than necessary
[
https://issues.apache.org/jira/browse/CASSANDRA-14649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Branimir Lambov updated CASSANDRA-14649:
Reviewers: (was: Benedict)
> Index summaries fail when their size gets > 2G and use more space than
> necessary
>
>
> Key: CASSANDRA-14649
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14649
> Project: Cassandra
> Issue Type: Bug
>Reporter: Branimir Lambov
>Assignee: Branimir Lambov
>Priority: Major
>
> After building a summary, {{IndexSummaryBuilder}} tries to trim the memory
> writers by calling {{SafeMemoryWriter.setCapacity(capacity())}}. Instead of
> trimming, this ends up allocating at least as much extra space and failing
> the {{Buffer.position()}} call when the size is greater than
> {{Integer.MAX_VALUE}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14649) Index summaries fail when their size gets > 2G and use more space than necessary
[
https://issues.apache.org/jira/browse/CASSANDRA-14649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582619#comment-16582619
]
Branimir Lambov commented on CASSANDRA-14649:
-
Right, I messed it up too... I can't see any problems with {{length}}, though.
Updated patch to address the other comments and also added size checking to the
test.
> Index summaries fail when their size gets > 2G and use more space than
> necessary
>
>
> Key: CASSANDRA-14649
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14649
> Project: Cassandra
> Issue Type: Bug
>Reporter: Branimir Lambov
>Assignee: Branimir Lambov
>Priority: Major
>
> After building a summary, {{IndexSummaryBuilder}} tries to trim the memory
> writers by calling {{SafeMemoryWriter.setCapacity(capacity())}}. Instead of
> trimming, this ends up allocating at least as much extra space and failing
> the {{Buffer.position()}} call when the size is greater than
> {{Integer.MAX_VALUE}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14649) Index summaries fail when their size gets > 2G and use more space than necessary
[
https://issues.apache.org/jira/browse/CASSANDRA-14649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Branimir Lambov updated CASSANDRA-14649:
Reviewer: Benedict (was: Ariel Weisberg)
> Index summaries fail when their size gets > 2G and use more space than
> necessary
>
>
> Key: CASSANDRA-14649
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14649
> Project: Cassandra
> Issue Type: Bug
>Reporter: Branimir Lambov
>Assignee: Branimir Lambov
>Priority: Major
>
> After building a summary, {{IndexSummaryBuilder}} tries to trim the memory
> writers by calling {{SafeMemoryWriter.setCapacity(capacity())}}. Instead of
> trimming, this ends up allocating at least as much extra space and failing
> the {{Buffer.position()}} call when the size is greater than
> {{Integer.MAX_VALUE}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14638) Column result order can change in 'SELECT *' results when upgrading from 2.1 to 3.0 causing response corruption for queries using prepared statements when static c
[
https://issues.apache.org/jira/browse/CASSANDRA-14638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582609#comment-16582609
]
Benedict commented on CASSANDRA-14638:
--
+1
> Column result order can change in 'SELECT *' results when upgrading from 2.1
> to 3.0 causing response corruption for queries using prepared statements when
> static columns are used
> --
>
> Key: CASSANDRA-14638
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14638
> Project: Cassandra
> Issue Type: Bug
> Environment: Single C* node ccm cluster upgraded from C* 2.1.20 to
> 3.0.17
>Reporter: Andy Tolbert
>Assignee: Aleksey Yeschenko
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
>
> When performing an upgrade from C* 2.1.20 to 3.0.17 I observed that the order
> of columns returned from a 'SELECT *' query changes, particularly when static
> columns are involved.
> This may not seem like that much of a problem, however if using Prepared
> Statements, any clients that remain connected during the upgrade may
> encounter issues consuming results from these queries, as data is reordered
> and the client not aware of it. The result definition is sent in the
> original prepared statement response, so if order changes the client has no
> way of knowing (until C* 4.0 via CASSANDRA-10786) without re-preparing, which
> is non-trivial as most client drivers cache prepared statements.
> This could lead to reading the wrong values for columns, which could result
> in some kind of deserialization exception or if the data types of the
> switched columns are compatible, the wrong values. This happens even if the
> client attempts to retrieve a column value by name (i.e. row.getInt("colx")).
> Unfortunately I don't think there is an easy fix for this. If the order was
> changed back to the previous format, you risk issues for users upgrading from
> older 3.0 version. I think it would be nice to add a note in the NEWS file
> in the 3.0 upgrade section that describes this issue, and how to work around
> it (specify all column names of interest explicitly in query).
> Example schema and code to reproduce:
>
> {noformat}
> create keyspace ks with replication = {'class': 'SimpleStrategy',
> 'replication_factor': 1};
> create table ks.tbl (p0 text,
> p1 text,
> m map static,
> t text,
> u text static,
> primary key (p0, p1)
> );
> insert into ks.tbl (p0, p1, m, t, u) values ('p0', 'p1', { 'm0' : 'm1' },
> 't', 'u');{noformat}
>
> When querying with 2.1 you'll observe the following order via cqlsh:
> {noformat}
> p0 | p1 | m| u | t
> ++--+---+---
> p0 | p1 | {'m0': 'm1'} | u | t{noformat}
>
> With 3.0, observe that u and m are transposed:
>
> {noformat}
> p0 | p1 | u | m | t
> ++---+--+---
> p0 | p1 | u | {'m0': 'm1'} | t{noformat}
>
>
> {code:java}
> import com.datastax.driver.core.BoundStatement;
> import com.datastax.driver.core.Cluster;
> import com.datastax.driver.core.ColumnDefinitions;
> import com.datastax.driver.core.PreparedStatement;
> import com.datastax.driver.core.ResultSet;
> import com.datastax.driver.core.Row;
> import com.datastax.driver.core.Session;
> import com.google.common.util.concurrent.Uninterruptibles;
> import java.util.concurrent.TimeUnit;
> public class LiveUpgradeTest {
> public static void main(String args[]) {
> Cluster cluster = Cluster.builder().addContactPoints("127.0.0.1").build();
> try {
> Session session = cluster.connect();
> PreparedStatement p = session.prepare("SELECT * from ks.tbl");
> BoundStatement bs = p.bind();
> // continually query every 30 seconds
> while (true) {
> try {
> ResultSet r = session.execute(bs);
> Row row = r.one();
> int i = 0;
> // iterate over the result metadata in order printing the
> // index, name, type, and length of the first row of data.
> for (ColumnDefinitions.Definition d : r.getColumnDefinitions()) {
> System.out.println(
> i++
> + ": "
> + d.getName()
> + " -> "
> + d.getType()
> + " -> val = "
> + row.getBytesUnsafe(d.getName()).array().length);
> }
> } catch (Throwable t) {
> t.printStackTrace();
> } finally {
> Uninterruptibles.sleepUninterruptibly(30, TimeUnit.SECONDS);
> }
> }
> } finally {
> cluster.close();
> }
> }
> }
> {code}
> To reproduce, set up a cluster, the schema,
[jira] [Updated] (CASSANDRA-14649) Index summaries fail when their size gets > 2G and use more space than necessary
[
https://issues.apache.org/jira/browse/CASSANDRA-14649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benedict updated CASSANDRA-14649:
-
Reviewers: Benedict
> Index summaries fail when their size gets > 2G and use more space than
> necessary
>
>
> Key: CASSANDRA-14649
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14649
> Project: Cassandra
> Issue Type: Bug
>Reporter: Branimir Lambov
>Assignee: Branimir Lambov
>Priority: Major
>
> After building a summary, {{IndexSummaryBuilder}} tries to trim the memory
> writers by calling {{SafeMemoryWriter.setCapacity(capacity())}}. Instead of
> trimming, this ends up allocating at least as much extra space and failing
> the {{Buffer.position()}} call when the size is greater than
> {{Integer.MAX_VALUE}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14649) Index summaries fail when their size gets > 2G and use more space than necessary
[
https://issues.apache.org/jira/browse/CASSANDRA-14649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582586#comment-16582586
]
Benedict commented on CASSANDRA-14649:
--
I think this patch is also broken.
Previously, the logical trim invoked {{setCapacity(length())}}, which I can see
is buggy for a size > 2GiB (but is otherwise consistent). Now, it seems to be
invoking {{setCapacity(capacity())}}, which is surely a no-op?
It seems that there's a bunch of bugs here, and that really we should be:
# fix {{length}} to work for sizes > 2GiB
# implement {{trim}} as {{resizeTo(length())}}
# rename {{reallocate}} to something like {{ensureCapacity}}, to avoid this
kind of misuse mistake in future
> Index summaries fail when their size gets > 2G and use more space than
> necessary
>
>
> Key: CASSANDRA-14649
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14649
> Project: Cassandra
> Issue Type: Bug
>Reporter: Branimir Lambov
>Assignee: Branimir Lambov
>Priority: Major
>
> After building a summary, {{IndexSummaryBuilder}} tries to trim the memory
> writers by calling {{SafeMemoryWriter.setCapacity(capacity())}}. Instead of
> trimming, this ends up allocating at least as much extra space and failing
> the {{Buffer.position()}} call when the size is greater than
> {{Integer.MAX_VALUE}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13304) Add checksumming to the native protocol
[
https://issues.apache.org/jira/browse/CASSANDRA-13304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582561#comment-16582561
]
Tom van der Woerdt commented on CASSANDRA-13304:
I ran a quick SHA1 benchmark to be sure, but it looks like it's able to hash
~1GB/s per thread on my test machine, which should be fast enough for it to go
unnoticeable :)
> Add checksumming to the native protocol
> ---
>
> Key: CASSANDRA-13304
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13304
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Sam Tunnicliffe
>Priority: Blocker
> Labels: client-impacting
> Fix For: 4.x
>
> Attachments: 13304_v1.diff, boxplot-read-throughput.png,
> boxplot-write-throughput.png
>
>
> The native binary transport implementation doesn't include checksums. This
> makes it highly susceptible to silently inserting corrupted data either due
> to hardware issues causing bit flips on the sender/client side, C*/receiver
> side, or network in between.
> Attaching an implementation that makes checksum'ing mandatory (assuming both
> client and server know about a protocol version that supports checksums) --
> and also adds checksumming to clients that request compression.
> The serialized format looks something like this:
> {noformat}
> * 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
> * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Number of Compressed Chunks | Compressed Length (e1)/
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * / Compressed Length cont. (e1) |Uncompressed Length (e1) /
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length cont. (e1)| CRC32 Checksum of Lengths (e1)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Checksum of Lengths cont. (e1)|Compressed Bytes (e1)+//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e1) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (e2)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (e2) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e2) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (en)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (en) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (en) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> {noformat}
> The first pass here adds checksums only to the actual contents of the frame
> body itself (and doesn't actually checksum lengths and headers). While it
> would be great to fully add checksuming across the entire protocol, the
> proposed implementation will ensure we at least catch corrupted data and
> likely protect ourselves pretty well anyways.
> I didn't go to the trouble of implementing a Snappy Checksum'ed Compressor
> implementation as it's been deprecated for a while -- is really slow and
> crappy compared to LZ4 -- and we should do everything in our power to make
> sure no one in the community is still using it. I left it in (for obvious
> backwards compatibility aspects) old for clients that don't know about the
> new protocol.
> The current protocol has a 256MB (max) frame body -- where the serialized
> contents are simply written in to the frame body.
> If the client sends a compression option in the startup, we will install a
> FrameCompressor inline. Unfortunately, we went with a decision to treat the
> frame body separately
[jira] [Commented] (CASSANDRA-13457) Diag. Events: Add base classes
[ https://issues.apache.org/jira/browse/CASSANDRA-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582559#comment-16582559 ] Jason Brown commented on CASSANDRA-13457: - [~spo...@gmail.com] thanks for changing the access to the Gossiper state. That was my biggest concern with this patch. I am fine with the rest of the patch as a whole. If [~michaelsembwever]'s +1 still stands, I think you are good to commit. > Diag. Events: Add base classes > -- > > Key: CASSANDRA-13457 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13457 > Project: Cassandra > Issue Type: Sub-task > Components: Core, Observability >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > > Base ticket for adding classes that will allow you to implement and subscribe > to events. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14650) sstablerepairedset failing looking for tmp-lb sstables part
Maciej Lasyk created CASSANDRA-14650:
Summary: sstablerepairedset failing looking for tmp-lb sstables
part
Key: CASSANDRA-14650
URL: https://issues.apache.org/jira/browse/CASSANDRA-14650
Project: Cassandra
Issue Type: Bug
Components: Tools
Reporter: Maciej Lasyk
While moving back from incremental repairs to full repairs (C* 2.2.12) we're
setting all our sstables as "unrepaired" just like described here:
[https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsSSTableRepairedSet.html]
So basically we're running sstablerepairedset against sstables. First we're
creating sstables list:
{code:java}
find "/var/lib/cassandra/data/keyspace/" -iname "*Data.db*" >
/var/lib/cassandra/sstables.txt{code}
and afterwards:
{code:java}
/usr/bin/sstablerepairedset --really-set --is-unrepaired -f
/var/lib/cassandra/sstables.txt{code}
Unfortunately, operation is unfinished w/error:
{code:java}
root@host $ /usr/bin/sstablerepairedset --really-set --is-unrepaired -f
/var/lib/cassandra/sstables.txt
Exception in thread "main" java.io.FileNotFoundException:
/var/lib/cassandra/data/keyspace/some-cf/tmp-lb-74-big-Statistics.db (No such
file or directory)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.(FileOutputStream.java:213)
at java.io.FileOutputStream.(FileOutputStream.java:101)
at
org.apache.cassandra.io.sstable.metadata.MetadataSerializer.rewriteSSTableMetadata(MetadataSerializer.java:155)
at
org.apache.cassandra.io.sstable.metadata.MetadataSerializer.mutateRepairedAt(MetadataSerializer.java:148)
at
org.apache.cassandra.tools.SSTableRepairedAtSetter.main(SSTableRepairedAtSetter.java:92){code}
Afaik tmp-part of sstable is created for a compaction time. There is no single
"tmp" part of sstables in our whole C* datadir.
Do you know any workaround for having those sstables marked as unrepaired?
Cheers,
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13304) Add checksumming to the native protocol
[
https://issues.apache.org/jira/browse/CASSANDRA-13304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582544#comment-16582544
]
Benedict commented on CASSANDRA-13304:
--
Is that their expectation? Or that the checksum is reliable?
Probably xxhash is sufficiently reliable wrt detecting errors, when combined
with lz4 compression. But I'm not certain. Perhaps this is also faster than
plain SHA1, even with the SHA acceleration extensions.
But using the transport layer for this certainly seems like the best baseline
behaviour to me, in that it almost certainly does what we want, and does it
well. A more performant version would usually come after, as an optimisation,
IMO?
> Add checksumming to the native protocol
> ---
>
> Key: CASSANDRA-13304
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13304
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Sam Tunnicliffe
>Priority: Blocker
> Labels: client-impacting
> Fix For: 4.x
>
> Attachments: 13304_v1.diff, boxplot-read-throughput.png,
> boxplot-write-throughput.png
>
>
> The native binary transport implementation doesn't include checksums. This
> makes it highly susceptible to silently inserting corrupted data either due
> to hardware issues causing bit flips on the sender/client side, C*/receiver
> side, or network in between.
> Attaching an implementation that makes checksum'ing mandatory (assuming both
> client and server know about a protocol version that supports checksums) --
> and also adds checksumming to clients that request compression.
> The serialized format looks something like this:
> {noformat}
> * 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
> * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Number of Compressed Chunks | Compressed Length (e1)/
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * / Compressed Length cont. (e1) |Uncompressed Length (e1) /
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length cont. (e1)| CRC32 Checksum of Lengths (e1)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Checksum of Lengths cont. (e1)|Compressed Bytes (e1)+//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e1) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (e2)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (e2) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e2) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (en)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (en) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (en) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> {noformat}
> The first pass here adds checksums only to the actual contents of the frame
> body itself (and doesn't actually checksum lengths and headers). While it
> would be great to fully add checksuming across the entire protocol, the
> proposed implementation will ensure we at least catch corrupted data and
> likely protect ourselves pretty well anyways.
> I didn't go to the trouble of implementing a Snappy Checksum'ed Compressor
> implementation as it's been deprecated for a while -- is really slow and
> crappy compared to LZ4 -- and we should do everything in our power to make
> sure no one in the community is still using it. I left it in (for obvious
> backwards compatibility aspects) old for clients that don't know about the
[jira] [Commented] (CASSANDRA-14574) Incomplete handling of exceptions when decoding incoming messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582529#comment-16582529
]
Jason Brown commented on CASSANDRA-14574:
-
dtest branch [added
here|https://github.com/jasobrown/cassandra-dtest/tree/14574]. Waiting for
tests to run.
> Incomplete handling of exceptions when decoding incoming messages
> --
>
> Key: CASSANDRA-14574
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14574
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
>Reporter: Aleksey Yeschenko
>Assignee: Jason Brown
>Priority: Major
> Fix For: 4.0
>
>
> {{MessageInHandler.decode()}} occasionally reads the payload incorrectly,
> passing the full message to {{MessageIn.read()}} instead of just the payload
> bytes.
> You can see the stack trace in the logs from this [CI
> run|https://circleci.com/gh/iamaleksey/cassandra/437#tests/containers/38].
> {code}
> Caused by: java.lang.AssertionError: null
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:351)
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:371)
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:335)
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:158)
> at
> org.apache.cassandra.net.async.MessageInHandler.decode(MessageInHandler.java:132)
> {code}
> Reconstructed, truncated stream passed to {{MessageIn.read()}}:
> {{000b000743414c5f42414301002a01e1a5c9b089fd11e8b517436ee124300704005d10fc50ec}}
> You can clearly see parameters in there encoded before the payload:
> {{[43414c5f424143 - CAL_BAC] [01 - ONE_BYTE] [002a - 42, payload size] 01 e1
> a5 c9 b0 89 fd 11 e8 b5 17 43 6e e1 24 30 07 04 00 00 00 1d 10 fc 50 ec}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14648) CircleCI dtest runs should (by default) depend upon successful unit tests
[ https://issues.apache.org/jira/browse/CASSANDRA-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582510#comment-16582510 ] Benedict commented on CASSANDRA-14648: -- Well, that isn't a very good citizen approach IMO. For starters, we shouldn't have flakey tests, and this would encourage us to fix them. It's also trivial to manually kick off the dtests if the failed unit tests are inspected to be OK. But forcing other users to wait for your likely-to-fail dtests because you're pushing a branch for safe keeping isn't very neighbourly, and I think probably covers a lot of CircleCI traffic. This means anyone legitimately waiting for their CI results is bottlenecked for longer. I'd personally prefer we *only* ran dtests on a manual request, as they should only be needed a fraction of the frequency I think they're requested. > CircleCI dtest runs should (by default) depend upon successful unit tests > - > > Key: CASSANDRA-14648 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14648 > Project: Cassandra > Issue Type: Improvement >Reporter: Benedict >Assignee: Benedict >Priority: Major > > Unit tests are very quick to run, and if they fail to pass there’s probably > no value in running dtests - particularly if we are honouring our > expectations of never committing code that breaks either unit or dtests. > When sharing CircleCI resources between multiple branches (or multiple > users), it is wasteful to have two dtest runs kicked off for every incomplete > branch that is pushed to GitHub for safe keeping. So I think a better > default CircleCI config file would only run the dtests after a successful > unit test run, and those who want to modify this behaviour can do so > consciously by editing the config file for themselves. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14648) CircleCI dtest runs should (by default) depend upon successful unit tests
[ https://issues.apache.org/jira/browse/CASSANDRA-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582505#comment-16582505 ] Ariel Weisberg commented on CASSANDRA-14648: In principle I get why you might want to cause people pain when the tests aren’t passing. In practice the first thing I will do is disable this on my branches so a flakey test doesn’t mean I have to start over when I check the results the next day and it isn’t done. This was how it originally worked and I think Michael “fixed” it so they all ran. > CircleCI dtest runs should (by default) depend upon successful unit tests > - > > Key: CASSANDRA-14648 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14648 > Project: Cassandra > Issue Type: Improvement >Reporter: Benedict >Assignee: Benedict >Priority: Major > > Unit tests are very quick to run, and if they fail to pass there’s probably > no value in running dtests - particularly if we are honouring our > expectations of never committing code that breaks either unit or dtests. > When sharing CircleCI resources between multiple branches (or multiple > users), it is wasteful to have two dtest runs kicked off for every incomplete > branch that is pushed to GitHub for safe keeping. So I think a better > default CircleCI config file would only run the dtests after a successful > unit test run, and those who want to modify this behaviour can do so > consciously by editing the config file for themselves. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14649) Index summaries fail when their size gets > 2G and use more space than necessary
[
https://issues.apache.org/jira/browse/CASSANDRA-14649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Branimir Lambov updated CASSANDRA-14649:
Reviewer: Ariel Weisberg
Status: Patch Available (was: Open)
Patch uploaded here: https://github.com/blambov/cassandra/tree/14649
> Index summaries fail when their size gets > 2G and use more space than
> necessary
>
>
> Key: CASSANDRA-14649
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14649
> Project: Cassandra
> Issue Type: Bug
>Reporter: Branimir Lambov
>Assignee: Branimir Lambov
>Priority: Major
>
> After building a summary, {{IndexSummaryBuilder}} tries to trim the memory
> writers by calling {{SafeMemoryWriter.setCapacity(capacity())}}. Instead of
> trimming, this ends up allocating at least as much extra space and failing
> the {{Buffer.position()}} call when the size is greater than
> {{Integer.MAX_VALUE}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13458) Diag. Events: Add unit testing support
[ https://issues.apache.org/jira/browse/CASSANDRA-13458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-13458: --- Status: Open (was: Patch Available) > Diag. Events: Add unit testing support > -- > > Key: CASSANDRA-13458 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13458 > Project: Cassandra > Issue Type: Sub-task > Components: Testing >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > > Diagnostic events will improve unit testing by > * providing test execution control instances based on CompletableFutures (see > [PendingRangeCalculatorServiceTest.java|https://github.com/spodkowinski/cassandra/blob/WIP-13458/test/unit/org/apache/cassandra/gms/PendingRangeCalculatorServiceTest.java]) > > * validate state and behavior by allowing you to inspect generated events > (see > [HintsServiceEventsTest.java|https://github.com/spodkowinski/cassandra/blob/WIP-13458/test/unit/org/apache/cassandra/hints/HintsServiceEventsTest.java]) > See included > [testing.rst|https://github.com/spodkowinski/cassandra/blob/WIP-13458/doc/source/development/testing.rst#diagnostic-events-40] > draft for more details. Let me know if this would be useful for you as a > developer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13304) Add checksumming to the native protocol
[
https://issues.apache.org/jira/browse/CASSANDRA-13304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582498#comment-16582498
]
Ariel Weisberg commented on CASSANDRA-13304:
If someone just wants a checksum their performance expectations are going to be
as currently available checksum implementations are extremely fast when used
correctly. How does a TLS based implementation using a cryptographic hash
perform in comparison?
> Add checksumming to the native protocol
> ---
>
> Key: CASSANDRA-13304
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13304
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Sam Tunnicliffe
>Priority: Blocker
> Labels: client-impacting
> Fix For: 4.x
>
> Attachments: 13304_v1.diff, boxplot-read-throughput.png,
> boxplot-write-throughput.png
>
>
> The native binary transport implementation doesn't include checksums. This
> makes it highly susceptible to silently inserting corrupted data either due
> to hardware issues causing bit flips on the sender/client side, C*/receiver
> side, or network in between.
> Attaching an implementation that makes checksum'ing mandatory (assuming both
> client and server know about a protocol version that supports checksums) --
> and also adds checksumming to clients that request compression.
> The serialized format looks something like this:
> {noformat}
> * 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
> * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Number of Compressed Chunks | Compressed Length (e1)/
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * / Compressed Length cont. (e1) |Uncompressed Length (e1) /
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length cont. (e1)| CRC32 Checksum of Lengths (e1)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Checksum of Lengths cont. (e1)|Compressed Bytes (e1)+//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e1) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (e2)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (e2) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e2) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (en)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (en) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (en) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> {noformat}
> The first pass here adds checksums only to the actual contents of the frame
> body itself (and doesn't actually checksum lengths and headers). While it
> would be great to fully add checksuming across the entire protocol, the
> proposed implementation will ensure we at least catch corrupted data and
> likely protect ourselves pretty well anyways.
> I didn't go to the trouble of implementing a Snappy Checksum'ed Compressor
> implementation as it's been deprecated for a while -- is really slow and
> crappy compared to LZ4 -- and we should do everything in our power to make
> sure no one in the community is still using it. I left it in (for obvious
> backwards compatibility aspects) old for clients that don't know about the
> new protocol.
> The current protocol has a 256MB (max) frame body -- where the serialized
> contents are simply written in to the frame body.
> If the client sends a compression option in the startup, we will install a
> FrameCompressor inline.
[jira] [Assigned] (CASSANDRA-14649) Index summaries fail when their size gets > 2G and use more space than necessary
[
https://issues.apache.org/jira/browse/CASSANDRA-14649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Branimir Lambov reassigned CASSANDRA-14649:
---
Assignee: Branimir Lambov
> Index summaries fail when their size gets > 2G and use more space than
> necessary
>
>
> Key: CASSANDRA-14649
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14649
> Project: Cassandra
> Issue Type: Bug
>Reporter: Branimir Lambov
>Assignee: Branimir Lambov
>Priority: Major
>
> After building a summary, {{IndexSummaryBuilder}} tries to trim the memory
> writers by calling {{SafeMemoryWriter.setCapacity(capacity())}}. Instead of
> trimming, this ends up allocating at least as much extra space and failing
> the {{Buffer.position()}} call when the size is greater than
> {{Integer.MAX_VALUE}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14649) Index summaries fail when their size gets > 2G and use more space than necessary
Branimir Lambov created CASSANDRA-14649:
---
Summary: Index summaries fail when their size gets > 2G and use
more space than necessary
Key: CASSANDRA-14649
URL: https://issues.apache.org/jira/browse/CASSANDRA-14649
Project: Cassandra
Issue Type: Bug
Reporter: Branimir Lambov
After building a summary, {{IndexSummaryBuilder}} tries to trim the memory
writers by calling {{SafeMemoryWriter.setCapacity(capacity())}}. Instead of
trimming, this ends up allocating at least as much extra space and failing the
{{Buffer.position()}} call when the size is greater than {{Integer.MAX_VALUE}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13457) Diag. Events: Add base classes
[
https://issues.apache.org/jira/browse/CASSANDRA-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582478#comment-16582478
]
Stefan Podkowinski commented on CASSANDRA-13457:
Rebased and pushed some changes for CASSANDRA-13457
([a5be8dc|https://github.com/spodkowinski/cassandra/commit/a5be8dc0214dff4974f5d299c9eebc69f04b650b]),
CASSANDRA-14435
([2d3549c|https://github.com/spodkowinski/cassandra/commit/2d3549c53ce11d5d94951f093b5db15a44f71325]),
CASSANDRA-13668
([c26219d|https://github.com/apache/cassandra/commit/c26219d36e611f0c72c1e82fc3d6dd9753571b2b])
(last one rebase only).
* Removed {{enableDiagnostics()}} in {{DiagnosticEventServiceMBean}} to enforce
explicit opt-in via cassandra yaml, as exposed data could contain security
sensitive details. The {{disableDiagnostics()}} method still exists to stop
events without having to restart a node.
* {{SchemaAnnouncementEvent}} will no longer depend on
{{SchemaTransformation.toString()}} output
([a5be8dc#L|https://github.com/spodkowinski/cassandra/commit/a5be8dc0214dff4974f5d299c9eebc69f04b650b#diff-4e80da086bdbd68b1295161b00719a10R66])
** The added {{toString()}} methods have been left as before, but can also be
removed if anyone minds having them
* Added threadName to returned events
([2d3549c#L|https://github.com/spodkowinski/cassandra/commit/2d3549c53ce11d5d94951f093b5db15a44f71325#diff-734577dc8c504d46702efa2262dfaac6R89])
* Moved getSource() to CASSANDRA-13458, so we don't have to discuss it now
Also
* Expose Gossiper state by calling getters instead of inspecting members
directly
([a5be8dc#L|https://github.com/spodkowinski/cassandra/commit/a5be8dc0214dff4974f5d299c9eebc69f04b650b#diff-8be666c70553b1f0017a01458c490f47R903])
There are several other ways to solve this (making a snapshot of the internal
state of a service like Gossiper).
* calling getters from the event class (as in latest version)
* have gossiper return an event or part of the event details (merged in
{{toMap()}})
* pass a builder object to gossip and have gossiper add any internal state to
that
I'd still prefer the first approach due to separations of concerns and to avoid
introducing leaky abstractions between services, such as Gossiper, and actual
event handling and persistence. We'd first have to agree on a more precise
contract for the kind of data to expose in events. The current {{Map}} take is probably already too ambiguous, but I don't have any
better ideas for that, without opening the discussion again on how to persist
and remotely access events in general.
> Diag. Events: Add base classes
> --
>
> Key: CASSANDRA-13457
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13457
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core, Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>
> Base ticket for adding classes that will allow you to implement and subscribe
> to events.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14638) Column result order can change in 'SELECT *' results when upgrading from 2.1 to 3.0 causing response corruption for queries using prepared statements when static c
[
https://issues.apache.org/jira/browse/CASSANDRA-14638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582451#comment-16582451
]
Aleksey Yeschenko commented on CASSANDRA-14638:
---
[~benedict] That's indeed better and less invasive. Force-pushed the new
version, updated CI links in-place in the previous comment.
[~slebresne] Agreed on all points. Updated documentation in the v2 of the patch.
> Column result order can change in 'SELECT *' results when upgrading from 2.1
> to 3.0 causing response corruption for queries using prepared statements when
> static columns are used
> --
>
> Key: CASSANDRA-14638
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14638
> Project: Cassandra
> Issue Type: Bug
> Environment: Single C* node ccm cluster upgraded from C* 2.1.20 to
> 3.0.17
>Reporter: Andy Tolbert
>Assignee: Aleksey Yeschenko
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
>
> When performing an upgrade from C* 2.1.20 to 3.0.17 I observed that the order
> of columns returned from a 'SELECT *' query changes, particularly when static
> columns are involved.
> This may not seem like that much of a problem, however if using Prepared
> Statements, any clients that remain connected during the upgrade may
> encounter issues consuming results from these queries, as data is reordered
> and the client not aware of it. The result definition is sent in the
> original prepared statement response, so if order changes the client has no
> way of knowing (until C* 4.0 via CASSANDRA-10786) without re-preparing, which
> is non-trivial as most client drivers cache prepared statements.
> This could lead to reading the wrong values for columns, which could result
> in some kind of deserialization exception or if the data types of the
> switched columns are compatible, the wrong values. This happens even if the
> client attempts to retrieve a column value by name (i.e. row.getInt("colx")).
> Unfortunately I don't think there is an easy fix for this. If the order was
> changed back to the previous format, you risk issues for users upgrading from
> older 3.0 version. I think it would be nice to add a note in the NEWS file
> in the 3.0 upgrade section that describes this issue, and how to work around
> it (specify all column names of interest explicitly in query).
> Example schema and code to reproduce:
>
> {noformat}
> create keyspace ks with replication = {'class': 'SimpleStrategy',
> 'replication_factor': 1};
> create table ks.tbl (p0 text,
> p1 text,
> m map static,
> t text,
> u text static,
> primary key (p0, p1)
> );
> insert into ks.tbl (p0, p1, m, t, u) values ('p0', 'p1', { 'm0' : 'm1' },
> 't', 'u');{noformat}
>
> When querying with 2.1 you'll observe the following order via cqlsh:
> {noformat}
> p0 | p1 | m| u | t
> ++--+---+---
> p0 | p1 | {'m0': 'm1'} | u | t{noformat}
>
> With 3.0, observe that u and m are transposed:
>
> {noformat}
> p0 | p1 | u | m | t
> ++---+--+---
> p0 | p1 | u | {'m0': 'm1'} | t{noformat}
>
>
> {code:java}
> import com.datastax.driver.core.BoundStatement;
> import com.datastax.driver.core.Cluster;
> import com.datastax.driver.core.ColumnDefinitions;
> import com.datastax.driver.core.PreparedStatement;
> import com.datastax.driver.core.ResultSet;
> import com.datastax.driver.core.Row;
> import com.datastax.driver.core.Session;
> import com.google.common.util.concurrent.Uninterruptibles;
> import java.util.concurrent.TimeUnit;
> public class LiveUpgradeTest {
> public static void main(String args[]) {
> Cluster cluster = Cluster.builder().addContactPoints("127.0.0.1").build();
> try {
> Session session = cluster.connect();
> PreparedStatement p = session.prepare("SELECT * from ks.tbl");
> BoundStatement bs = p.bind();
> // continually query every 30 seconds
> while (true) {
> try {
> ResultSet r = session.execute(bs);
> Row row = r.one();
> int i = 0;
> // iterate over the result metadata in order printing the
> // index, name, type, and length of the first row of data.
> for (ColumnDefinitions.Definition d : r.getColumnDefinitions()) {
> System.out.println(
> i++
> + ": "
> + d.getName()
> + " -> "
> + d.getType()
> + " -> val = "
> + row.getBytesUnsafe(d.getName()).array().length);
> }
> } catch (Throwable t) {
> t.printStackTrace();
>
[jira] [Commented] (CASSANDRA-13304) Add checksumming to the native protocol
[
https://issues.apache.org/jira/browse/CASSANDRA-13304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582450#comment-16582450
]
Benedict commented on CASSANDRA-13304:
--
Thanks [~tvdw]!
I will say that I'm personally +1 this approach. It may be that we need to
create a hidden default keystore, and if not specified in cassandra.yaml use
this. But otherwise, I cannot see a downside to this simpler, more robust
approach?
AFAICT this is fully cross platform, too (I had wondered if these cipher suites
might be OpenSSL specific, but they don't seem to be)
> Add checksumming to the native protocol
> ---
>
> Key: CASSANDRA-13304
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13304
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Sam Tunnicliffe
>Priority: Blocker
> Labels: client-impacting
> Fix For: 4.x
>
> Attachments: 13304_v1.diff, boxplot-read-throughput.png,
> boxplot-write-throughput.png
>
>
> The native binary transport implementation doesn't include checksums. This
> makes it highly susceptible to silently inserting corrupted data either due
> to hardware issues causing bit flips on the sender/client side, C*/receiver
> side, or network in between.
> Attaching an implementation that makes checksum'ing mandatory (assuming both
> client and server know about a protocol version that supports checksums) --
> and also adds checksumming to clients that request compression.
> The serialized format looks something like this:
> {noformat}
> * 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
> * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Number of Compressed Chunks | Compressed Length (e1)/
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * / Compressed Length cont. (e1) |Uncompressed Length (e1) /
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length cont. (e1)| CRC32 Checksum of Lengths (e1)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Checksum of Lengths cont. (e1)|Compressed Bytes (e1)+//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e1) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (e2)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (e2) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e2) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (en)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (en) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (en) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> {noformat}
> The first pass here adds checksums only to the actual contents of the frame
> body itself (and doesn't actually checksum lengths and headers). While it
> would be great to fully add checksuming across the entire protocol, the
> proposed implementation will ensure we at least catch corrupted data and
> likely protect ourselves pretty well anyways.
> I didn't go to the trouble of implementing a Snappy Checksum'ed Compressor
> implementation as it's been deprecated for a while -- is really slow and
> crappy compared to LZ4 -- and we should do everything in our power to make
> sure no one in the community is still using it. I left it in (for obvious
> backwards compatibility aspects) old for clients that don't know about the
> new protocol.
> The current protocol has a 256MB (max) frame body -- where the serialized
> contents are simply written in
[jira] [Comment Edited] (CASSANDRA-14638) Column result order can change in 'SELECT *' results when upgrading from 2.1 to 3.0 causing response corruption for queries using prepared statements when sta
[
https://issues.apache.org/jira/browse/CASSANDRA-14638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581500#comment-16581500
]
Aleksey Yeschenko edited comment on CASSANDRA-14638 at 8/16/18 12:36 PM:
-
Branches with the fix for
[3.0|https://github.com/iamaleksey/cassandra/commits/14638-3.0],
[3.11|https://github.com/iamaleksey/cassandra/commits/14638-3.11], and
[4.0|https://github.com/iamaleksey/cassandra/commits/14638-4.0]. CI,
respectively:
[3.0|https://circleci.com/workflow-run/6be0f2b2-292c-4c9e-81fe-a1b90a86421e],
[3.11|https://circleci.com/workflow-run/a771039b-aa8f-4eea-a268-822c759a1c8e],
and
[4.0|https://circleci.com/workflow-run/ce2620d7-2317-43ef-8f96-90d4f53ee175].
was (Author: iamaleksey):
Branches with the fix for
[3.0|https://github.com/iamaleksey/cassandra/commits/14638-3.0],
[3.11|https://github.com/iamaleksey/cassandra/commits/14638-3.11], and
[4.0|https://github.com/iamaleksey/cassandra/commits/14638-4.0]. CI,
respectively:
[3.0|https://circleci.com/workflow-run/86ae48d8-ef0c-45ee-a978-7afd99b7b1ad],
[3.11|https://circleci.com/workflow-run/a048437a-beaf-4ba8-9482-635ed3412737],
and
[4.0|https://circleci.com/workflow-run/9eb91625-60ff-46e6-9a8d-289b48314f77].
> Column result order can change in 'SELECT *' results when upgrading from 2.1
> to 3.0 causing response corruption for queries using prepared statements when
> static columns are used
> --
>
> Key: CASSANDRA-14638
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14638
> Project: Cassandra
> Issue Type: Bug
> Environment: Single C* node ccm cluster upgraded from C* 2.1.20 to
> 3.0.17
>Reporter: Andy Tolbert
>Assignee: Aleksey Yeschenko
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
>
> When performing an upgrade from C* 2.1.20 to 3.0.17 I observed that the order
> of columns returned from a 'SELECT *' query changes, particularly when static
> columns are involved.
> This may not seem like that much of a problem, however if using Prepared
> Statements, any clients that remain connected during the upgrade may
> encounter issues consuming results from these queries, as data is reordered
> and the client not aware of it. The result definition is sent in the
> original prepared statement response, so if order changes the client has no
> way of knowing (until C* 4.0 via CASSANDRA-10786) without re-preparing, which
> is non-trivial as most client drivers cache prepared statements.
> This could lead to reading the wrong values for columns, which could result
> in some kind of deserialization exception or if the data types of the
> switched columns are compatible, the wrong values. This happens even if the
> client attempts to retrieve a column value by name (i.e. row.getInt("colx")).
> Unfortunately I don't think there is an easy fix for this. If the order was
> changed back to the previous format, you risk issues for users upgrading from
> older 3.0 version. I think it would be nice to add a note in the NEWS file
> in the 3.0 upgrade section that describes this issue, and how to work around
> it (specify all column names of interest explicitly in query).
> Example schema and code to reproduce:
>
> {noformat}
> create keyspace ks with replication = {'class': 'SimpleStrategy',
> 'replication_factor': 1};
> create table ks.tbl (p0 text,
> p1 text,
> m map static,
> t text,
> u text static,
> primary key (p0, p1)
> );
> insert into ks.tbl (p0, p1, m, t, u) values ('p0', 'p1', { 'm0' : 'm1' },
> 't', 'u');{noformat}
>
> When querying with 2.1 you'll observe the following order via cqlsh:
> {noformat}
> p0 | p1 | m| u | t
> ++--+---+---
> p0 | p1 | {'m0': 'm1'} | u | t{noformat}
>
> With 3.0, observe that u and m are transposed:
>
> {noformat}
> p0 | p1 | u | m | t
> ++---+--+---
> p0 | p1 | u | {'m0': 'm1'} | t{noformat}
>
>
> {code:java}
> import com.datastax.driver.core.BoundStatement;
> import com.datastax.driver.core.Cluster;
> import com.datastax.driver.core.ColumnDefinitions;
> import com.datastax.driver.core.PreparedStatement;
> import com.datastax.driver.core.ResultSet;
> import com.datastax.driver.core.Row;
> import com.datastax.driver.core.Session;
> import com.google.common.util.concurrent.Uninterruptibles;
> import java.util.concurrent.TimeUnit;
> public class LiveUpgradeTest {
> public static void main(String args[]) {
> Cluster cluster = Cluster.builder().addContactPoints("127.0.0.1").build();
> try {
> Session session = cluster.connect();
> PreparedStatement p =
[jira] [Commented] (CASSANDRA-13304) Add checksumming to the native protocol
[
https://issues.apache.org/jira/browse/CASSANDRA-13304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582449#comment-16582449
]
Tom van der Woerdt commented on CASSANDRA-13304:
It looks like this can be configured entirely from the cassandra.yaml file, so
this could be done as a change to the default configuration. Specifically, this
is what I did to my cassandra.yaml to make this work:
{code:java}
client_encryption_options:
enabled: true
optional: true
keystore: conf/.keystore
keystore_password: cassandra
cipher_suites: [TLS_ECDH_anon_WITH_NULL_SHA]{code}
I had to then create an empty keystore, or Cassandra crashes, but we don't
actually need it.
This allowed me to connect:
{code:java}
$ openssl s_client -connect localhost:9042 -cipher NULL
CONNECTED(0005)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 290 bytes and written 390 bytes
---
New, TLSv1/SSLv3, Cipher is AECDH-NULL-SHA
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : AECDH-NULL-SHA
Session-ID: 5B756CA79320B7AF5A4DB9FAE00CDAA3F64AFAACE14C57A51F732CB9BCAC0807
Session-ID-ctx:
Master-Key:
652BE33C8F4579236966B50554F52A4C8C53BECAC4420B26BB7D22F33DFA6CE810A29E9BEA1FB8E3C9C0D22782D82A33
Start Time: 1534422183
Timeout : 300 (sec)
Verify return code: 0 (ok)
---{code}
I have to explicitly tell openssl that it's acceptable to not encrypt, and then
we have a connection. The negotiated cipher is AECDH-NULL-SHA, effectively
giving us a TLSv1.2 connection using no encryption or authentication, but with
integrity protected by SHA1.
Cassandra already has support for running both TLS and non-TLS connections over
the same port, so port 9042 is still usable for unprotected connections, making
this change fully backwards compatible.
Hope that helps!
> Add checksumming to the native protocol
> ---
>
> Key: CASSANDRA-13304
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13304
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Sam Tunnicliffe
>Priority: Blocker
> Labels: client-impacting
> Fix For: 4.x
>
> Attachments: 13304_v1.diff, boxplot-read-throughput.png,
> boxplot-write-throughput.png
>
>
> The native binary transport implementation doesn't include checksums. This
> makes it highly susceptible to silently inserting corrupted data either due
> to hardware issues causing bit flips on the sender/client side, C*/receiver
> side, or network in between.
> Attaching an implementation that makes checksum'ing mandatory (assuming both
> client and server know about a protocol version that supports checksums) --
> and also adds checksumming to clients that request compression.
> The serialized format looks something like this:
> {noformat}
> * 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
> * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Number of Compressed Chunks | Compressed Length (e1)/
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * / Compressed Length cont. (e1) |Uncompressed Length (e1) /
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length cont. (e1)| CRC32 Checksum of Lengths (e1)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Checksum of Lengths cont. (e1)|Compressed Bytes (e1)+//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e1) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (e2)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (e2) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e2) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (en)|
[jira] [Assigned] (CASSANDRA-14648) CircleCI dtest runs should (by default) depend upon successful unit tests
[ https://issues.apache.org/jira/browse/CASSANDRA-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict reassigned CASSANDRA-14648: Assignee: Benedict > CircleCI dtest runs should (by default) depend upon successful unit tests > - > > Key: CASSANDRA-14648 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14648 > Project: Cassandra > Issue Type: Improvement >Reporter: Benedict >Assignee: Benedict >Priority: Major > > Unit tests are very quick to run, and if they fail to pass there’s probably > no value in running dtests - particularly if we are honouring our > expectations of never committing code that breaks either unit or dtests. > When sharing CircleCI resources between multiple branches (or multiple > users), it is wasteful to have two dtest runs kicked off for every incomplete > branch that is pushed to GitHub for safe keeping. So I think a better > default CircleCI config file would only run the dtests after a successful > unit test run, and those who want to modify this behaviour can do so > consciously by editing the config file for themselves. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14648) CircleCI dtest runs should (by default) depend upon successful unit tests
Benedict created CASSANDRA-14648: Summary: CircleCI dtest runs should (by default) depend upon successful unit tests Key: CASSANDRA-14648 URL: https://issues.apache.org/jira/browse/CASSANDRA-14648 Project: Cassandra Issue Type: Improvement Reporter: Benedict Unit tests are very quick to run, and if they fail to pass there’s probably no value in running dtests - particularly if we are honouring our expectations of never committing code that breaks either unit or dtests. When sharing CircleCI resources between multiple branches (or multiple users), it is wasteful to have two dtest runs kicked off for every incomplete branch that is pushed to GitHub for safe keeping. So I think a better default CircleCI config file would only run the dtests after a successful unit test run, and those who want to modify this behaviour can do so consciously by editing the config file for themselves. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14645) Clean up some dead code post CASSANDRA-13910
[ https://issues.apache.org/jira/browse/CASSANDRA-14645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-14645: -- Status: Patch Available (was: Open) > Clean up some dead code post CASSANDRA-13910 > > > Key: CASSANDRA-14645 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14645 > Project: Cassandra > Issue Type: Improvement >Reporter: Aleksey Yeschenko >Assignee: Aleksey Yeschenko >Priority: Trivial > Fix For: 4.0.x > > > Wasn't quite surgical *enough* and missed one unused field and one dead > branch of code. The former mentioned to me by [~ifesdjeen], the latter by > [~benedict]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14645) Clean up some dead code post CASSANDRA-13910
[ https://issues.apache.org/jira/browse/CASSANDRA-14645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-14645: -- Resolution: Fixed Fix Version/s: (was: 4.0.x) 4.0 Status: Resolved (was: Patch Available) > Clean up some dead code post CASSANDRA-13910 > > > Key: CASSANDRA-14645 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14645 > Project: Cassandra > Issue Type: Improvement >Reporter: Aleksey Yeschenko >Assignee: Aleksey Yeschenko >Priority: Trivial > Fix For: 4.0 > > > Wasn't quite surgical *enough* and missed one unused field and one dead > branch of code. The former mentioned to me by [~ifesdjeen], the latter by > [~benedict]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14645) Clean up some dead code post CASSANDRA-13910
[ https://issues.apache.org/jira/browse/CASSANDRA-14645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582376#comment-16582376 ] Aleksey Yeschenko commented on CASSANDRA-14645: --- Committed as [07b0aca6d3b423f9af405b2d64cd285c597023c4|https://github.com/apache/cassandra/commit/07b0aca6d3b423f9af405b2d64cd285c597023c4] to trunk, without a CHANGES.txt entry. Thanks. > Clean up some dead code post CASSANDRA-13910 > > > Key: CASSANDRA-14645 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14645 > Project: Cassandra > Issue Type: Improvement >Reporter: Aleksey Yeschenko >Assignee: Aleksey Yeschenko >Priority: Trivial > Fix For: 4.0 > > > Wasn't quite surgical *enough* and missed one unused field and one dead > branch of code. The former mentioned to me by [~ifesdjeen], the latter by > [~benedict]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
cassandra git commit: Remove post-13910 dead code
Repository: cassandra
Updated Branches:
refs/heads/trunk 7925f91dc -> 07b0aca6d
Remove post-13910 dead code
patch by Aleksey Yeschenko; reviewed by Benedict Elliott Smith for
CASSANDRA-14645
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/07b0aca6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/07b0aca6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/07b0aca6
Branch: refs/heads/trunk
Commit: 07b0aca6d3b423f9af405b2d64cd285c597023c4
Parents: 7925f91
Author: Aleksey Yeshchenko
Authored: Thu Aug 16 01:44:08 2018 +0100
Committer: Aleksey Yeshchenko
Committed: Thu Aug 16 12:28:13 2018 +0100
--
.../apache/cassandra/service/StorageProxy.java | 2 +-
.../service/reads/AbstractReadExecutor.java | 47 ++--
.../cassandra/service/reads/ReadCallback.java | 11 ++---
.../reads/ShortReadPartitionsProtection.java| 2 +-
.../reads/repair/BlockingReadRepair.java| 2 +-
5 files changed, 30 insertions(+), 34 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/07b0aca6/src/java/org/apache/cassandra/service/StorageProxy.java
--
diff --git a/src/java/org/apache/cassandra/service/StorageProxy.java
b/src/java/org/apache/cassandra/service/StorageProxy.java
index 7e9b0f9..81e6dae 100644
--- a/src/java/org/apache/cassandra/service/StorageProxy.java
+++ b/src/java/org/apache/cassandra/service/StorageProxy.java
@@ -2171,7 +2171,7 @@ public class StorageProxy implements StorageProxyMBean
int blockFor = consistency.blockFor(keyspace);
int minResponses = Math.min(toQuery.filteredEndpoints.size(),
blockFor);
List minimalEndpoints =
toQuery.filteredEndpoints.subList(0, minResponses);
-ReadCallback handler = new ReadCallback(resolver, consistency,
rangeCommand, minimalEndpoints, queryStartNanoTime, readRepair);
+ReadCallback handler = new ReadCallback(resolver, consistency,
rangeCommand, minimalEndpoints, queryStartNanoTime);
handler.assureSufficientLiveNodes();
http://git-wip-us.apache.org/repos/asf/cassandra/blob/07b0aca6/src/java/org/apache/cassandra/service/reads/AbstractReadExecutor.java
--
diff --git
a/src/java/org/apache/cassandra/service/reads/AbstractReadExecutor.java
b/src/java/org/apache/cassandra/service/reads/AbstractReadExecutor.java
index 6c8a45a..e531e0c 100644
--- a/src/java/org/apache/cassandra/service/reads/AbstractReadExecutor.java
+++ b/src/java/org/apache/cassandra/service/reads/AbstractReadExecutor.java
@@ -76,7 +76,7 @@ public abstract class AbstractReadExecutor
this.targetReplicas = targetReplicas;
this.readRepair = ReadRepair.create(command, targetReplicas,
queryStartNanoTime, consistency);
this.digestResolver = new DigestResolver(keyspace, command,
consistency, readRepair, targetReplicas.size());
-this.handler = new ReadCallback(digestResolver, consistency, command,
targetReplicas, queryStartNanoTime, readRepair);
+this.handler = new ReadCallback(digestResolver, consistency, command,
targetReplicas, queryStartNanoTime);
this.cfs = cfs;
this.traceState = Tracing.instance.get();
this.queryStartNanoTime = queryStartNanoTime;
@@ -165,11 +165,12 @@ public abstract class AbstractReadExecutor
public static AbstractReadExecutor
getReadExecutor(SinglePartitionReadCommand command, ConsistencyLevel
consistencyLevel, long queryStartNanoTime) throws UnavailableException
{
Keyspace keyspace = Keyspace.open(command.metadata().keyspace);
-List allReplicas =
StorageProxy.getLiveSortedEndpoints(keyspace, command.partitionKey());
-List targetReplicas =
consistencyLevel.filterForQuery(keyspace, allReplicas);
+
+List allLiveReplicas =
StorageProxy.getLiveSortedEndpoints(keyspace, command.partitionKey());
+List selectedReplicas =
consistencyLevel.filterForQuery(keyspace, allLiveReplicas);
// Throw UAE early if we don't have enough replicas.
-consistencyLevel.assureSufficientLiveNodes(keyspace, targetReplicas);
+consistencyLevel.assureSufficientLiveNodes(keyspace, selectedReplicas);
ColumnFamilyStore cfs =
keyspace.getColumnFamilyStore(command.metadata().id);
SpeculativeRetryPolicy retry = cfs.metadata().params.speculativeRetry;
@@ -177,27 +178,21 @@ public abstract class AbstractReadExecutor
// Speculative retry is disabled *OR*
// 11980: Disable speculative retry if using EACH_QUORUM in order to
prevent miscounting DC responses
if (retry.equals(NeverSpeculativeRetryPolicy.INSTANCE) ||
consistencyLevel ==
[jira] [Comment Edited] (CASSANDRA-14645) Clean up some dead code post CASSANDRA-13910
[ https://issues.apache.org/jira/browse/CASSANDRA-14645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581798#comment-16581798 ] Aleksey Yeschenko edited comment on CASSANDRA-14645 at 8/16/18 10:49 AM: - Code [here|https://github.com/iamaleksey/cassandra/commits/14645-4.0], CI, just in case, [here|https://circleci.com/workflow-run/d0acf81b-bab2-4575-98e3-ba9716b1dcb2] was (Author: iamaleksey): Code [here|https://github.com/iamaleksey/cassandra/commits/14645-4.0], CI, just in case, [here|https://circleci.com/workflow-run/24504926-80a3-4c79-9c80-95fe54a5de53] > Clean up some dead code post CASSANDRA-13910 > > > Key: CASSANDRA-14645 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14645 > Project: Cassandra > Issue Type: Improvement >Reporter: Aleksey Yeschenko >Assignee: Aleksey Yeschenko >Priority: Trivial > Fix For: 4.0.x > > > Wasn't quite surgical *enough* and missed one unused field and one dead > branch of code. The former mentioned to me by [~ifesdjeen], the latter by > [~benedict]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14647) Reading cardinality from Statistics.db failed
Vitali Djatsuk created CASSANDRA-14647: -- Summary: Reading cardinality from Statistics.db failed Key: CASSANDRA-14647 URL: https://issues.apache.org/jira/browse/CASSANDRA-14647 Project: Cassandra Issue Type: Bug Components: Compaction Environment: Clients are doing only writes with Local One, cluster consist of 3 regions with RF3. Storage is configured wth jbod/XFS on 10 x 1Tb disks IOPS limit for each disk 500 (total 5000 iops) Bandwith for each disk 60mb/s (600 total) OS is Debian linux. Reporter: Vitali Djatsuk Fix For: 3.0.x Attachments: cassandra_compaction_pending_tasks_7days.png There is some issue with sstable metadata which is visible in system.log, the messages says: WARN [Thread-6] 2018-07-25 07:12:47,928 SSTableReader.java:249 - Reading cardinality from Statistics.db failed for /opt/data/disk5/data/keyspace/table/mc-big-Data.db. Although there is no such file. The message has appeared after i've changed the compaction strategy from SizeTiered to Leveled. Compaction strategy has been changed region by region (total 3 regions) and it has coincided with the double client write traffic increase. I have tried to run nodetool scrub to rebuilt the sstable, but that does not fix the issue. So very hard to define the steps to reproduce, probably it will be: 1) run stress tool with write traffic 2) under load change compaction strategy from SireTiered to Leveled for the bunch of hosts 3) add more write traffic Reading the code it is said that if this metadata is broken, then "estimating the keys will be done using index summary". [https://github.com/apache/cassandra/blob/cassandra-3.0.17/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L247] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14638) Column result order can change in 'SELECT *' results when upgrading from 2.1 to 3.0 causing response corruption for queries using prepared statements when static c
[
https://issues.apache.org/jira/browse/CASSANDRA-14638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582322#comment-16582322
]
Benedict commented on CASSANDRA-14638:
--
The patch looks good to me, so I'd be happy to commit as-is.
There is an alternative approach that might be simpler, or at least fewer
edits; namely, to modify findFirstComplexId to auto-detect isStatic, and avoid
the caller worrying about the distinction. Since we already get the end
ColumnDefinition to avoid a binary search in the case there are no complex
cells, we can also extract its Kind for (almost) free.
There is an argument to be made either way, since we implicitly require that
you never mix regular and static columns in one of these collections, but
presently never actually impose it either semantically or via assertion.
> Column result order can change in 'SELECT *' results when upgrading from 2.1
> to 3.0 causing response corruption for queries using prepared statements when
> static columns are used
> --
>
> Key: CASSANDRA-14638
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14638
> Project: Cassandra
> Issue Type: Bug
> Environment: Single C* node ccm cluster upgraded from C* 2.1.20 to
> 3.0.17
>Reporter: Andy Tolbert
>Assignee: Aleksey Yeschenko
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
>
> When performing an upgrade from C* 2.1.20 to 3.0.17 I observed that the order
> of columns returned from a 'SELECT *' query changes, particularly when static
> columns are involved.
> This may not seem like that much of a problem, however if using Prepared
> Statements, any clients that remain connected during the upgrade may
> encounter issues consuming results from these queries, as data is reordered
> and the client not aware of it. The result definition is sent in the
> original prepared statement response, so if order changes the client has no
> way of knowing (until C* 4.0 via CASSANDRA-10786) without re-preparing, which
> is non-trivial as most client drivers cache prepared statements.
> This could lead to reading the wrong values for columns, which could result
> in some kind of deserialization exception or if the data types of the
> switched columns are compatible, the wrong values. This happens even if the
> client attempts to retrieve a column value by name (i.e. row.getInt("colx")).
> Unfortunately I don't think there is an easy fix for this. If the order was
> changed back to the previous format, you risk issues for users upgrading from
> older 3.0 version. I think it would be nice to add a note in the NEWS file
> in the 3.0 upgrade section that describes this issue, and how to work around
> it (specify all column names of interest explicitly in query).
> Example schema and code to reproduce:
>
> {noformat}
> create keyspace ks with replication = {'class': 'SimpleStrategy',
> 'replication_factor': 1};
> create table ks.tbl (p0 text,
> p1 text,
> m map static,
> t text,
> u text static,
> primary key (p0, p1)
> );
> insert into ks.tbl (p0, p1, m, t, u) values ('p0', 'p1', { 'm0' : 'm1' },
> 't', 'u');{noformat}
>
> When querying with 2.1 you'll observe the following order via cqlsh:
> {noformat}
> p0 | p1 | m| u | t
> ++--+---+---
> p0 | p1 | {'m0': 'm1'} | u | t{noformat}
>
> With 3.0, observe that u and m are transposed:
>
> {noformat}
> p0 | p1 | u | m | t
> ++---+--+---
> p0 | p1 | u | {'m0': 'm1'} | t{noformat}
>
>
> {code:java}
> import com.datastax.driver.core.BoundStatement;
> import com.datastax.driver.core.Cluster;
> import com.datastax.driver.core.ColumnDefinitions;
> import com.datastax.driver.core.PreparedStatement;
> import com.datastax.driver.core.ResultSet;
> import com.datastax.driver.core.Row;
> import com.datastax.driver.core.Session;
> import com.google.common.util.concurrent.Uninterruptibles;
> import java.util.concurrent.TimeUnit;
> public class LiveUpgradeTest {
> public static void main(String args[]) {
> Cluster cluster = Cluster.builder().addContactPoints("127.0.0.1").build();
> try {
> Session session = cluster.connect();
> PreparedStatement p = session.prepare("SELECT * from ks.tbl");
> BoundStatement bs = p.bind();
> // continually query every 30 seconds
> while (true) {
> try {
> ResultSet r = session.execute(bs);
> Row row = r.one();
> int i = 0;
> // iterate over the result metadata in order printing the
> // index, name, type, and length of the first row of data.
> for
[jira] [Comment Edited] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582286#comment-16582286
]
Benedict edited comment on CASSANDRA-14568 at 8/16/18 9:48 AM:
---
OK. So, [here|https://github.com/belliottsmith/cassandra/commits/14568-2] is
may second attempt at fixing this.
In the process of adding improved assertion logic, I realised we might have
another bug around dropped static collection column, that could have resulted
in decoding a static collection deletion as a whole-static-row deletion (with
unknown semantics, since I vaguely recall that our correctness in some places
depends on there being no such deletions).
In essence: if on looking up the collectionNameBytes, we found no
collectionName (due, for instance, to it having been dropped), we would be left
with a only a complete static row bound to construct.
Perhaps I should split this fix into a separate ticket, for a separate
CHANGES.txt mention?
We clearly need to introduce some upgrade dtests to cover these cases as well
was (Author: benedict):
OK. So, [here|https://github.com/belliottsmith/cassandra/commits/14568-2] is
may second attempt at fixing this.
In the process of adding improved assertion logic, I realised we might have
another bug around dropped static collection column, that could have resulted
in decoding a static collection deletion as a whole-static-row deletion (with
unknown semantics, since I vaguely recall that our correctness in some places
depends on there being no such deletions).
In essence: if on looking up the collectionNameBytes, we found no
collectionName (due, for instance, to it having been dropped), we would be left
with a only a complete static row bound to construct.
We clearly need to introduce some upgrade dtests to cover these cases as well
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.0.17, 3.11.3
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582286#comment-16582286
]
Benedict commented on CASSANDRA-14568:
--
OK. So, [here|https://github.com/belliottsmith/cassandra/commits/14568-2] is
may second attempt at fixing this.
In the process of adding improved assertion logic, I realised we might have
another bug around dropped static collection column, that could have resulted
in decoding a static collection deletion as a whole-static-row deletion (with
unknown semantics, since I vaguely recall that our correctness in some places
depends on there being no such deletions).
In essence: if on looking up the collectionNameBytes, we found no
collectionName (due, for instance, to it having been dropped), we would be left
with a only a complete static row bound to construct.
We clearly need to introduce some upgrade dtests to cover these cases as well
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.0.17, 3.11.3
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14645) Clean up some dead code post CASSANDRA-13910
[ https://issues.apache.org/jira/browse/CASSANDRA-14645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-14645: - Reviewer: Benedict > Clean up some dead code post CASSANDRA-13910 > > > Key: CASSANDRA-14645 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14645 > Project: Cassandra > Issue Type: Improvement >Reporter: Aleksey Yeschenko >Assignee: Aleksey Yeschenko >Priority: Trivial > Fix For: 4.0.x > > > Wasn't quite surgical *enough* and missed one unused field and one dead > branch of code. The former mentioned to me by [~ifesdjeen], the latter by > [~benedict]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14645) Clean up some dead code post CASSANDRA-13910
[ https://issues.apache.org/jira/browse/CASSANDRA-14645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582259#comment-16582259 ] Benedict commented on CASSANDRA-14645: -- +1 > Clean up some dead code post CASSANDRA-13910 > > > Key: CASSANDRA-14645 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14645 > Project: Cassandra > Issue Type: Improvement >Reporter: Aleksey Yeschenko >Assignee: Aleksey Yeschenko >Priority: Trivial > Fix For: 4.0.x > > > Wasn't quite surgical *enough* and missed one unused field and one dead > branch of code. The former mentioned to me by [~ifesdjeen], the latter by > [~benedict]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14621) Refactor CompactionStrategyManager
[
https://issues.apache.org/jira/browse/CASSANDRA-14621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582254#comment-16582254
]
Marcus Eriksson commented on CASSANDRA-14621:
-
this lgtm in general, just a few comments
* some of the new code is non-trivial and should probably have a few tests, for
example {{AbstractStrategyHolder.GroupedSSTableContainer}} and
{{CompactionStrategyManager#groupSSTables}} (but there are probably more things
that should have tests)
* Comments on the non-obvious methods in the new classes
* Update CSM class comment
pushed a branch:
https://github.com/krummas/cassandra/commits/blake/csm-refactor containing
* nits
* noticed that we used to call startup for each strategy twice in
{{CSM#startup}}, which seems wrong, removed the call outside the {{readLock}}
* renamed {{compactionStrategyIndexFor(Descriptor descriptor)}} to
{{compactionStrategyIndexForDirectory(Descriptor descriptor)}} to make it a bit
clearer what it does
there also seems to be a few test errors which might not be unrelated
> Refactor CompactionStrategyManager
> --
>
> Key: CASSANDRA-14621
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14621
> Project: Cassandra
> Issue Type: Improvement
> Components: Compaction
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
> Fix For: 4.x
>
>
> CompactionStrategyManager grew a decent amount of duplicated code as part of
> CASSANDRA-9143, which added pendingRepairs alongside the repaired and
> unrepaired buckets. At this point, the logic that routes sstables between the
> different buckets, and the different partition range divisions has gotten a
> little complex, and editing it is tedious and error prone. With transient
> replication requiring yet another bucket for, this seems like a good time to
> split some of the functionality of CSM into other classes, and make sstable
> routing a bit more generalized.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582245#comment-16582245
]
Benedict commented on CASSANDRA-14568:
--
bq. My concern was that LRT.start.bound is directly referenced in
UnfilteredDeserializer when converting to a RTMarker.
Hmm. Thinking on it some more, I guess this is not a problem due to the fact
that we never (in any extant version) actually issue any deletions that (in
3.0) would be represented as RTs spanning static rows, so the problematic cases
*should* all be converted to collection tombstones only. I will add some
comments to LegacyLayout elaborating the inconsistencies of modern/legacy
static clusterings as part of the patch.
So, I'm now comfortable with fixing either location, I think. Though I need to
code dive a bit more to be absolutely certain.
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.0.17, 3.11.3
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582147#comment-16582147
]
Benedict commented on CASSANDRA-14568:
--
Wouldn't the "more correct" place to change it be [in the bound
construction|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/LegacyLayout.java#L803]?
Presumably the LegacyBound.bound property should be correctly legacy-fied so
that any other LegacyLayout code correctly interprets / sorts it? This is what
I was planning on with [my wip
branch|https://github.com/belliottsmith/cassandra/commit/5b6742a7a6104bbd88d784db4d3bf7cd990cf057#diff-1a4af2aebd51e301cf0e73126722a8a4R805].
My concern was that LRT.start.bound is [directly referenced in
UnfilteredDeserializer when converting to a
RTMarker|https://github.com/belliottsmith/cassandra/blob/14568-2/src/java/org/apache/cassandra/db/UnfilteredDeserializer.java#L728].
I realise in the light of morning that this would not be a problem during
_serialization_, but maybe this is another bug in deserialization?
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.0.17, 3.11.3
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582108#comment-16582108
]
Sylvain Lebresne commented on CASSANDRA-14568:
--
It's absolutely possible I'm missing some problem, as god knows those backward
compatibility issues often comes with nasty surprises, but I don't think the
problem is too hard to solve.
That is, {{LegacyBound}} is only here for backward compatibility and never used
by 3.0 before being converting first, so I'm sure to understand what you mean
by "(setting it) to something that breaks the 3.0 definition of a static
clustering".
I believe we should simply make sure that when encoding to 2.x a
{{STATIC_CLUSTERING}}, we append "clustering size" empty values before adding
the rest (the column name), and when decoding, we assume those empty values are
here (but ignore them). This is what we do for "cell names"
([here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/LegacyLayout.java#L287-L291]
and
[here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/LegacyLayout.java#L329-L330])
and to the best of my knowledge is working properly.
In other words, I "think" we need to do the 2 following changes:
# revert the change made by this patch in
[{{LegacyLayout.decodeBound}}|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/LegacyLayout.java#L210-L215].
# make sure that in
[{{LegacyLayout.serializeCompound}}|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/LegacyLayout.java#L2394-L2400]
(and {{serializeSizeCompound}}), the static case is specialized to
automatically add the empty values, which might possibly be better handler
directly within {{CompositeType.Builder}}.
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.0.17, 3.11.3
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14638) Column result order can change in 'SELECT *' results when upgrading from 2.1 to 3.0 causing response corruption for queries using prepared statements when static c
[
https://issues.apache.org/jira/browse/CASSANDRA-14638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582078#comment-16582078
]
Sylvain Lebresne commented on CASSANDRA-14638:
--
I agree both that this is unfortunate, and that I don't see any reasonable
backward compatible option. And I'm also in favor or restoring the pre-3.0
order, partly because this is indeed the order we've used for the longest, but
mainly because it doesn't make sense to have inconsistent ordering for static
and normal columns.
I would also argue that relying that strongly on the order of columns of a
{{SELECT *}} is a bad idea in the first place, and to the best of my knowledge,
no official documentation has ever pretended that the order was guaranteed
(don't get me wrong, this is a weak argument in that our documentation are
historically bad and if we were to guarantee only what is documented, we
wouldn't be guaranteeing much; still, it would clearly have been worth if we
had ever documented the order as guaranteed), so hopefully this won't break too
many users. In fact, on top of the big fat NEWS entry already mentioned, I'd be
in favor of updating the documentation to explicitly mention that the order of
columns is not specified and not guaranteed to be stable, and should thus not
be relied on.
> Column result order can change in 'SELECT *' results when upgrading from 2.1
> to 3.0 causing response corruption for queries using prepared statements when
> static columns are used
> --
>
> Key: CASSANDRA-14638
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14638
> Project: Cassandra
> Issue Type: Bug
> Environment: Single C* node ccm cluster upgraded from C* 2.1.20 to
> 3.0.17
>Reporter: Andy Tolbert
>Assignee: Aleksey Yeschenko
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
>
> When performing an upgrade from C* 2.1.20 to 3.0.17 I observed that the order
> of columns returned from a 'SELECT *' query changes, particularly when static
> columns are involved.
> This may not seem like that much of a problem, however if using Prepared
> Statements, any clients that remain connected during the upgrade may
> encounter issues consuming results from these queries, as data is reordered
> and the client not aware of it. The result definition is sent in the
> original prepared statement response, so if order changes the client has no
> way of knowing (until C* 4.0 via CASSANDRA-10786) without re-preparing, which
> is non-trivial as most client drivers cache prepared statements.
> This could lead to reading the wrong values for columns, which could result
> in some kind of deserialization exception or if the data types of the
> switched columns are compatible, the wrong values. This happens even if the
> client attempts to retrieve a column value by name (i.e. row.getInt("colx")).
> Unfortunately I don't think there is an easy fix for this. If the order was
> changed back to the previous format, you risk issues for users upgrading from
> older 3.0 version. I think it would be nice to add a note in the NEWS file
> in the 3.0 upgrade section that describes this issue, and how to work around
> it (specify all column names of interest explicitly in query).
> Example schema and code to reproduce:
>
> {noformat}
> create keyspace ks with replication = {'class': 'SimpleStrategy',
> 'replication_factor': 1};
> create table ks.tbl (p0 text,
> p1 text,
> m map static,
> t text,
> u text static,
> primary key (p0, p1)
> );
> insert into ks.tbl (p0, p1, m, t, u) values ('p0', 'p1', { 'm0' : 'm1' },
> 't', 'u');{noformat}
>
> When querying with 2.1 you'll observe the following order via cqlsh:
> {noformat}
> p0 | p1 | m| u | t
> ++--+---+---
> p0 | p1 | {'m0': 'm1'} | u | t{noformat}
>
> With 3.0, observe that u and m are transposed:
>
> {noformat}
> p0 | p1 | u | m | t
> ++---+--+---
> p0 | p1 | u | {'m0': 'm1'} | t{noformat}
>
>
> {code:java}
> import com.datastax.driver.core.BoundStatement;
> import com.datastax.driver.core.Cluster;
> import com.datastax.driver.core.ColumnDefinitions;
> import com.datastax.driver.core.PreparedStatement;
> import com.datastax.driver.core.ResultSet;
> import com.datastax.driver.core.Row;
> import com.datastax.driver.core.Session;
> import com.google.common.util.concurrent.Uninterruptibles;
> import java.util.concurrent.TimeUnit;
> public class LiveUpgradeTest {
> public static void main(String args[]) {
> Cluster cluster = Cluster.builder().addContactPoints("127.0.0.1").build();
> try {
> Session session =
[jira] [Commented] (CASSANDRA-14436) Add sampler for query time and expose with nodetool
[
https://issues.apache.org/jira/browse/CASSANDRA-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582036#comment-16582036
]
Dinesh Joshi commented on CASSANDRA-14436:
--
Hey [~cnlwsu], I have left a few more comments. I think other than those, this
looks good.
> Add sampler for query time and expose with nodetool
> ---
>
> Key: CASSANDRA-14436
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14436
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Chris Lohfink
>Assignee: Chris Lohfink
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Create a new {{nodetool profileload}} that functions just like toppartitions
> but with more data, returning the slowest local reads and writes on the host
> during a given duration and highest frequency touched partitions (same as
> {{nodetool toppartitions}}). Refactor included to extend use of the sampler
> for uses outside of top frequency (max instead of total sample values).
> Future work to this is to include top cpu and allocations by query and
> possibly tasks/cpu/allocations by stage during time window.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14436) Add sampler for query time and expose with nodetool
[
https://issues.apache.org/jira/browse/CASSANDRA-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated CASSANDRA-14436:
---
Labels: pull-request-available (was: )
> Add sampler for query time and expose with nodetool
> ---
>
> Key: CASSANDRA-14436
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14436
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Chris Lohfink
>Assignee: Chris Lohfink
>Priority: Major
> Labels: pull-request-available
>
> Create a new {{nodetool profileload}} that functions just like toppartitions
> but with more data, returning the slowest local reads and writes on the host
> during a given duration and highest frequency touched partitions (same as
> {{nodetool toppartitions}}). Refactor included to extend use of the sampler
> for uses outside of top frequency (max instead of total sample values).
> Future work to this is to include top cpu and allocations by query and
> possibly tasks/cpu/allocations by stage during time window.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14436) Add sampler for query time and expose with nodetool
[
https://issues.apache.org/jira/browse/CASSANDRA-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated CASSANDRA-14436:
---
Labels: pull-request-available (was: )
> Add sampler for query time and expose with nodetool
> ---
>
> Key: CASSANDRA-14436
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14436
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Chris Lohfink
>Assignee: Chris Lohfink
>Priority: Major
> Labels: pull-request-available
>
> Create a new {{nodetool profileload}} that functions just like toppartitions
> but with more data, returning the slowest local reads and writes on the host
> during a given duration and highest frequency touched partitions (same as
> {{nodetool toppartitions}}). Refactor included to extend use of the sampler
> for uses outside of top frequency (max instead of total sample values).
> Future work to this is to include top cpu and allocations by query and
> possibly tasks/cpu/allocations by stage during time window.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14646) built_views entries are not removed after dropping keyspace
ZhaoYang created CASSANDRA-14646:
Summary: built_views entries are not removed after dropping
keyspace
Key: CASSANDRA-14646
URL: https://issues.apache.org/jira/browse/CASSANDRA-14646
Project: Cassandra
Issue Type: Bug
Components: Distributed Metadata, Materialized Views
Reporter: ZhaoYang
Assignee: ZhaoYang
Fix For: 4.x
If we restore view schema after dropping keyspace, view build won't be
triggered because it was marked as SUCCESS in {{built_views}} table.
| patch | CI |
| [trunk|https://github.com/jasonstack/cassandra/commits/mv_drop_ks] |
[utest|https://circleci.com/gh/jasonstack/cassandra/739] |
| [dtest|https://github.com/apache/cassandra-dtest/pull/36]|
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
