[jira] [Commented] (CASSANDRA-15299) CASSANDRA-13304 follow-up: improve checksumming and compression in protocol v5-beta
[
https://issues.apache.org/jira/browse/CASSANDRA-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113741#comment-17113741
]
Olivier Michallat commented on CASSANDRA-15299:
---
The server is hitting [this
line|https://github.com/beobal/cassandra/blob/1a8a79b1f5ff6f863263a28b1160431b8ba6eef5/src/java/org/apache/cassandra/net/FrameEncoderLZ4.java#L75]
when it sends back a compressed large response. I haven't dug further, but the
the offending length is exactly equal to 128KiB, it's probably just an
off-by-one error when it's being split it into chunks.
> CASSANDRA-13304 follow-up: improve checksumming and compression in protocol
> v5-beta
> ---
>
> Key: CASSANDRA-15299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15299
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Aleksey Yeschenko
>Assignee: Sam Tunnicliffe
>Priority: Normal
> Labels: protocolv5
> Fix For: 4.0-alpha
>
>
> CASSANDRA-13304 made an important improvement to our native protocol: it
> introduced checksumming/CRC32 to request and response bodies. It’s an
> important step forward, but it doesn’t cover the entire stream. In
> particular, the message header is not covered by a checksum or a crc, which
> poses a correctness issue if, for example, {{streamId}} gets corrupted.
> Additionally, we aren’t quite using CRC32 correctly, in two ways:
> 1. We are calculating the CRC32 of the *decompressed* value instead of
> computing the CRC32 on the bytes written on the wire - losing the properties
> of the CRC32. In some cases, due to this sequencing, attempting to decompress
> a corrupt stream can cause a segfault by LZ4.
> 2. When using CRC32, the CRC32 value is written in the incorrect byte order,
> also losing some of the protections.
> See https://users.ece.cmu.edu/~koopman/pubs/KoopmanCRCWebinar9May2012.pdf for
> explanation for the two points above.
> Separately, there are some long-standing issues with the protocol - since
> *way* before CASSANDRA-13304. Importantly, both checksumming and compression
> operate on individual message bodies rather than frames of multiple complete
> messages. In reality, this has several important additional downsides. To
> name a couple:
> # For compression, we are getting poor compression ratios for smaller
> messages - when operating on tiny sequences of bytes. In reality, for most
> small requests and responses we are discarding the compressed value as it’d
> be smaller than the uncompressed one - incurring both redundant allocations
> and compressions.
> # For checksumming and CRC32 we pay a high overhead price for small messages.
> 4 bytes extra is *a lot* for an empty write response, for example.
> To address the correctness issue of {{streamId}} not being covered by the
> checksum/CRC32 and the inefficiency in compression and checksumming/CRC32, we
> should switch to a framing protocol with multiple messages in a single frame.
> I suggest we reuse the framing protocol recently implemented for internode
> messaging in CASSANDRA-15066 to the extent that its logic can be borrowed,
> and that we do it before native protocol v5 graduates from beta. See
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderCrc.java
> and
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderLZ4.java.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15299) CASSANDRA-13304 follow-up: improve checksumming and compression in protocol v5-beta
[
https://issues.apache.org/jira/browse/CASSANDRA-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113615#comment-17113615
]
Olivier Michallat commented on CASSANDRA-15299:
---
Adding an issue link to the Java driver pull request.
Apart from the negotiation issue (that I "fixed" with a hacky server-side
workaround), it seems to work pretty nicely, I see small messages getting
batched and large messages getting split.
Compression crashes for some reason, I'll look into it.
> CASSANDRA-13304 follow-up: improve checksumming and compression in protocol
> v5-beta
> ---
>
> Key: CASSANDRA-15299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15299
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Aleksey Yeschenko
>Assignee: Sam Tunnicliffe
>Priority: Normal
> Labels: protocolv5
> Fix For: 4.0-alpha
>
>
> CASSANDRA-13304 made an important improvement to our native protocol: it
> introduced checksumming/CRC32 to request and response bodies. It’s an
> important step forward, but it doesn’t cover the entire stream. In
> particular, the message header is not covered by a checksum or a crc, which
> poses a correctness issue if, for example, {{streamId}} gets corrupted.
> Additionally, we aren’t quite using CRC32 correctly, in two ways:
> 1. We are calculating the CRC32 of the *decompressed* value instead of
> computing the CRC32 on the bytes written on the wire - losing the properties
> of the CRC32. In some cases, due to this sequencing, attempting to decompress
> a corrupt stream can cause a segfault by LZ4.
> 2. When using CRC32, the CRC32 value is written in the incorrect byte order,
> also losing some of the protections.
> See https://users.ece.cmu.edu/~koopman/pubs/KoopmanCRCWebinar9May2012.pdf for
> explanation for the two points above.
> Separately, there are some long-standing issues with the protocol - since
> *way* before CASSANDRA-13304. Importantly, both checksumming and compression
> operate on individual message bodies rather than frames of multiple complete
> messages. In reality, this has several important additional downsides. To
> name a couple:
> # For compression, we are getting poor compression ratios for smaller
> messages - when operating on tiny sequences of bytes. In reality, for most
> small requests and responses we are discarding the compressed value as it’d
> be smaller than the uncompressed one - incurring both redundant allocations
> and compressions.
> # For checksumming and CRC32 we pay a high overhead price for small messages.
> 4 bytes extra is *a lot* for an empty write response, for example.
> To address the correctness issue of {{streamId}} not being covered by the
> checksum/CRC32 and the inefficiency in compression and checksumming/CRC32, we
> should switch to a framing protocol with multiple messages in a single frame.
> I suggest we reuse the framing protocol recently implemented for internode
> messaging in CASSANDRA-15066 to the extent that its logic can be borrowed,
> and that we do it before native protocol v5 graduates from beta. See
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderCrc.java
> and
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderLZ4.java.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15829) Upgrade to logback 1.2.3 to address CVE
[ https://issues.apache.org/jira/browse/CASSANDRA-15829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Eveker updated CASSANDRA-15829: - Description: Recent scan results identified the following CVE that requires this upgrade to address [https://nvd.nist.gov/vuln/detail/CVE-2017-5929] was: Recent scan results identified the following CVE that require this upgrade to address [https://nvd.nist.gov/vuln/detail/CVE-2019-10172] > Upgrade to logback 1.2.3 to address CVE > --- > > Key: CASSANDRA-15829 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15829 > Project: Cassandra > Issue Type: Improvement >Reporter: Kevin Eveker >Priority: Normal > > Recent scan results identified the following CVE that requires this upgrade > to address > [https://nvd.nist.gov/vuln/detail/CVE-2017-5929] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15829) Upgrade to logback 1.2.3 to address CVE
Kevin Eveker created CASSANDRA-15829: Summary: Upgrade to logback 1.2.3 to address CVE Key: CASSANDRA-15829 URL: https://issues.apache.org/jira/browse/CASSANDRA-15829 Project: Cassandra Issue Type: Improvement Reporter: Kevin Eveker Recent scan results identified the following CVE that require this upgrade to address [https://nvd.nist.gov/vuln/detail/CVE-2019-10172] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15828) Remove jackson-mapper-asl-1.9.13 to address CVE
[ https://issues.apache.org/jira/browse/CASSANDRA-15828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Eveker updated CASSANDRA-15828: - Description: Recent scan results identified the following CVE that require this upgrade to address [https://nvd.nist.gov/vuln/detail/CVE-2019-10172] was: Recent scan results identified the following CVE that require this upgrade to address [https://nvd.nist.gov/vuln/detail/]CVE-2019-10172 > Remove jackson-mapper-asl-1.9.13 to address CVE > --- > > Key: CASSANDRA-15828 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15828 > Project: Cassandra > Issue Type: Improvement >Reporter: Kevin Eveker >Priority: Normal > > Recent scan results identified the following CVE that require this upgrade to > address > [https://nvd.nist.gov/vuln/detail/CVE-2019-10172] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15827) Upgrade to Jackson Databind 2.9.10.4 or later to address CVEs
[ https://issues.apache.org/jira/browse/CASSANDRA-15827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Eveker updated CASSANDRA-15827: - Description: Recent scan results identified the following CVEs that require this upgrade to address [https://nvd.nist.gov/vuln/detail/CVE-2020-8840] [https://nvd.nist.gov/vuln/detail/CVE-2020-9548] [https://nvd.nist.gov/vuln/detail/CVE-2020-9547] [https://nvd.nist.gov/vuln/detail/CVE-2020-9546] [https://nvd.nist.gov/vuln/detail/CVE-2020-10673] [https://nvd.nist.gov/vuln/detail/CVE-2020-10672] [https://nvd.nist.gov/vuln/detail/CVE-2020-10968] [https://nvd.nist.gov/vuln/detail/CVE-2020-10969] [https://nvd.nist.gov/vuln/detail/CVE-2020-2] [https://nvd.nist.gov/vuln/detail/CVE-2020-3] [https://nvd.nist.gov/vuln/detail/CVE-2020-1] [https://nvd.nist.gov/vuln/detail/CVE-2020-11619] [https://nvd.nist.gov/vuln/detail/CVE-2020-11620] was: Recent scan results identified the following CVEs that require this upgrade to address CVE-2020-8840 CVE-2020-9548 CVE-2020-9547 CVE-2020-9546 CVE-2020-10673 CVE-2020-10672 CVE-2020-10968 CVE-2020-10969 CVE-2020-2 CVE-2020-3 CVE-2020-1 CVE-2020-11619 CVE-2020-11620 > Upgrade to Jackson Databind 2.9.10.4 or later to address CVEs > - > > Key: CASSANDRA-15827 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15827 > Project: Cassandra > Issue Type: Improvement >Reporter: Kevin Eveker >Priority: Normal > > Recent scan results identified the following CVEs that require this upgrade > to address > [https://nvd.nist.gov/vuln/detail/CVE-2020-8840] > [https://nvd.nist.gov/vuln/detail/CVE-2020-9548] > [https://nvd.nist.gov/vuln/detail/CVE-2020-9547] > [https://nvd.nist.gov/vuln/detail/CVE-2020-9546] > [https://nvd.nist.gov/vuln/detail/CVE-2020-10673] > [https://nvd.nist.gov/vuln/detail/CVE-2020-10672] > [https://nvd.nist.gov/vuln/detail/CVE-2020-10968] > [https://nvd.nist.gov/vuln/detail/CVE-2020-10969] > [https://nvd.nist.gov/vuln/detail/CVE-2020-2] > [https://nvd.nist.gov/vuln/detail/CVE-2020-3] > [https://nvd.nist.gov/vuln/detail/CVE-2020-1] > [https://nvd.nist.gov/vuln/detail/CVE-2020-11619] > [https://nvd.nist.gov/vuln/detail/CVE-2020-11620] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15828) Remove jackson-mapper-asl-1.9.13 to address CVE
[ https://issues.apache.org/jira/browse/CASSANDRA-15828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Eveker updated CASSANDRA-15828: - Description: Recent scan results identified the following CVE that require this upgrade to address [https://nvd.nist.gov/vuln/detail/]CVE-2019-10172 was: Recent scan results identified the following CVEs that require this upgrade to address CVE-2020-8840 CVE-2020-9548 CVE-2020-9547 CVE-2020-9546 CVE-2020-10673 CVE-2020-10672 CVE-2020-10968 CVE-2020-10969 CVE-2020-2 CVE-2020-3 CVE-2020-1 CVE-2020-11619 CVE-2020-11620 > Remove jackson-mapper-asl-1.9.13 to address CVE > --- > > Key: CASSANDRA-15828 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15828 > Project: Cassandra > Issue Type: Improvement >Reporter: Kevin Eveker >Priority: Normal > > Recent scan results identified the following CVE that require this upgrade to > address > [https://nvd.nist.gov/vuln/detail/]CVE-2019-10172 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15828) Remove jackson-mapper-asl-1.9.13 to address CVE
Kevin Eveker created CASSANDRA-15828: Summary: Remove jackson-mapper-asl-1.9.13 to address CVE Key: CASSANDRA-15828 URL: https://issues.apache.org/jira/browse/CASSANDRA-15828 Project: Cassandra Issue Type: Improvement Reporter: Kevin Eveker Recent scan results identified the following CVEs that require this upgrade to address CVE-2020-8840 CVE-2020-9548 CVE-2020-9547 CVE-2020-9546 CVE-2020-10673 CVE-2020-10672 CVE-2020-10968 CVE-2020-10969 CVE-2020-2 CVE-2020-3 CVE-2020-1 CVE-2020-11619 CVE-2020-11620 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15827) Upgrade to Jackson Databind 2.9.10.4 or later to address CVEs
[ https://issues.apache.org/jira/browse/CASSANDRA-15827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Eveker updated CASSANDRA-15827: - Impacts: Security (was: None) > Upgrade to Jackson Databind 2.9.10.4 or later to address CVEs > - > > Key: CASSANDRA-15827 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15827 > Project: Cassandra > Issue Type: Improvement >Reporter: Kevin Eveker >Priority: Normal > > Recent scan results identified the following CVEs that require this upgrade > to address > CVE-2020-8840 > CVE-2020-9548 > CVE-2020-9547 > CVE-2020-9546 > CVE-2020-10673 > CVE-2020-10672 > CVE-2020-10968 > CVE-2020-10969 > CVE-2020-2 > CVE-2020-3 > CVE-2020-1 > CVE-2020-11619 > CVE-2020-11620 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15827) Upgrade to Jackson Databind 2.9.10.4 or later to address CVEs
Kevin Eveker created CASSANDRA-15827: Summary: Upgrade to Jackson Databind 2.9.10.4 or later to address CVEs Key: CASSANDRA-15827 URL: https://issues.apache.org/jira/browse/CASSANDRA-15827 Project: Cassandra Issue Type: Improvement Reporter: Kevin Eveker Recent scan results identified the following CVEs that require this upgrade to address CVE-2020-8840 CVE-2020-9548 CVE-2020-9547 CVE-2020-9546 CVE-2020-10673 CVE-2020-10672 CVE-2020-10968 CVE-2020-10969 CVE-2020-2 CVE-2020-3 CVE-2020-1 CVE-2020-11619 CVE-2020-11620 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-8272) 2ndary indexes can return stale data
[
https://issues.apache.org/jira/browse/CASSANDRA-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Caleb Rackliffe updated CASSANDRA-8272:
---
Reviewers: Benjamin Lerer, Caleb Rackliffe, ZhaoYang (was: Benjamin Lerer,
ZhaoYang)
> 2ndary indexes can return stale data
>
>
> Key: CASSANDRA-8272
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8272
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/2i Index
>Reporter: Sylvain Lebresne
>Assignee: Andres de la Peña
>Priority: Normal
> Labels: pull-request-available
> Fix For: 3.0.x, 4.0-beta
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> When replica return 2ndary index results, it's possible for a single replica
> to return a stale result and that result will be sent back to the user,
> potentially failing the CL contract.
> For instance, consider 3 replicas A, B and C, and the following situation:
> {noformat}
> CREATE TABLE test (k int PRIMARY KEY, v text);
> CREATE INDEX ON test(v);
> INSERT INTO test(k, v) VALUES (0, 'foo');
> {noformat}
> with every replica up to date. Now, suppose that the following queries are
> done at {{QUORUM}}:
> {noformat}
> UPDATE test SET v = 'bar' WHERE k = 0;
> SELECT * FROM test WHERE v = 'foo';
> {noformat}
> then, if A and B acknowledge the insert but C respond to the read before
> having applied the insert, then the now stale result will be returned (since
> C will return it and A or B will return nothing).
> A potential solution would be that when we read a tombstone in the index (and
> provided we make the index inherit the gcGrace of it's parent CF), instead of
> skipping that tombstone, we'd insert in the result a corresponding range
> tombstone.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15826) Jenkins dtests don't need to be category throttled any longer
[ https://issues.apache.org/jira/browse/CASSANDRA-15826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-15826: --- Test and Documentation Plan: review and ASF CI Status: Patch Available (was: Open) > Jenkins dtests don't need to be category throttled any longer > - > > Key: CASSANDRA-15826 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15826 > Project: Cassandra > Issue Type: Task > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-beta > > > Remove the "Cassandra" category throttle on dtests. > Add docs section on configuring the Jenkins master to create the "Cassandra" > category throttle. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15826) Jenkins dtests don't need to be category throttled any longer
[ https://issues.apache.org/jira/browse/CASSANDRA-15826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-15826: --- Change Category: Quality Assurance Complexity: Low Hanging Fruit Status: Open (was: Triage Needed) > Jenkins dtests don't need to be category throttled any longer > - > > Key: CASSANDRA-15826 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15826 > Project: Cassandra > Issue Type: Task > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-beta > > > Remove the "Cassandra" category throttle on dtests. > Add docs section on configuring the Jenkins master to create the "Cassandra" > category throttle. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15802) Commented-out lines that end in a semicolon cause an error.
[ https://issues.apache.org/jira/browse/CASSANDRA-15802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rens Groothuijsen reassigned CASSANDRA-15802: - Assignee: Rens Groothuijsen > Commented-out lines that end in a semicolon cause an error. > --- > > Key: CASSANDRA-15802 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15802 > Project: Cassandra > Issue Type: Bug > Components: CQL/Interpreter, CQL/Syntax >Reporter: null >Assignee: Rens Groothuijsen >Priority: Normal > Attachments: cqlsh.png > > Time Spent: 10m > Remaining Estimate: 0h > > Commented-out lines that end in a semicolon cause an error. > For example: > /* > describe keyspaces; > */ > > This produces an error: > SyntaxException: line 2:22 no viable alternative at input ' (...* > describe keyspaces;...) > > It works as expected if you use syntax: > -- describe keyspaces; > > Environment: > python:3.7.7-slim-stretch (docker image) > > I found that this was first seen here, and was patched, but the bug appears > to have resurfaced: > https://issues.apache.org/jira/browse/CASSANDRA-2488 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15799) CorruptSSTableException when compacting a 3.0 format sstable that was originally created as an outcome of 2.1 sstable upgrade
[
https://issues.apache.org/jira/browse/CASSANDRA-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Capwell updated CASSANDRA-15799:
--
Attachment: fake-deletedcell-if-bad.patch
> CorruptSSTableException when compacting a 3.0 format sstable that was
> originally created as an outcome of 2.1 sstable upgrade
> -
>
> Key: CASSANDRA-15799
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15799
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction, Local/SSTable
>Reporter: Sumanth Pasupuleti
>Assignee: David Capwell
>Priority: Normal
> Fix For: 3.0.x
>
> Attachments: fake-deletedcell-if-bad.patch
>
>
> Below is the exception with stack trace. This issue is reproduce-able.
> {code:java}
> DEBUG [CompactionExecutor:10] 2020-05-07 19:33:34,268 CompactionTask.java:158
> - Compacting (a3ea9fc0-9099-11ea-933f-c5e852f71338)
> [/mnt/data/cassandra/data/ks/cf/md-10802-big-Data.db:level=0, ]
> ERROR [CompactionExecutor:10] 2020-05-07 19:33:34,275
> CassandraDaemon.java:208 - Exception in thread
> Thread[CompactionExecutor:10,1,RMI Runtime]
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
> /mnt/data/cassandra/data/ks/cf/md-10802-big-Data.db
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:105)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:534)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:394)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:111)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:52)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:165)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:126)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:57)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:675)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
>
[jira] [Commented] (CASSANDRA-15826) Jenkins dtests don't need to be category throttled any longer
[ https://issues.apache.org/jira/browse/CASSANDRA-15826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113074#comment-17113074 ] Michael Semb Wever commented on CASSANDRA-15826: Patches: - https://github.com/apache/cassandra-builds/pull/21 - https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/trunk_15826 fyi [~slebresne] > Jenkins dtests don't need to be category throttled any longer > - > > Key: CASSANDRA-15826 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15826 > Project: Cassandra > Issue Type: Task > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-beta > > > Remove the "Cassandra" category throttle on dtests. > Add docs section on configuring the Jenkins master to create the "Cassandra" > category throttle. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15826) Jenkins dtests don't need to be category throttled any longer
[ https://issues.apache.org/jira/browse/CASSANDRA-15826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-15826: --- Fix Version/s: 4.0-beta > Jenkins dtests don't need to be category throttled any longer > - > > Key: CASSANDRA-15826 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15826 > Project: Cassandra > Issue Type: Task > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-beta > > > Remove the "Cassandra" category throttle on dtests. > Add docs section on configuring the Jenkins master to create the "Cassandra" > category throttle. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15826) Jenkins dtests don't need to be category throttled any longer
Michael Semb Wever created CASSANDRA-15826: -- Summary: Jenkins dtests don't need to be category throttled any longer Key: CASSANDRA-15826 URL: https://issues.apache.org/jira/browse/CASSANDRA-15826 Project: Cassandra Issue Type: Task Components: CI Reporter: Michael Semb Wever Remove the "Cassandra" category throttle on dtests. Add docs section on configuring the Jenkins master to create the "Cassandra" category throttle. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15826) Jenkins dtests don't need to be category throttled any longer
[ https://issues.apache.org/jira/browse/CASSANDRA-15826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever reassigned CASSANDRA-15826: -- Assignee: Michael Semb Wever > Jenkins dtests don't need to be category throttled any longer > - > > Key: CASSANDRA-15826 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15826 > Project: Cassandra > Issue Type: Task > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > > Remove the "Cassandra" category throttle on dtests. > Add docs section on configuring the Jenkins master to create the "Cassandra" > category throttle. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15677) Topology events are not sent to clients if the nodes use the same network interface
[
https://issues.apache.org/jira/browse/CASSANDRA-15677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113044#comment-17113044
]
Bryn Cooke commented on CASSANDRA-15677:
[~dcapwell] I think that I'll need a bump to cassandra-in-jvm-dtest-api to make
this happen. I need
[https://github.com/apache/cassandra-in-jvm-dtest-api/commit/326045f699791686efa1ecf43b7353397c956494]
Is cassandra-in-jvm-dtest-api in a position to create a new release? If so I
could create a PR to handle the changes on the C* side.
> Topology events are not sent to clients if the nodes use the same network
> interface
> ---
>
> Key: CASSANDRA-15677
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15677
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Client
>Reporter: Alan Boudreault
>Assignee: Bryn Cooke
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-rc
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> *This bug only happens when the cassandra nodes are configured to use a
> single network interface (ip) but different ports. See CASSANDRA-7544.*
> Issue: The topology events aren't sent to clients. The problem is that the
> port is not taken into account when determining if we send it or not:
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/transport/Server.java#L624
> To reproduce:
> {code}
> # I think the cassandra-test branch is required to get the -S option
> (USE_SINGLE_INTERFACE)
> ccm create -n4 local40 -v 4.0-alpha2 -S
> {code}
>
> Then run this small python driver script:
> {code}
> import time
> from cassandra.cluster import Cluster
> cluster = Cluster()
> session = cluster.connect()
> while True:
> print(cluster.metadata.all_hosts())
> print([h.is_up for h in cluster.metadata.all_hosts()])
> time.sleep(5)
> {code}
> Then decommission a node:
> {code}
> ccm node2 nodetool disablebinary
> ccm node2 nodetool decommission
> {code}
>
> You should see that the node is never removed from the client cluster
> metadata and the reconnector started.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15725) Add support for adding custom Verbs
[
https://issues.apache.org/jira/browse/CASSANDRA-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113030#comment-17113030
]
Benedict Elliott Smith commented on CASSANDRA-15725:
Sorry for the slow response. I really like the approach taken, of permitting
authors of custom verbs to specify integers in the normal way, just with their
own distinct number space.
I have just some minor cosmetic suggestions, none of them important and happy
to be overruled:
1. Perhaps we want to specify a maximum custom id, so that we leave plenty of
room for Cassandra growing upwards without accidentally infringing on custom
spaces that have begun to be used by others?
2. I think it _might_ help to introduce a {{Kind}} enum with e.g. {{NORMAL,
CUSTOM}}, and for the custom constructor accept {{Kind}} as the first argument,
so you would see {{UNUSED_CUSTOM_VERB (CUSTOM, 0...}} which might help prevent
fat finger errors, and also demarcate more visibly the compiler-enforced divide
between the verbs
3. In {{fromId}} it might be ever so slightly cleaner to simply choose the
array to look inside, and update {{id}}, e.g.:
{code}
public static Verb fromId(int id)
{
Verb[] verbs = idToVerbMap;
if (id >= minCustomId)
{
id = idForCustomVerb(id);
verbs = idToCustomVerbMap;
}
Verb verb = id >= 0 && id < verbs.length ? verbs[id] : null;
if (verb == null)
throw new IllegalArgumentException("Unknown verb id " + id);
return verb;
}
{code}
Either way, LGTM +1
> Add support for adding custom Verbs
> ---
>
> Key: CASSANDRA-15725
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15725
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Internode
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 4.0-alpha
>
> Attachments: feedback_15725.patch
>
>
> It should be possible to safely add custom/internal Verbs - without risking
> conflicts when new ones are added.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15299) CASSANDRA-13304 follow-up: improve checksumming and compression in protocol v5-beta
[
https://issues.apache.org/jira/browse/CASSANDRA-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112997#comment-17112997
]
Sam Tunnicliffe commented on CASSANDRA-15299:
-
bq. Last remark on the protocol: I think the response to the initial STARTUP
message should never be wrapped either.
I went back and forth on this a bit myself, and to be honest I don't feel super
strongly about it. It was primarily implementation concerns that lead me to
end up with the framing starting on the initial response. What you say makes
sense though and that's clearly more important, so I'll look at reworking along
those lines.
> CASSANDRA-13304 follow-up: improve checksumming and compression in protocol
> v5-beta
> ---
>
> Key: CASSANDRA-15299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15299
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Aleksey Yeschenko
>Assignee: Sam Tunnicliffe
>Priority: Normal
> Labels: protocolv5
> Fix For: 4.0-alpha
>
>
> CASSANDRA-13304 made an important improvement to our native protocol: it
> introduced checksumming/CRC32 to request and response bodies. It’s an
> important step forward, but it doesn’t cover the entire stream. In
> particular, the message header is not covered by a checksum or a crc, which
> poses a correctness issue if, for example, {{streamId}} gets corrupted.
> Additionally, we aren’t quite using CRC32 correctly, in two ways:
> 1. We are calculating the CRC32 of the *decompressed* value instead of
> computing the CRC32 on the bytes written on the wire - losing the properties
> of the CRC32. In some cases, due to this sequencing, attempting to decompress
> a corrupt stream can cause a segfault by LZ4.
> 2. When using CRC32, the CRC32 value is written in the incorrect byte order,
> also losing some of the protections.
> See https://users.ece.cmu.edu/~koopman/pubs/KoopmanCRCWebinar9May2012.pdf for
> explanation for the two points above.
> Separately, there are some long-standing issues with the protocol - since
> *way* before CASSANDRA-13304. Importantly, both checksumming and compression
> operate on individual message bodies rather than frames of multiple complete
> messages. In reality, this has several important additional downsides. To
> name a couple:
> # For compression, we are getting poor compression ratios for smaller
> messages - when operating on tiny sequences of bytes. In reality, for most
> small requests and responses we are discarding the compressed value as it’d
> be smaller than the uncompressed one - incurring both redundant allocations
> and compressions.
> # For checksumming and CRC32 we pay a high overhead price for small messages.
> 4 bytes extra is *a lot* for an empty write response, for example.
> To address the correctness issue of {{streamId}} not being covered by the
> checksum/CRC32 and the inefficiency in compression and checksumming/CRC32, we
> should switch to a framing protocol with multiple messages in a single frame.
> I suggest we reuse the framing protocol recently implemented for internode
> messaging in CASSANDRA-15066 to the extent that its logic can be borrowed,
> and that we do it before native protocol v5 graduates from beta. See
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderCrc.java
> and
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderLZ4.java.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15299) CASSANDRA-13304 follow-up: improve checksumming and compression in protocol v5-beta
[
https://issues.apache.org/jira/browse/CASSANDRA-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112995#comment-17112995
]
Sam Tunnicliffe commented on CASSANDRA-15299:
-
When we spoke in on slack, and I'm afraid the answers I gave were only
partially correct.
https://the-asf.slack.com/archives/CK23JSY2K/p1589982060404400
The good news is that your approach is right and it will correctly decode the
header, but the diagram is also correct. The disconnect is that the diagram is
describing the bytes as they are laid out on the wire at the data layer,
whereas in C* and the Java Driver we're working at a point somewhat higher up
the stack. That's perhaps not as clear as it could be and I'll definitely work
on improving the docs in that regard.
To deconstruct your example, you have the bytes in the buffer with a binary
representation of
{{0100 0010 0001 1001 0010}}
The first 3 bytes {{0100 0010}} are the header length + flag.
{{4 |= (1 << 17) == 131076 == 0010 0100}}
Writing them to the buffer in little endian ordering (in
{{FrameEncodedCrc::put3b}}):
{{0100 0010}}
and reading them off the wire, where ethernet transmission specifies bytes are
sent least significant bit first:
{{0100 0010}}
so the 18th bit read off the wire is the {{1}} representing the
{{selfContained}} flag, but to extract it requires the steps you outlined
including the endianness switch.
The second 3 bytes in your buffer {{0001 1001 0010}} is the crc24
in little endian order. Here though, the bit order is less important as up the
stack we're dealing in whole bytes.
So on the wire we'd see {{1000 1001 0100}}, and read that as
{{0001 1001 0010}}.
Then in {{FrameDecoderCrc::readHeader6b}} we change the endianness to give
{{0010 1001 0001 == 15923457 == crc24(131076)}}
> CASSANDRA-13304 follow-up: improve checksumming and compression in protocol
> v5-beta
> ---
>
> Key: CASSANDRA-15299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15299
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Aleksey Yeschenko
>Assignee: Sam Tunnicliffe
>Priority: Normal
> Labels: protocolv5
> Fix For: 4.0-alpha
>
>
> CASSANDRA-13304 made an important improvement to our native protocol: it
> introduced checksumming/CRC32 to request and response bodies. It’s an
> important step forward, but it doesn’t cover the entire stream. In
> particular, the message header is not covered by a checksum or a crc, which
> poses a correctness issue if, for example, {{streamId}} gets corrupted.
> Additionally, we aren’t quite using CRC32 correctly, in two ways:
> 1. We are calculating the CRC32 of the *decompressed* value instead of
> computing the CRC32 on the bytes written on the wire - losing the properties
> of the CRC32. In some cases, due to this sequencing, attempting to decompress
> a corrupt stream can cause a segfault by LZ4.
> 2. When using CRC32, the CRC32 value is written in the incorrect byte order,
> also losing some of the protections.
> See https://users.ece.cmu.edu/~koopman/pubs/KoopmanCRCWebinar9May2012.pdf for
> explanation for the two points above.
> Separately, there are some long-standing issues with the protocol - since
> *way* before CASSANDRA-13304. Importantly, both checksumming and compression
> operate on individual message bodies rather than frames of multiple complete
> messages. In reality, this has several important additional downsides. To
> name a couple:
> # For compression, we are getting poor compression ratios for smaller
> messages - when operating on tiny sequences of bytes. In reality, for most
> small requests and responses we are discarding the compressed value as it’d
> be smaller than the uncompressed one - incurring both redundant allocations
> and compressions.
> # For checksumming and CRC32 we pay a high overhead price for small messages.
> 4 bytes extra is *a lot* for an empty write response, for example.
> To address the correctness issue of {{streamId}} not being covered by the
> checksum/CRC32 and the inefficiency in compression and checksumming/CRC32, we
> should switch to a framing protocol with multiple messages in a single frame.
> I suggest we reuse the framing protocol recently implemented for internode
> messaging in CASSANDRA-15066 to the extent that its logic can be borrowed,
> and that we do it before native protocol v5 graduates from beta. See
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderCrc.java
> and
>
[jira] [Updated] (CASSANDRA-15797) Fix flaky BinLogTest - org.apache.cassandra.utils.binlog.BinLogTest
[
https://issues.apache.org/jira/browse/CASSANDRA-15797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinay Chella updated CASSANDRA-15797:
-
Fix Version/s: (was: 4.0-alpha)
4.0-alpha5
4.0
Since Version: 4.0-alpha
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
Thank you for the [~yifanc]. Squashed your changes and committed into trunk at
[43c19878e38fbe260f9e6143aa43836e85cf2f44|https://github.com/apache/cassandra/commit/43c19878e38fbe260f9e6143aa43836e85cf2f44]
> Fix flaky BinLogTest - org.apache.cassandra.utils.binlog.BinLogTest
> ---
>
> Key: CASSANDRA-15797
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15797
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Jon Meredith
>Assignee: Yifan Cai
>Priority: Normal
> Fix For: 4.0, 4.0-alpha5
>
>
> An internal CI system is failing BinLogTest somewhat frequently under JDK11.
> Configuration was recently changed to reduce the number of cores the tests
> run with, however it is reproducible on an 8 core laptop.
> {code}
> [junit-timeout] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC
> was deprecated in version 9.0 and will likely be removed in a future release.
> [junit-timeout] Testsuite: org.apache.cassandra.utils.binlog.BinLogTest
> [Junit-timeout] WARNING: An illegal reflective access operation has occurred
> [junit-timeout] WARNING: Illegal reflective access by
> net.openhft.chronicle.core.Jvm (file:/.../lib/chronicle-core-1.16.4.jar) to
> field java.nio.Bits.RESERVED_MEMORY
> [junit-timeout] WARNING: Please consider reporting this to the maintainers of
> net.openhft.chronicle.core.Jvm
> [junit-timeout] WARNING: Use --illegal-access=warn to enable warnings of
> further illegal reflective access operations
> [junit-timeout] WARNING: All illegal access operations will be denied in a
> future release
> [junit-timeout] Testsuite: org.apache.cassandra.utils.binlog.BinLogTest Tests
> run: 13, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 13.895 sec
> [junit-timeout]
> [junit-timeout] Testcase:
> testPutAfterStop(org.apache.cassandra.utils.binlog.BinLogTest): FAILED
> [junit-timeout] expected: but
> was:
> [junit-timeout] junit.framework.AssertionFailedError: expected: but
> was:
> [junit-timeout] at
> org.apache.cassandra.utils.binlog.BinLogTest.testPutAfterStop(BinLogTest.java:431)
> [junit-timeout] at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit-timeout] at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [junit-timeout] at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit-timeout]
> [junit-timeout]
> [junit-timeout] Test org.apache.cassandra.utils.binlog.BinLogTest FAILED
> {code}
> There's also a different failure under JDK8
> {code}
> junit-timeout] Testsuite: org.apache.cassandra.utils.binlog.BinLogTest
> [junit-timeout] Testsuite: org.apache.cassandra.utils.binlog.BinLogTest Tests
> run: 13, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.273 sec
> [junit-timeout]
> [junit-timeout] Testcase:
> testBinLogStartStop(org.apache.cassandra.utils.binlog.BinLogTest): FAILED
> [junit-timeout] expected:<2> but was:<0>
> [junit-timeout] junit.framework.AssertionFailedError: expected:<2> but was:<0>
> [junit-timeout] at
> org.apache.cassandra.utils.binlog.BinLogTest.testBinLogStartStop(BinLogTest.java:172)
> [junit-timeout]
> [junit-timeout]
> [junit-timeout] Test org.apache.cassandra.utils.binlog.BinLogTest FAILED
> {code}
> Reproducer
> {code}
> PASSED=0; time { while ant testclasslist -Dtest.classlistprefix=unit
> -Dtest.classlistfile=<(echo
> org/apache/cassandra/utils/binlog/BinLogTest.java); do PASSED=$((PASSED+1));
> echo PASSED $PASSED; done }; echo FAILED after $PASSED runs.
> {code}
> In the last four attempts it has taken 31, 38, 27 and 10 rounds respectively
> under JDK11 and took 51 under JDK8 (about 15 minutes).
> I have not tried running in a cpu-limited container or anything like that yet.
> Additionally, this went past in the logs a few times (under JDK11). No idea
> if it's just an artifact of weird test setup, or something more serious.
> {code}
> [junit-timeout] WARNING: Please consider reporting this to the maintainers of
> net.openhft.chronicle.core.Jvm
> [junit-timeout] WARNING: Use --illegal-access=warn to enable warnings of
> further illegal reflective access operations
> [junit-timeout] WARNING: All illegal access operations will be denied in a
> future release
> [junit-timeout] Testsuite: org.apache.cassandra.utils.binlog.BinLogTest
[jira] [Commented] (CASSANDRA-15797) Fix flaky BinLogTest - org.apache.cassandra.utils.binlog.BinLogTest
[
https://issues.apache.org/jira/browse/CASSANDRA-15797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112920#comment-17112920
]
Vinay Chella commented on CASSANDRA-15797:
--
Following dtests failed across java8 and java11, however these dtests are not
related to BingLogTest changes
- test_remote_query - cql_test.TestCQLSlowQuery
- test_local_query - cql_test.TestCQLSlowQuery
> Fix flaky BinLogTest - org.apache.cassandra.utils.binlog.BinLogTest
> ---
>
> Key: CASSANDRA-15797
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15797
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Jon Meredith
>Assignee: Yifan Cai
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> An internal CI system is failing BinLogTest somewhat frequently under JDK11.
> Configuration was recently changed to reduce the number of cores the tests
> run with, however it is reproducible on an 8 core laptop.
> {code}
> [junit-timeout] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC
> was deprecated in version 9.0 and will likely be removed in a future release.
> [junit-timeout] Testsuite: org.apache.cassandra.utils.binlog.BinLogTest
> [Junit-timeout] WARNING: An illegal reflective access operation has occurred
> [junit-timeout] WARNING: Illegal reflective access by
> net.openhft.chronicle.core.Jvm (file:/.../lib/chronicle-core-1.16.4.jar) to
> field java.nio.Bits.RESERVED_MEMORY
> [junit-timeout] WARNING: Please consider reporting this to the maintainers of
> net.openhft.chronicle.core.Jvm
> [junit-timeout] WARNING: Use --illegal-access=warn to enable warnings of
> further illegal reflective access operations
> [junit-timeout] WARNING: All illegal access operations will be denied in a
> future release
> [junit-timeout] Testsuite: org.apache.cassandra.utils.binlog.BinLogTest Tests
> run: 13, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 13.895 sec
> [junit-timeout]
> [junit-timeout] Testcase:
> testPutAfterStop(org.apache.cassandra.utils.binlog.BinLogTest): FAILED
> [junit-timeout] expected: but
> was:
> [junit-timeout] junit.framework.AssertionFailedError: expected: but
> was:
> [junit-timeout] at
> org.apache.cassandra.utils.binlog.BinLogTest.testPutAfterStop(BinLogTest.java:431)
> [junit-timeout] at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit-timeout] at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [junit-timeout] at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit-timeout]
> [junit-timeout]
> [junit-timeout] Test org.apache.cassandra.utils.binlog.BinLogTest FAILED
> {code}
> There's also a different failure under JDK8
> {code}
> junit-timeout] Testsuite: org.apache.cassandra.utils.binlog.BinLogTest
> [junit-timeout] Testsuite: org.apache.cassandra.utils.binlog.BinLogTest Tests
> run: 13, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.273 sec
> [junit-timeout]
> [junit-timeout] Testcase:
> testBinLogStartStop(org.apache.cassandra.utils.binlog.BinLogTest): FAILED
> [junit-timeout] expected:<2> but was:<0>
> [junit-timeout] junit.framework.AssertionFailedError: expected:<2> but was:<0>
> [junit-timeout] at
> org.apache.cassandra.utils.binlog.BinLogTest.testBinLogStartStop(BinLogTest.java:172)
> [junit-timeout]
> [junit-timeout]
> [junit-timeout] Test org.apache.cassandra.utils.binlog.BinLogTest FAILED
> {code}
> Reproducer
> {code}
> PASSED=0; time { while ant testclasslist -Dtest.classlistprefix=unit
> -Dtest.classlistfile=<(echo
> org/apache/cassandra/utils/binlog/BinLogTest.java); do PASSED=$((PASSED+1));
> echo PASSED $PASSED; done }; echo FAILED after $PASSED runs.
> {code}
> In the last four attempts it has taken 31, 38, 27 and 10 rounds respectively
> under JDK11 and took 51 under JDK8 (about 15 minutes).
> I have not tried running in a cpu-limited container or anything like that yet.
> Additionally, this went past in the logs a few times (under JDK11). No idea
> if it's just an artifact of weird test setup, or something more serious.
> {code}
> [junit-timeout] WARNING: Please consider reporting this to the maintainers of
> net.openhft.chronicle.core.Jvm
> [junit-timeout] WARNING: Use --illegal-access=warn to enable warnings of
> further illegal reflective access operations
> [junit-timeout] WARNING: All illegal access operations will be denied in a
> future release
> [junit-timeout] Testsuite: org.apache.cassandra.utils.binlog.BinLogTest Tests
> run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.839 sec
> [junit-timeout]
> [junit-timeout] java.lang.Throwable: 1e53135d-main creation
[cassandra] branch trunk updated: Fix flaky test o.a.c.u.binlog.BinLogTest patch by Yifan Cai; reviewed by Vinay Chella for CASSANDRA-15797
This is an automated email from the ASF dual-hosted git repository.
vinaychella pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new 43c1987 Fix flaky test o.a.c.u.binlog.BinLogTest patch by Yifan Cai;
reviewed by Vinay Chella for CASSANDRA-15797
43c1987 is described below
commit 43c19878e38fbe260f9e6143aa43836e85cf2f44
Author: yifan-c
AuthorDate: Thu May 21 00:56:52 2020 -0700
Fix flaky test o.a.c.u.binlog.BinLogTest
patch by Yifan Cai; reviewed by Vinay Chella for CASSANDRA-15797
---
.../apache/cassandra/utils/binlog/BinLogTest.java | 59 --
1 file changed, 21 insertions(+), 38 deletions(-)
diff --git a/test/unit/org/apache/cassandra/utils/binlog/BinLogTest.java
b/test/unit/org/apache/cassandra/utils/binlog/BinLogTest.java
index 9919af3..dbce6a9 100644
--- a/test/unit/org/apache/cassandra/utils/binlog/BinLogTest.java
+++ b/test/unit/org/apache/cassandra/utils/binlog/BinLogTest.java
@@ -23,8 +23,11 @@ import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
+import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Supplier;
import org.junit.After;
import org.junit.Before;
@@ -40,6 +43,7 @@ import org.apache.cassandra.io.util.FileUtils;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotEquals;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
@@ -115,9 +119,10 @@ public class BinLogTest
@Test
public void testBinLogStartStop() throws Exception
{
-Semaphore blockBinLog = new Semaphore(1);
AtomicInteger releaseCount = new AtomicInteger();
-binLog.put(new BinLog.ReleaseableWriteMarshallable()
+CountDownLatch ready = new CountDownLatch(2);
+Supplier recordSupplier =
+() -> new BinLog.ReleaseableWriteMarshallable()
{
public void release()
{
@@ -136,40 +141,13 @@ public class BinLogTest
public void writeMarshallablePayload(WireOut wire)
{
-try
-{
-blockBinLog.acquire();
-}
-catch (InterruptedException e)
-{
-throw new RuntimeException(e);
-}
-}
-});
-binLog.put(new BinLog.ReleaseableWriteMarshallable()
-{
-protected long version()
-{
-return 0;
-}
-
-protected String type()
-{
-return "test";
+ready.countDown();
}
-
-public void writeMarshallablePayload(WireOut wire)
-{
-
-}
-
-public void release()
-{
-releaseCount.incrementAndGet();
-}
-});
-Thread.sleep(1000);
-assertEquals(2, releaseCount.get());
+};
+binLog.put(recordSupplier.get());
+binLog.put(recordSupplier.get());
+ready.await(1, TimeUnit.MINUTES);
+Util.spinAssertEquals("Both records should be released", 2,
releaseCount::get, 10, TimeUnit.SECONDS);
Thread t = new Thread(() -> {
try
{
@@ -182,7 +160,7 @@ public class BinLogTest
});
t.start();
t.join(60 * 1000);
-assertEquals(t.getState(), Thread.State.TERMINATED);
+assertEquals("BinLog should not take more than 1 minute to stop",
t.getState(), Thread.State.TERMINATED);
Util.spinAssertEquals(2, releaseCount::get, 60);
Util.spinAssertEquals(Thread.State.TERMINATED,
binLog.binLogThread::getState, 60);
@@ -426,9 +404,14 @@ public class BinLogTest
@Test
public void testPutAfterStop() throws Exception
{
+final BinLog.ReleaseableWriteMarshallable unexpected =
record(testString);
binLog.stop();
-binLog.put(record(testString));
-assertEquals(null, binLog.sampleQueue.poll());
+binLog.put(unexpected);
+BinLog.ReleaseableWriteMarshallable record;
+while (null != (record = binLog.sampleQueue.poll()))
+{
+assertNotEquals("A stopped BinLog should no longer accept",
unexpected, record);
+}
}
/**
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15759) Add progress column in percents for system_views.sstable_tasks
[
https://issues.apache.org/jira/browse/CASSANDRA-15759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112901#comment-17112901
]
Stefan Miklosovic commented on CASSANDRA-15759:
---
[~aleksey] any chance to merge this one please? Its a low-hanger pretty much ...
> Add progress column in percents for system_views.sstable_tasks
> --
>
> Key: CASSANDRA-15759
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15759
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/Virtual Tables
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It would be very handy to have a percentage column in
> system_views.sstable_tasks which would say how far a respective task is.
> Indeed, there are currently "progress" and "total" columns but honestly, for
> every day usage, it is rather strange to expect that humans will divide these
> numbers in head if they want to roughly know what the overall progress is.
> One just does not have a rough estimation of the task progress when he is
> presented with two quite big numbers and to estimate the progress from the.
> In the following output, the field "progress_in_percents" is introduced.
>
> PR is here [https://github.com/apache/cassandra/pull/566]
>
> {code:java}
> admin@cqlsh> select * from system_views.sstable_tasks ;
> @ Row 1
> ---+--
> keyspace_name | mykeyspace
> table_name| mytable
> task_id | 0db5d9b1-8726-11ea-8a6c-b92f3be367bb
> kind | secondary index build
> progress | 19456965
> progress_in_percents | 8.17
> total | 238208674
> unit | bytes
> @ Row 2
> --+--
> keyspace_name| mykeyspace
> table_name | mytable.mytable_surname_idx
> task_id | 1817ee71-8726-11ea-8a6c-b92f3be367bb
> kind | compaction
> progress | 284396233
> progress_in_percents | 75.92
> total| 374598446
> unit | bytes
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15406) Add command to show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112900#comment-17112900 ] Stefan Miklosovic commented on CASSANDRA-15406: --- [~djoshi] any chance to merge this one? > Add command to show the progress of data streaming and index build > --- > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0, 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . > > PR [https://github.com/apache/cassandra/pull/558] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15748) Provide ability to configure IAuditLogger
[ https://issues.apache.org/jira/browse/CASSANDRA-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112870#comment-17112870 ] Vinay Chella commented on CASSANDRA-15748: -- Thank you [~stefan.miklosovic] for the patch, [~marcuse] for committing it. > Provide ability to configure IAuditLogger > - > > Key: CASSANDRA-15748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15748 > Project: Cassandra > Issue Type: Improvement > Components: Local/Other >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0-beta > > Time Spent: 50m > Remaining Estimate: 0h > > Currently there is a parameterless constructor expected for IAuditLogger > instances. There is not any way how to "inject" configuration properties from > cassandra.yaml into these concrete classes. This would be very handy for > custom IAuditLogger implementations. > > PR: [https://github.com/apache/cassandra/pull/555] > [|https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15748] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15797) Fix flaky BinLogTest - org.apache.cassandra.utils.binlog.BinLogTest
[
https://issues.apache.org/jira/browse/CASSANDRA-15797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112860#comment-17112860
]
Vinay Chella commented on CASSANDRA-15797:
--
circle ci tests:
https://app.circleci.com/pipelines/github/vinaykumarchella/cassandra?branch=15797-trunk
> Fix flaky BinLogTest - org.apache.cassandra.utils.binlog.BinLogTest
> ---
>
> Key: CASSANDRA-15797
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15797
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Jon Meredith
>Assignee: Yifan Cai
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> An internal CI system is failing BinLogTest somewhat frequently under JDK11.
> Configuration was recently changed to reduce the number of cores the tests
> run with, however it is reproducible on an 8 core laptop.
> {code}
> [junit-timeout] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC
> was deprecated in version 9.0 and will likely be removed in a future release.
> [junit-timeout] Testsuite: org.apache.cassandra.utils.binlog.BinLogTest
> [Junit-timeout] WARNING: An illegal reflective access operation has occurred
> [junit-timeout] WARNING: Illegal reflective access by
> net.openhft.chronicle.core.Jvm (file:/.../lib/chronicle-core-1.16.4.jar) to
> field java.nio.Bits.RESERVED_MEMORY
> [junit-timeout] WARNING: Please consider reporting this to the maintainers of
> net.openhft.chronicle.core.Jvm
> [junit-timeout] WARNING: Use --illegal-access=warn to enable warnings of
> further illegal reflective access operations
> [junit-timeout] WARNING: All illegal access operations will be denied in a
> future release
> [junit-timeout] Testsuite: org.apache.cassandra.utils.binlog.BinLogTest Tests
> run: 13, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 13.895 sec
> [junit-timeout]
> [junit-timeout] Testcase:
> testPutAfterStop(org.apache.cassandra.utils.binlog.BinLogTest): FAILED
> [junit-timeout] expected: but
> was:
> [junit-timeout] junit.framework.AssertionFailedError: expected: but
> was:
> [junit-timeout] at
> org.apache.cassandra.utils.binlog.BinLogTest.testPutAfterStop(BinLogTest.java:431)
> [junit-timeout] at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit-timeout] at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [junit-timeout] at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit-timeout]
> [junit-timeout]
> [junit-timeout] Test org.apache.cassandra.utils.binlog.BinLogTest FAILED
> {code}
> There's also a different failure under JDK8
> {code}
> junit-timeout] Testsuite: org.apache.cassandra.utils.binlog.BinLogTest
> [junit-timeout] Testsuite: org.apache.cassandra.utils.binlog.BinLogTest Tests
> run: 13, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.273 sec
> [junit-timeout]
> [junit-timeout] Testcase:
> testBinLogStartStop(org.apache.cassandra.utils.binlog.BinLogTest): FAILED
> [junit-timeout] expected:<2> but was:<0>
> [junit-timeout] junit.framework.AssertionFailedError: expected:<2> but was:<0>
> [junit-timeout] at
> org.apache.cassandra.utils.binlog.BinLogTest.testBinLogStartStop(BinLogTest.java:172)
> [junit-timeout]
> [junit-timeout]
> [junit-timeout] Test org.apache.cassandra.utils.binlog.BinLogTest FAILED
> {code}
> Reproducer
> {code}
> PASSED=0; time { while ant testclasslist -Dtest.classlistprefix=unit
> -Dtest.classlistfile=<(echo
> org/apache/cassandra/utils/binlog/BinLogTest.java); do PASSED=$((PASSED+1));
> echo PASSED $PASSED; done }; echo FAILED after $PASSED runs.
> {code}
> In the last four attempts it has taken 31, 38, 27 and 10 rounds respectively
> under JDK11 and took 51 under JDK8 (about 15 minutes).
> I have not tried running in a cpu-limited container or anything like that yet.
> Additionally, this went past in the logs a few times (under JDK11). No idea
> if it's just an artifact of weird test setup, or something more serious.
> {code}
> [junit-timeout] WARNING: Please consider reporting this to the maintainers of
> net.openhft.chronicle.core.Jvm
> [junit-timeout] WARNING: Use --illegal-access=warn to enable warnings of
> further illegal reflective access operations
> [junit-timeout] WARNING: All illegal access operations will be denied in a
> future release
> [junit-timeout] Testsuite: org.apache.cassandra.utils.binlog.BinLogTest Tests
> run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.839 sec
> [junit-timeout]
> [junit-timeout] java.lang.Throwable: 1e53135d-main creation ref-count=1
> [junit-timeout] at
>
