[jira] [Commented] (CASSANDRA-15763) Tunable RangeTombstoneList initial size and resize factor
[ https://issues.apache.org/jira/browse/CASSANDRA-15763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099561#comment-17099561 ] Yifan Cai commented on CASSANDRA-15763: --- Hi [~djoshi], making the initial size and the growth factor tunable is especially useful in the case when getting many deletes for a single partition. With a larger initial size and growth factor, it helps to reduce the cost of frequent array resizing/allocation underneath. > Tunable RangeTombstoneList initial size and resize factor > - > > Key: CASSANDRA-15763 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15763 > Project: Cassandra > Issue Type: Improvement > Components: Local/Config >Reporter: Yifan Cai >Assignee: Yifan Cai >Priority: Normal > > Cassandra allocates a backing array with a fixed size of one when creating a > new RangeTombstoneList object. The resize factor is fixed to 1.5. > The initial size and resize factor could be made tunable so it could reduce > cost in some cases. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15729) Jenkins Test Results Report in plaintext for ASF ML
[ https://issues.apache.org/jira/browse/CASSANDRA-15729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-15729: --- Status: Patch Available (was: In Progress) > Jenkins Test Results Report in plaintext for ASF ML > --- > > Key: CASSANDRA-15729 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15729 > Project: Cassandra > Issue Type: Task > Components: Build, CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Labels: Jenkins > Fix For: 4.0-beta > > Attachments: .Screenshot 2020-05-01 at 01.12.28.png > > > The Jenkins pipeline builds now aggregate all test reports. > For example: > - https://ci-cassandra.apache.org/job/Cassandra-trunk/68/testReport/ > - > https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-trunk/detail/Cassandra-trunk/68/tests > But Jenkins can only keep a limited amount of build history, so those links > are not permanent, can't be used as references, and don't help for bisecting > and blame on regressions (and flakey tests) over a longer period of time. > The builds@ ML can provide a permanent record of test results. > This was first brought up in these two threads: > - > https://lists.apache.org/thread.html/re8122e4fdd8629e7fbca2abf27d72054b3bc0e3690ece8b8e66f618b%40%3Cdev.cassandra.apache.org%3E > - > https://lists.apache.org/thread.html/ra5f6aeea89546825fe7ccc4a80898c62f8ed57decabf709d81d9c720%40%3Cdev.cassandra.apache.org%3E > An example plaintext report, to demonstrate feasibility, is available here: > https://lists.apache.org/thread.html/r80d13f7af706bf8dfbf2387fab46004c1fbd3917b7bc339c49e69aa8%40%3Cbuilds.cassandra.apache.org%3E > Hurdles: > - the ASF mailing lists won't accept html, attachments, or any message body > over 1MB. > - packages are used as a differentiator in the final aggregated report. The > cqlsh and dtests currently don't specify it. It needs to be added as a > "dot-separated" prefix to the testsuite and testcase name. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15764) Optimize logging with lazy log parameter evaluation
[ https://issues.apache.org/jira/browse/CASSANDRA-15764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yifan Cai updated CASSANDRA-15764: -- Resolution: Duplicate Status: Resolved (was: Open) > Optimize logging with lazy log parameter evaluation > --- > > Key: CASSANDRA-15764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15764 > Project: Cassandra > Issue Type: Improvement > Components: Observability/Logging >Reporter: Yifan Cai >Assignee: Yifan Cai >Priority: Normal > > Multiple logging statements in the source evaluate methods and use the return > values as log parameter. > It does no harm when the log level permits for the statement. > However, for the logs that are permitted, the evaluation is a wastes of CPU. > The log parameters are not needed, but still being evaluated! > For such cases, lazy evaluation (by leveraging lambda) can be introduced to > skip competing the string parameter. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15782) Compression test failure
[ https://issues.apache.org/jira/browse/CASSANDRA-15782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Berenguer Blasi updated CASSANDRA-15782: Status: Ready to Commit (was: Review In Progress) > Compression test failure > > > Key: CASSANDRA-15782 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15782 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: Berenguer Blasi >Assignee: Joey Lynch >Priority: Normal > Fix For: 4.0, 4.0-beta > > > On CASSANDRA-15560 compression test failed. This was bisected to > [9c1bbf3|https://github.com/apache/cassandra/commit/9c1bbf3ac913f9bdf7a0e0922106804af42d2c1e] > from CASSANDRA-15379. > Full details here > CC/ [~jolynch] in case he can spot it quick. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15782) Compression test failure
[ https://issues.apache.org/jira/browse/CASSANDRA-15782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099616#comment-17099616 ] Berenguer Blasi edited comment on CASSANDRA-15782 at 5/5/20, 7:50 AM: -- Wow that was fast :-) LGTM If a compaction is needed to make that deflation effective +1 was (Author: bereng): LGTM If a compaction is needed to make that deflation effective +1 > Compression test failure > > > Key: CASSANDRA-15782 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15782 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: Berenguer Blasi >Assignee: Joey Lynch >Priority: Normal > Fix For: 4.0, 4.0-beta > > > On CASSANDRA-15560 compression test failed. This was bisected to > [9c1bbf3|https://github.com/apache/cassandra/commit/9c1bbf3ac913f9bdf7a0e0922106804af42d2c1e] > from CASSANDRA-15379. > Full details here > CC/ [~jolynch] in case he can spot it quick. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15786) ChecksummingTransformerTest#corruptionCausesFailure fails for seed 43595190254702
[
https://issues.apache.org/jira/browse/CASSANDRA-15786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Stupp updated CASSANDRA-15786:
-
Resolution: Duplicate
Status: Resolved (was: Triage Needed)
> ChecksummingTransformerTest#corruptionCausesFailure fails for seed
> 43595190254702
> -
>
> Key: CASSANDRA-15786
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15786
> Project: Cassandra
> Issue Type: Bug
>Reporter: Robert Stupp
>Priority: Normal
> Fix For: 4.0
>
>
> Reproducer:
> * latest trunk (e.g. df8e736700ae2a06675ff50381788d708bc22b96)
> * change 1st line in {{corruptionCausesFailure()}} to
> {{qt().withFixedSeed(43595190254702L)}}
> * run test
> Fails with:
> {code:java}
> java.lang.AssertionError: Property falsified after 617 example(s)
> Smallest found falsifying value(s) :-
>
[jira] [Updated] (CASSANDRA-15782) Compression test failure
[ https://issues.apache.org/jira/browse/CASSANDRA-15782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Berenguer Blasi updated CASSANDRA-15782: Test and Documentation Plan: See comments Status: Patch Available (was: In Progress) > Compression test failure > > > Key: CASSANDRA-15782 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15782 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: Berenguer Blasi >Assignee: Joey Lynch >Priority: Normal > Fix For: 4.0, 4.0-beta > > > On CASSANDRA-15560 compression test failed. This was bisected to > [9c1bbf3|https://github.com/apache/cassandra/commit/9c1bbf3ac913f9bdf7a0e0922106804af42d2c1e] > from CASSANDRA-15379. > Full details here > CC/ [~jolynch] in case he can spot it quick. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15782) Compression test failure
[ https://issues.apache.org/jira/browse/CASSANDRA-15782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Berenguer Blasi updated CASSANDRA-15782: Reviewers: Berenguer Blasi > Compression test failure > > > Key: CASSANDRA-15782 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15782 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: Berenguer Blasi >Assignee: Joey Lynch >Priority: Normal > Fix For: 4.0, 4.0-beta > > > On CASSANDRA-15560 compression test failed. This was bisected to > [9c1bbf3|https://github.com/apache/cassandra/commit/9c1bbf3ac913f9bdf7a0e0922106804af42d2c1e] > from CASSANDRA-15379. > Full details here > CC/ [~jolynch] in case he can spot it quick. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15729) Jenkins Test Results Report in plaintext for ASF ML
[
https://issues.apache.org/jira/browse/CASSANDRA-15729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099583#comment-17099583
]
Michael Semb Wever commented on CASSANDRA-15729:
Emailing the plaintext report has been tested against the
{{Cassandra-devbranch}}. See
https://lists.apache.org/thread.html/raf2038993e87a9540d74469a378167134ef026b130a013e6cda2c042%40%3Cbuilds.cassandra.apache.org%3E
bq. The remaining work in this ticket is to make the in-tree Jenkinsfile to do
the same
The patches for this are in
-
[cassandra-2.2_15729|https://github.com/apache/cassandra/compare/cassandra-2.2...thelastpickle:mck/cassandra-2.2_15729]
-
[cassandra-3.0_15729|https://github.com/apache/cassandra/compare/cassandra-3.0...thelastpickle:mck/cassandra-3.0_15729]
-
[cassandra-3.11_15729|https://github.com/apache/cassandra/compare/cassandra-3.11...thelastpickle:mck/cassandra-3.11_15729]
-
[trunk_15729|https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/trunk_15729]
> Jenkins Test Results Report in plaintext for ASF ML
> ---
>
> Key: CASSANDRA-15729
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15729
> Project: Cassandra
> Issue Type: Task
> Components: Build, CI
>Reporter: Michael Semb Wever
>Assignee: Michael Semb Wever
>Priority: Normal
> Labels: Jenkins
> Fix For: 4.0-beta
>
> Attachments: .Screenshot 2020-05-01 at 01.12.28.png
>
>
> The Jenkins pipeline builds now aggregate all test reports.
> For example:
> - https://ci-cassandra.apache.org/job/Cassandra-trunk/68/testReport/
> -
> https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-trunk/detail/Cassandra-trunk/68/tests
> But Jenkins can only keep a limited amount of build history, so those links
> are not permanent, can't be used as references, and don't help for bisecting
> and blame on regressions (and flakey tests) over a longer period of time.
> The builds@ ML can provide a permanent record of test results.
> This was first brought up in these two threads:
> -
> https://lists.apache.org/thread.html/re8122e4fdd8629e7fbca2abf27d72054b3bc0e3690ece8b8e66f618b%40%3Cdev.cassandra.apache.org%3E
> -
> https://lists.apache.org/thread.html/ra5f6aeea89546825fe7ccc4a80898c62f8ed57decabf709d81d9c720%40%3Cdev.cassandra.apache.org%3E
> An example plaintext report, to demonstrate feasibility, is available here:
> https://lists.apache.org/thread.html/r80d13f7af706bf8dfbf2387fab46004c1fbd3917b7bc339c49e69aa8%40%3Cbuilds.cassandra.apache.org%3E
> Hurdles:
> - the ASF mailing lists won't accept html, attachments, or any message body
> over 1MB.
> - packages are used as a differentiator in the final aggregated report. The
> cqlsh and dtests currently don't specify it. It needs to be added as a
> "dot-separated" prefix to the testsuite and testcase name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15214) OOMs caught and not rethrown
[
https://issues.apache.org/jira/browse/CASSANDRA-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099582#comment-17099582
]
Yifan Cai commented on CASSANDRA-15214:
---
Thanks [~jolynch] for the update.
{quote}force the JVM into a "normal" OOM by [allocating large long
arrays|https://github.com/Netflix-Skunkworks/jvmquake/blob/master/src/jvmquake.c#L103]
{quote}
An alternative way could be programmatically grab the heap dump via
[JMX|https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/jdk.management/share/classes/com/sun/management/HotSpotDiagnosticMXBean.java#L75]
and exit.
> OOMs caught and not rethrown
>
>
> Key: CASSANDRA-15214
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15214
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Client, Messaging/Internode
>Reporter: Benedict Elliott Smith
>Priority: Normal
> Fix For: 4.0, 4.0-rc
>
> Attachments: oom-experiments.zip
>
>
> Netty (at least, and perhaps elsewhere in Executors) catches all exceptions,
> so presently there is no way to ensure that an OOM reaches the JVM handler to
> trigger a crash/heapdump.
> It may be that the simplest most consistent way to do this would be to have a
> single thread spawned at startup that waits for any exceptions we must
> propagate to the Runtime.
> We could probably submit a patch upstream to Netty, but for a guaranteed
> future proof approach, it may be worth paying the cost of a single thread.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15729) Jenkins Test Results Report in plaintext for ASF ML
[ https://issues.apache.org/jira/browse/CASSANDRA-15729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099584#comment-17099584 ] Michael Semb Wever commented on CASSANDRA-15729: bq. The remaining work in this ticket is to make the in-tree Jenkinsfile to do the same … The patches for this are ready for review in - [cassandra-2.2_15729|https://github.com/apache/cassandra/compare/cassandra-2.2...thelastpickle:mck/cassandra-2.2_15729] - [cassandra-3.0_15729|https://github.com/apache/cassandra/compare/cassandra-3.0...thelastpickle:mck/cassandra-3.0_15729] - [cassandra-3.11_15729|https://github.com/apache/cassandra/compare/cassandra-3.11...thelastpickle:mck/cassandra-3.11_15729] - [trunk_15729|https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/trunk_15729] > Jenkins Test Results Report in plaintext for ASF ML > --- > > Key: CASSANDRA-15729 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15729 > Project: Cassandra > Issue Type: Task > Components: Build, CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Labels: Jenkins > Fix For: 4.0-beta > > Attachments: .Screenshot 2020-05-01 at 01.12.28.png > > > The Jenkins pipeline builds now aggregate all test reports. > For example: > - https://ci-cassandra.apache.org/job/Cassandra-trunk/68/testReport/ > - > https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-trunk/detail/Cassandra-trunk/68/tests > But Jenkins can only keep a limited amount of build history, so those links > are not permanent, can't be used as references, and don't help for bisecting > and blame on regressions (and flakey tests) over a longer period of time. > The builds@ ML can provide a permanent record of test results. > This was first brought up in these two threads: > - > https://lists.apache.org/thread.html/re8122e4fdd8629e7fbca2abf27d72054b3bc0e3690ece8b8e66f618b%40%3Cdev.cassandra.apache.org%3E > - > https://lists.apache.org/thread.html/ra5f6aeea89546825fe7ccc4a80898c62f8ed57decabf709d81d9c720%40%3Cdev.cassandra.apache.org%3E > An example plaintext report, to demonstrate feasibility, is available here: > https://lists.apache.org/thread.html/r80d13f7af706bf8dfbf2387fab46004c1fbd3917b7bc339c49e69aa8%40%3Cbuilds.cassandra.apache.org%3E > Hurdles: > - the ASF mailing lists won't accept html, attachments, or any message body > over 1MB. > - packages are used as a differentiator in the final aggregated report. The > cqlsh and dtests currently don't specify it. It needs to be added as a > "dot-separated" prefix to the testsuite and testcase name. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15729) Jenkins Test Results Report in plaintext for ASF ML
[
https://issues.apache.org/jira/browse/CASSANDRA-15729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099583#comment-17099583
]
Michael Semb Wever edited comment on CASSANDRA-15729 at 5/5/20, 7:07 AM:
-
Emailing the plaintext report has been tested against the
{{Cassandra-devbranch}}. See
https://lists.apache.org/thread.html/raf2038993e87a9540d74469a378167134ef026b130a013e6cda2c042%40%3Cbuilds.cassandra.apache.org%3E
was (Author: michaelsembwever):
Emailing the plaintext report has been tested against the
{{Cassandra-devbranch}}. See
https://lists.apache.org/thread.html/raf2038993e87a9540d74469a378167134ef026b130a013e6cda2c042%40%3Cbuilds.cassandra.apache.org%3E
bq. The remaining work in this ticket is to make the in-tree Jenkinsfile to do
the same
The patches for this are in
-
[cassandra-2.2_15729|https://github.com/apache/cassandra/compare/cassandra-2.2...thelastpickle:mck/cassandra-2.2_15729]
-
[cassandra-3.0_15729|https://github.com/apache/cassandra/compare/cassandra-3.0...thelastpickle:mck/cassandra-3.0_15729]
-
[cassandra-3.11_15729|https://github.com/apache/cassandra/compare/cassandra-3.11...thelastpickle:mck/cassandra-3.11_15729]
-
[trunk_15729|https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/trunk_15729]
> Jenkins Test Results Report in plaintext for ASF ML
> ---
>
> Key: CASSANDRA-15729
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15729
> Project: Cassandra
> Issue Type: Task
> Components: Build, CI
>Reporter: Michael Semb Wever
>Assignee: Michael Semb Wever
>Priority: Normal
> Labels: Jenkins
> Fix For: 4.0-beta
>
> Attachments: .Screenshot 2020-05-01 at 01.12.28.png
>
>
> The Jenkins pipeline builds now aggregate all test reports.
> For example:
> - https://ci-cassandra.apache.org/job/Cassandra-trunk/68/testReport/
> -
> https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-trunk/detail/Cassandra-trunk/68/tests
> But Jenkins can only keep a limited amount of build history, so those links
> are not permanent, can't be used as references, and don't help for bisecting
> and blame on regressions (and flakey tests) over a longer period of time.
> The builds@ ML can provide a permanent record of test results.
> This was first brought up in these two threads:
> -
> https://lists.apache.org/thread.html/re8122e4fdd8629e7fbca2abf27d72054b3bc0e3690ece8b8e66f618b%40%3Cdev.cassandra.apache.org%3E
> -
> https://lists.apache.org/thread.html/ra5f6aeea89546825fe7ccc4a80898c62f8ed57decabf709d81d9c720%40%3Cdev.cassandra.apache.org%3E
> An example plaintext report, to demonstrate feasibility, is available here:
> https://lists.apache.org/thread.html/r80d13f7af706bf8dfbf2387fab46004c1fbd3917b7bc339c49e69aa8%40%3Cbuilds.cassandra.apache.org%3E
> Hurdles:
> - the ASF mailing lists won't accept html, attachments, or any message body
> over 1MB.
> - packages are used as a differentiator in the final aggregated report. The
> cqlsh and dtests currently don't specify it. It needs to be added as a
> "dot-separated" prefix to the testsuite and testcase name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15766) NoSpamLogger arguments building objects on hot paths
[
https://issues.apache.org/jira/browse/CASSANDRA-15766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099605#comment-17099605
]
Yifan Cai commented on CASSANDRA-15766:
---
I have only done a quick glance at the patch.
Essentially, it wraps any expensive computation within the lambda, so in the
case of the log level not permits, no CPU cycles are wasted on computing the
log arguments that are not going to be used. The trade-off is the wrapper,
{{Supplier}}.
Whether it is worthy or not. It depends on how expensive the unnecessary
argument computation is. The computation itself may produce multiple
intermediate objects.
It would be great to have a benchmark to determine the how the lazy logging
helps in the scenarios of 1) no computation, 2) light computation, 3) moderate
computation and 4) heavy computation. So it serves the purpose of guidance on
when to invoke the lazy logging and when to not.
This patch only applies to the {{NoSpamLogger}}. There are other logging
statements that use normal logger in the code base that may have expensive
argument computation. If the lazy impl is proven to have performance benefits,
how about applying to those if any?
The patch does what is wanted in CASSANDRA-15764. I will mark CASSANDRA-15764
as a dup and close once this one is completed.
> NoSpamLogger arguments building objects on hot paths
>
>
> Key: CASSANDRA-15766
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15766
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Logging
>Reporter: Jon Meredith
>Assignee: Jon Meredith
>Priority: Normal
> Fix For: 4.0-rc
>
>
> NoSpamLogger is used in hot logging paths to prevent logs being overrun. For
> that to be most effective the arguments to the logger need to be cheap to
> construct. During the internode messaging refactor CASSANDRA-15066,
> performance changes to BufferPool for CASSANDRA-14416
> were accidentally reverted in the merge up from 3.11.
> Reviewing other uses since, it looks like there are a few places where the
> arguments require some form of String building.
> org.apache.cassandra.net.InboundSink#accept
> org.apache.cassandra.net.InboundMessageHandler#processCorruptFrame
> org.apache.cassandra.net.InboundMessageHandler.LargeMessage#deserialize
> org.apache.cassandra.net.OutboundConnection#onOverloaded
> org.apache.cassandra.utils.memory.BufferPool.GlobalPool#allocateMoreChunks
> Formatting arguments should either be precomputed, or if expensive they
> should be computed after the decision on whether to noSpamLog has been made.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15764) Optimize logging with lazy log parameter evaluation
[ https://issues.apache.org/jira/browse/CASSANDRA-15764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yifan Cai updated CASSANDRA-15764: -- Status: Open (was: Resolved) > Optimize logging with lazy log parameter evaluation > --- > > Key: CASSANDRA-15764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15764 > Project: Cassandra > Issue Type: Improvement > Components: Observability/Logging >Reporter: Yifan Cai >Assignee: Yifan Cai >Priority: Normal > > Multiple logging statements in the source evaluate methods and use the return > values as log parameter. > It does no harm when the log level permits for the statement. > However, for the logs that are permitted, the evaluation is a wastes of CPU. > The log parameters are not needed, but still being evaluated! > For such cases, lazy evaluation (by leveraging lambda) can be introduced to > skip competing the string parameter. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15782) Compression test failure
[ https://issues.apache.org/jira/browse/CASSANDRA-15782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Berenguer Blasi updated CASSANDRA-15782: Reviewers: Berenguer Blasi, Berenguer Blasi (was: Berenguer Blasi) Berenguer Blasi, Berenguer Blasi (was: Berenguer Blasi) Status: Review In Progress (was: Patch Available) > Compression test failure > > > Key: CASSANDRA-15782 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15782 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: Berenguer Blasi >Assignee: Joey Lynch >Priority: Normal > Fix For: 4.0, 4.0-beta > > > On CASSANDRA-15560 compression test failed. This was bisected to > [9c1bbf3|https://github.com/apache/cassandra/commit/9c1bbf3ac913f9bdf7a0e0922106804af42d2c1e] > from CASSANDRA-15379. > Full details here > CC/ [~jolynch] in case he can spot it quick. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15782) Compression test failure
[ https://issues.apache.org/jira/browse/CASSANDRA-15782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099616#comment-17099616 ] Berenguer Blasi commented on CASSANDRA-15782: - LGTM If a compaction is needed to make that deflation effective +1 > Compression test failure > > > Key: CASSANDRA-15782 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15782 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: Berenguer Blasi >Assignee: Joey Lynch >Priority: Normal > Fix For: 4.0, 4.0-beta > > > On CASSANDRA-15560 compression test failed. This was bisected to > [9c1bbf3|https://github.com/apache/cassandra/commit/9c1bbf3ac913f9bdf7a0e0922106804af42d2c1e] > from CASSANDRA-15379. > Full details here > CC/ [~jolynch] in case he can spot it quick. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15766) NoSpamLogger arguments building objects on hot paths
[
https://issues.apache.org/jira/browse/CASSANDRA-15766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099691#comment-17099691
]
Berenguer Blasi commented on CASSANDRA-15766:
-
Great, Jira lost my comment. Here we go again...
I am a newbie here, so feel free to ignore me:
* I wonder why we don't defend with a {{wrapped.isLevelEnabled()}} at
{{NoSpamLogger}} top methods. It might pay off given we'd spare the many nested
method calls, the {{getStatement()}}, {{nanoTime()}}, {{shouldLog()},... calls
* The {{Supplier}} approach is very nice, I like it a lot.
** If the overhead is negligible I would make that the only option. But given
my, maybe outdated, experiences with streams, lambdas, etc I am going to bet it
will be not :shrug:
** If it were not I'd rename top level methods as {{warnWithLazyParams()}} and
{{warnWithoutLazyParams()}}. If the dev is educated enough to have opted for
{{NoSpamLogger}} sure he will make the right choice here. I don't think we can
infer at call time which option is best, only the dev can.
+1 to a general 'other logs audit'. I would do it in another ticket as they
won't be in the hot path, they should be {{NoSpamLogger}} if they were, so that
is not as 'urgent' as this one imo.
Hope it makes sense. My 2cts
> NoSpamLogger arguments building objects on hot paths
>
>
> Key: CASSANDRA-15766
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15766
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Logging
>Reporter: Jon Meredith
>Assignee: Jon Meredith
>Priority: Normal
> Fix For: 4.0-rc
>
>
> NoSpamLogger is used in hot logging paths to prevent logs being overrun. For
> that to be most effective the arguments to the logger need to be cheap to
> construct. During the internode messaging refactor CASSANDRA-15066,
> performance changes to BufferPool for CASSANDRA-14416
> were accidentally reverted in the merge up from 3.11.
> Reviewing other uses since, it looks like there are a few places where the
> arguments require some form of String building.
> org.apache.cassandra.net.InboundSink#accept
> org.apache.cassandra.net.InboundMessageHandler#processCorruptFrame
> org.apache.cassandra.net.InboundMessageHandler.LargeMessage#deserialize
> org.apache.cassandra.net.OutboundConnection#onOverloaded
> org.apache.cassandra.utils.memory.BufferPool.GlobalPool#allocateMoreChunks
> Formatting arguments should either be precomputed, or if expensive they
> should be computed after the decision on whether to noSpamLog has been made.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13606) Improve handling of 2i initialization failures
[
https://issues.apache.org/jira/browse/CASSANDRA-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099695#comment-17099695
]
Berenguer Blasi commented on CASSANDRA-13606:
-
[~adelapena] thx for reviewing this. I also was wondering about what a 'failed
built' index should do regarding writes. On reads it simply fails so I did the
same for writes. Yes it is very restrictive.
The alternative would be to ignore those writes as you state. What about
instead calling a {{writeRequestedWhileIndexFailed()}} method on the index
itself? That would allow the implementation to count, noSpamLog them, throw an
exception or whatever. wdyt?
> Improve handling of 2i initialization failures
> --
>
> Key: CASSANDRA-13606
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13606
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/2i Index
>Reporter: Sergio Bossa
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> CASSANDRA-10130 fixes the 2i build management, but initialization failures
> are still not properly handled, most notably because:
> * Initialization failures make the index non-queryable, but it can still be
> written to.
> * Initialization failures can be recovered via full rebuilds.
> Both points above are probably suboptimal because the initialization logic
> could be more complex than just an index build, hence it shouldn't be made
> recoverable via a simple rebuild, and could cause the index to be fully
> unavailable not just for reads, but for writes as well.
> So, we should better handle initialization failures by:
> * Allowing the index implementation to specify if unavailable for reads,
> writes, or both.
> * Providing a proper method to recover, distinct from index rebuilds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15766) NoSpamLogger arguments building objects on hot paths
[
https://issues.apache.org/jira/browse/CASSANDRA-15766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099731#comment-17099731
]
Benedict Elliott Smith commented on CASSANDRA-15766:
Couldn't this be kept a bit more idiomatic with the provision of an object with
a {{toString()}} method that invokes {{id()}} ? We could even have such an
object instantiated once per {{InboundMessageHandler}} and once per
{{BufferPool}} so that this is garbage-free for most cases, and continues to
read like a normal logging call?
I've in the past considered introducing a special interface declaring only
{{toString()}} for declaring these via lambdas for each parameter, which might
be a nice way to do this for {{InboundSink}}.
A secondary API might be to provide method parameter options that accept pairs
of (param, function) so that e.g. {{InboundSink}} can provide (t,
{{Throwable::getMessage)}}, and if we wanted {{InboundMessageHandler}} could
provide {{(this, InboundMessageHandler::id)}}
> NoSpamLogger arguments building objects on hot paths
>
>
> Key: CASSANDRA-15766
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15766
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Logging
>Reporter: Jon Meredith
>Assignee: Jon Meredith
>Priority: Normal
> Fix For: 4.0-rc
>
>
> NoSpamLogger is used in hot logging paths to prevent logs being overrun. For
> that to be most effective the arguments to the logger need to be cheap to
> construct. During the internode messaging refactor CASSANDRA-15066,
> performance changes to BufferPool for CASSANDRA-14416
> were accidentally reverted in the merge up from 3.11.
> Reviewing other uses since, it looks like there are a few places where the
> arguments require some form of String building.
> org.apache.cassandra.net.InboundSink#accept
> org.apache.cassandra.net.InboundMessageHandler#processCorruptFrame
> org.apache.cassandra.net.InboundMessageHandler.LargeMessage#deserialize
> org.apache.cassandra.net.OutboundConnection#onOverloaded
> org.apache.cassandra.utils.memory.BufferPool.GlobalPool#allocateMoreChunks
> Formatting arguments should either be precomputed, or if expensive they
> should be computed after the decision on whether to noSpamLog has been made.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[
https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099928#comment-17099928
]
Stephen Mallette commented on CASSANDRA-15582:
--
I just added CASSANDRA-15788 to add tests for {{CacheMetrics}} and
{{ChunkCacheMetrics}}
> 4.0 quality testing: metrics
>
>
> Key: CASSANDRA-15582
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15582
> Project: Cassandra
> Issue Type: Task
> Components: Test/dtest
>Reporter: Josh McKenzie
>Assignee: Romain Hardouin
>Priority: Normal
> Fix For: 4.0-rc
>
> Attachments: Screen Shot 2020-04-07 at 5.47.17 PM.png
>
>
> In past releases we've unknowingly broken metrics integrations and introduced
> performance regressions in metrics collection and reporting. We strive in 4.0
> to not do that. Metrics should work well!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13606) Improve handling of 2i initialization failures
[
https://issues.apache.org/jira/browse/CASSANDRA-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099981#comment-17099981
]
Berenguer Blasi commented on CASSANDRA-13606:
-
Adding a new param to {{rebuild_index}} might not be an option. I'll go for
your last suggestion of calling recovery if index failed during init, thx.
> Improve handling of 2i initialization failures
> --
>
> Key: CASSANDRA-13606
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13606
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/2i Index
>Reporter: Sergio Bossa
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> CASSANDRA-10130 fixes the 2i build management, but initialization failures
> are still not properly handled, most notably because:
> * Initialization failures make the index non-queryable, but it can still be
> written to.
> * Initialization failures can be recovered via full rebuilds.
> Both points above are probably suboptimal because the initialization logic
> could be more complex than just an index build, hence it shouldn't be made
> recoverable via a simple rebuild, and could cause the index to be fully
> unavailable not just for reads, but for writes as well.
> So, we should better handle initialization failures by:
> * Allowing the index implementation to specify if unavailable for reads,
> writes, or both.
> * Providing a proper method to recover, distinct from index rebuilds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15785) 3.11.6 Image Vulnerabilities
[ https://issues.apache.org/jira/browse/CASSANDRA-15785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Capwell updated CASSANDRA-15785: -- Resolution: Won't Fix Status: Resolved (was: Triage Needed) We spoke about this in Slack, the Bitnami image is not owned by this project and the linked dependencies are not directly used by Apache Cassandra, so this issue should move to Bitnami's repo for reporting. > 3.11.6 Image Vulnerabilities > > > Key: CASSANDRA-15785 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15785 > Project: Cassandra > Issue Type: Bug > Components: Build >Reporter: Gil Tohar >Priority: Normal > Attachments: Screen Shot 2020-05-04 at 1.44.19 PM.png > > > My team has taken the Bitnami Cassandra image, 3.11.6, and scanned it using > our image vulnerability scanner, and discovered 4 issues. These are > CVE-2019-5436, CVE-2019-5481, CVE-2019-5482, CVE-2020-1967. The solutions > entail updating 1) curl and 2) openssl in the image, as described in the > attached image. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15776) python dtest regression caused by CASSANDRA-15637
[ https://issues.apache.org/jira/browse/CASSANDRA-15776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100055#comment-17100055 ] David Capwell commented on CASSANDRA-15776: --- [~jmckenzie] mostly need a review, a hand would help =) > python dtest regression caused by CASSANDRA-15637 > - > > Key: CASSANDRA-15776 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15776 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Fix For: 4.0-alpha > > Time Spent: 10m > Remaining Estimate: 0h > > CASSANDRA-15637 deprecated size_estimates in favor of table_estimates to > allow for local primary range estimates (needed for MapReduce). This appears > to have caused a regression in the python dtest nodetool_test.TestNodetool. > test_refresh_size_estimates_clears_invalid_entries (as seen by [Circle > CI|https://app.circleci.com/pipelines/github/dcapwell/cassandra/255/workflows/21907001-93ed-4963-9314-6a0ac6ea0f1d/jobs/1246/tests] > and > [Jenkins|https://ci-cassandra.apache.org/job/Cassandra-trunk-dtest/56/]). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15789) Rows can get duplicated in mixed major-version clusters and after full upgrade
[
https://issues.apache.org/jira/browse/CASSANDRA-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Yeschenko updated CASSANDRA-15789:
--
Description:
In a mixed 2.X/3.X major version cluster a sequence of row deletes, collection
overwrites, paging, and read repair can cause 3.X nodes to split individual
rows into several rows with identical clustering. This happens due to 2.X
paging and RT semantics, and a 3.X {{LegacyLayout}} deficiency.
To reproduce, set up a 2-node mixed major version cluster with the following
table:
{code}
CREATE TABLE distributed_test_keyspace.tlb (
pk int,
ck int,
v map,
PRIMARY KEY (pk, ck)
);
{code}
1. Using either node as the coordinator, delete the row with ck=2 using
timestamp 1
{code}
DELETE FROM tbl USING TIMESTAMP 1 WHERE pk = 1 AND ck = 2;
{code}
2. Using either node as the coordinator, insert the following 3 rows:
{code}
INSERT INTO tbl (pk, ck, v) VALUES (1, 1, {'e':'f'}) USING TIMESTAMP 3;
INSERT INTO tbl (pk, ck, v) VALUES (1, 2, {'g':'h'}) USING TIMESTAMP 3;
INSERT INTO tbl (pk, ck, v) VALUES (1, 3, {'i':'j'}) USING TIMESTAMP 3;
{code}
3. Flush the table on both nodes
4. Using the 2.2 node as the coordinator, force read repar by querying the
table with page size = 2:
{code}
SELECT * FROM tbl;
{code}
5. Overwrite the row with ck=2 using timestamp 5:
{code}
INSERT INTO tbl (pk, ck, v) VALUES (1, 2, {'g':'h'}) USING TIMESTAMP 5;}}
{code}
6. Query the 3.0 node and observe the split row:
{code}
cqlsh> select * from distributed_test_keyspace.tlb ;
pk | ck | v
++
1 | 1 | {'e': 'f'}
1 | 2 | {'g': 'h'}
1 | 2 | {'k': 'l'}
1 | 3 | {'i': 'j'}
{code}
This happens because the read to query the second page ends up generating the
following mutation for the 3.0 node:
{code}
ColumnFamily(tbl -{deletedAt=-9223372036854775808, localDeletion=2147483647,
ranges=[2:v:_-2:v:!, deletedAt=2, localDeletion=1588588821]
[2:v:!-2:!, deletedAt=1, localDeletion=1588588821]
[3:v:_-3:v:!, deletedAt=2, localDeletion=1588588821]}-
[2:v:63:false:1@3,])
{code}
Which on 3.0 side gets incorrectly deserialized as
{code}
Mutation(keyspace='distributed_test_keyspace', key='0001', modifications=[
[distributed_test_keyspace.tbl] key=1
partition_deletion=deletedAt=-9223372036854775808, localDeletion=2147483647
columns=[[] | [v]]
Row[info=[ts=-9223372036854775808] ]: ck=2 | del(v)=deletedAt=2,
localDeletion=1588588821, [v[c]=d ts=3]
Row[info=[ts=-9223372036854775808] del=deletedAt=1,
localDeletion=1588588821 ]: ck=2 |
Row[info=[ts=-9223372036854775808] ]: ck=3 | del(v)=deletedAt=2,
localDeletion=1588588821
])
{code}
{{LegacyLayout}} correctly interprets a range tombstone whose start and finish
{{collectionName}} values don't match as a wrapping fragment of a legacy row
deletion that's being interrupted by a collection deletion (correctly) - see
[code|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/LegacyLayout.java#L1874-L1889].
Quoting the comment inline:
{code}
// Because of the way RangeTombstoneList work, we can have a tombstone where
only one of
// the bound has a collectionName. That happens if we have a big tombstone A
(spanning one
// or multiple rows) and a collection tombstone B. In that case,
RangeTombstoneList will
// split this into 3 RTs: the first one from the beginning of A to the
beginning of B,
// then B, then a third one from the end of B to the end of A. To make this
simpler, if
// we detect that case we transform the 1st and 3rd tombstone so they don't
end in the middle
// of a row (which is still correct).
{code}
{{LegacyLayout#addRowTombstone()}} method then chokes when it encounters such a
tombstone in the middle of an existing row - having seen {{v[c]=d}} first, and
mistakenly starts a new row, while in the middle of an existing one: (see
[code|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/LegacyLayout.java#L1500-L1501]).
> Rows can get duplicated in mixed major-version clusters and after full upgrade
> --
>
> Key: CASSANDRA-15789
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15789
> Project: Cassandra
> Issue Type: Bug
>Reporter: Aleksey Yeschenko
>Assignee: Marcus Eriksson
>Priority: Normal
>
> In a mixed 2.X/3.X major version cluster a sequence of row deletes,
> collection overwrites, paging, and read repair can cause 3.X nodes to split
> individual rows into several rows with identical clustering. This happens due
> to 2.X paging and RT semantics, and a 3.X {{LegacyLayout}} deficiency.
> To reproduce, set up a 2-node mixed major version cluster with the following
[jira] [Commented] (CASSANDRA-14781) Log message when mutation passed to CommitLog#add(Mutation) is too large is not descriptive enough
[ https://issues.apache.org/jira/browse/CASSANDRA-14781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100077#comment-17100077 ] Venkata Harikrishna Nukala commented on CASSANDRA-14781: Thanks a lot [~jwest] for taking it forward!! > Log message when mutation passed to CommitLog#add(Mutation) is too large is > not descriptive enough > -- > > Key: CASSANDRA-14781 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14781 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Hints, Local/Commit Log, Messaging/Client >Reporter: Jordan West >Assignee: Venkata Harikrishna Nukala >Priority: Normal > Labels: protocolv5 > Fix For: 4.0-beta > > Attachments: CASSANDRA-14781.patch, CASSANDRA-14781_3.0.patch, > CASSANDRA-14781_3.11.patch > > > When hitting > [https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/commitlog/CommitLog.java#L256-L257], > the log message produced does not help the operator track down what data is > being written. At a minimum the keyspace and cfIds involved would be useful > (and are available) – more detail might not be reasonable to include. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13474) Cassandra pluggable storage engine
[ https://issues.apache.org/jira/browse/CASSANDRA-13474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100105#comment-17100105 ] Benjamin S commented on CASSANDRA-13474: Is the pluggable storage engine still planned to be available with 4.0? Seems progress has slowed. > Cassandra pluggable storage engine > -- > > Key: CASSANDRA-13474 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13474 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Core >Reporter: Dikang Gu >Priority: Normal > > Instagram is working on a project to significantly reduce Cassandra's tail > latency, by implementing a new storage engine on top of RocksDB, named > Rocksandra. > We started a prototype of single column (key-value) use case, and then > implemented a full design to support most of the data types and data models > in Cassandra, as well as streaming. > After a year of development and testing, we have rolled out the Rocksandra > project to our internal deployments, and observed 3-4X reduction on P99 read > latency in general, even more than 10 times reduction for some use cases. > We published a blog post about the wins and the benchmark metrics on AWS > environment. > https://engineering.instagram.com/open-sourcing-a-10x-reduction-in-apache-cassandra-tail-latency-d64f86b43589 > I think the biggest performance win comes from we get rid of most Java > garbages created by current read/write path and compactions, which reduces > the JVM overhead and makes the latency to be more predictable. > We are very excited about the potential performance gain. As the next step, I > propose to make the Cassandra storage engine to be pluggable (like Mysql and > MongoDB), and we are very interested in providing RocksDB as one storage > option with more predictable performance, together with community. > Design doc for pluggable storage engine: > https://docs.google.com/document/d/1suZlvhzgB6NIyBNpM9nxoHxz_Ri7qAm-UEO8v8AIFsc/edit -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15790) EmptyType doesn't override writeValue so could attempt to write bytes when expected not to
David Capwell created CASSANDRA-15790:
-
Summary: EmptyType doesn't override writeValue so could attempt to
write bytes when expected not to
Key: CASSANDRA-15790
URL: https://issues.apache.org/jira/browse/CASSANDRA-15790
Project: Cassandra
Issue Type: Bug
Components: CQL/Semantics
Reporter: David Capwell
Assignee: David Capwell
EmptyType.writeValues is defined here
https://github.com/apache/cassandra/blob/e394dc0bb32f612a476269010930c617dd1ed3cb/src/java/org/apache/cassandra/db/marshal/AbstractType.java#L407-L414
{code}
public void writeValue(ByteBuffer value, DataOutputPlus out) throws IOException
{
assert value.hasRemaining();
if (valueLengthIfFixed() >= 0)
out.write(value);
else
ByteBufferUtil.writeWithVIntLength(value, out);
}
{code}
This is fine when the value is empty as the write of empty no-ops (the
readValue also noops since the length is 0), but if the value is not empty
(possible during upgrades or random bugs) then this could silently cause
corruption; ideally this should throw a exception if the ByteBuffer has data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15790) EmptyType doesn't override writeValue so could attempt to write bytes when expected not to
[
https://issues.apache.org/jira/browse/CASSANDRA-15790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Capwell updated CASSANDRA-15790:
--
Bug Category: Parent values: Correctness(12982)Level 1 values:
Unrecoverable Corruption / Loss(13161)
Complexity: Low Hanging Fruit
Discovered By: User Report
Severity: Normal
Status: Open (was: Triage Needed)
> EmptyType doesn't override writeValue so could attempt to write bytes when
> expected not to
> --
>
> Key: CASSANDRA-15790
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15790
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Semantics
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>
> EmptyType.writeValues is defined here
> https://github.com/apache/cassandra/blob/e394dc0bb32f612a476269010930c617dd1ed3cb/src/java/org/apache/cassandra/db/marshal/AbstractType.java#L407-L414
> {code}
> public void writeValue(ByteBuffer value, DataOutputPlus out) throws
> IOException
> {
> assert value.hasRemaining();
> if (valueLengthIfFixed() >= 0)
> out.write(value);
> else
> ByteBufferUtil.writeWithVIntLength(value, out);
> }
> {code}
> This is fine when the value is empty as the write of empty no-ops (the
> readValue also noops since the length is 0), but if the value is not empty
> (possible during upgrades or random bugs) then this could silently cause
> corruption; ideally this should throw a exception if the ByteBuffer has data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15790) EmptyType doesn't override writeValue so could attempt to write bytes when expected not to
[
https://issues.apache.org/jira/browse/CASSANDRA-15790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100171#comment-17100171
]
David Capwell commented on CASSANDRA-15790:
---
Linked CASSANDRA-15778 as this was found while investigating.
> EmptyType doesn't override writeValue so could attempt to write bytes when
> expected not to
> --
>
> Key: CASSANDRA-15790
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15790
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Semantics
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>
> EmptyType.writeValues is defined here
> https://github.com/apache/cassandra/blob/e394dc0bb32f612a476269010930c617dd1ed3cb/src/java/org/apache/cassandra/db/marshal/AbstractType.java#L407-L414
> {code}
> public void writeValue(ByteBuffer value, DataOutputPlus out) throws
> IOException
> {
> assert value.hasRemaining();
> if (valueLengthIfFixed() >= 0)
> out.write(value);
> else
> ByteBufferUtil.writeWithVIntLength(value, out);
> }
> {code}
> This is fine when the value is empty as the write of empty no-ops (the
> readValue also noops since the length is 0), but if the value is not empty
> (possible during upgrades or random bugs) then this could silently cause
> corruption; ideally this should throw a exception if the ByteBuffer has data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15789) Rows can get duplicated in mixed major-version clusters and after full upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-15789: -- Summary: Rows can get duplicated in mixed major-version clusters and after full upgrade (was: TBD) > Rows can get duplicated in mixed major-version clusters and after full upgrade > -- > > Key: CASSANDRA-15789 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15789 > Project: Cassandra > Issue Type: Bug >Reporter: Aleksey Yeschenko >Assignee: Marcus Eriksson >Priority: Normal > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15214) OOMs caught and not rethrown
[ https://issues.apache.org/jira/browse/CASSANDRA-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100063#comment-17100063 ] Joey Lynch commented on CASSANDRA-15214: > An alternative way could be programmatically grab the heap dump via > [JMX|https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/jdk.management/share/classes/com/sun/management/HotSpotDiagnosticMXBean.java#L75] > and exit. I believe that was more or less what C* was doing before CASSANDRA-13006 if I'm reading the patch in [02aba73|https://github.com/apache/cassandra/commit/02aba73] correctly, and Eric Evans pointed out this approach in general can cause the C*'s jmap heap dump to race with the JVM heap dump and advocated for just letting the JVM handle it with built in options. The nice thing about the jvmquake technique of just running the heap out of memory is all the normal JVM options work as expected (logging and dumping heap to a particular location on disk mostly). That being said, I think that for the direct buffer issue in particular this won't be a problem since as we've established the JVM OOM report_java_out_of_memory isn't triggered on direct memory allocation failures. > OOMs caught and not rethrown > > > Key: CASSANDRA-15214 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15214 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client, Messaging/Internode >Reporter: Benedict Elliott Smith >Priority: Normal > Fix For: 4.0, 4.0-rc > > Attachments: oom-experiments.zip > > > Netty (at least, and perhaps elsewhere in Executors) catches all exceptions, > so presently there is no way to ensure that an OOM reaches the JVM handler to > trigger a crash/heapdump. > It may be that the simplest most consistent way to do this would be to have a > single thread spawned at startup that waits for any exceptions we must > propagate to the Runtime. > We could probably submit a patch upstream to Netty, but for a guaranteed > future proof approach, it may be worth paying the cost of a single thread. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15776) python dtest regression caused by CASSANDRA-15637
[ https://issues.apache.org/jira/browse/CASSANDRA-15776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100134#comment-17100134 ] Michael Semb Wever commented on CASSANDRA-15776: The patch is running through Jenkins [here|https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-devbranch/detail/Cassandra-devbranch/111/pipeline]. > python dtest regression caused by CASSANDRA-15637 > - > > Key: CASSANDRA-15776 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15776 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Fix For: 4.0-alpha > > Time Spent: 10m > Remaining Estimate: 0h > > CASSANDRA-15637 deprecated size_estimates in favor of table_estimates to > allow for local primary range estimates (needed for MapReduce). This appears > to have caused a regression in the python dtest nodetool_test.TestNodetool. > test_refresh_size_estimates_clears_invalid_entries (as seen by [Circle > CI|https://app.circleci.com/pipelines/github/dcapwell/cassandra/255/workflows/21907001-93ed-4963-9314-6a0ac6ea0f1d/jobs/1246/tests] > and > [Jenkins|https://ci-cassandra.apache.org/job/Cassandra-trunk-dtest/56/]). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15299) CASSANDRA-13304 follow-up: improve checksumming and compression in protocol v5-beta
[
https://issues.apache.org/jira/browse/CASSANDRA-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100165#comment-17100165
]
Sam Tunnicliffe commented on CASSANDRA-15299:
-
bq. For context, how common are flipped bits in your experience? Dropping the
whole connection every time might seem a bit heavy-handed, but if it's a rare
occurrence maybe that's an acceptable tradeoff.
Not that common, but I know they do happen very occasionally. Probably rarely
enough to make that tradeoff acceptable.
{quote}It might be worth mentioning that in the document as a hint for
implementors
{quote}
Will do.
> CASSANDRA-13304 follow-up: improve checksumming and compression in protocol
> v5-beta
> ---
>
> Key: CASSANDRA-15299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15299
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Aleksey Yeschenko
>Assignee: Sam Tunnicliffe
>Priority: Normal
> Labels: protocolv5
> Fix For: 4.0-beta
>
>
> CASSANDRA-13304 made an important improvement to our native protocol: it
> introduced checksumming/CRC32 to request and response bodies. It’s an
> important step forward, but it doesn’t cover the entire stream. In
> particular, the message header is not covered by a checksum or a crc, which
> poses a correctness issue if, for example, {{streamId}} gets corrupted.
> Additionally, we aren’t quite using CRC32 correctly, in two ways:
> 1. We are calculating the CRC32 of the *decompressed* value instead of
> computing the CRC32 on the bytes written on the wire - losing the properties
> of the CRC32. In some cases, due to this sequencing, attempting to decompress
> a corrupt stream can cause a segfault by LZ4.
> 2. When using CRC32, the CRC32 value is written in the incorrect byte order,
> also losing some of the protections.
> See https://users.ece.cmu.edu/~koopman/pubs/KoopmanCRCWebinar9May2012.pdf for
> explanation for the two points above.
> Separately, there are some long-standing issues with the protocol - since
> *way* before CASSANDRA-13304. Importantly, both checksumming and compression
> operate on individual message bodies rather than frames of multiple complete
> messages. In reality, this has several important additional downsides. To
> name a couple:
> # For compression, we are getting poor compression ratios for smaller
> messages - when operating on tiny sequences of bytes. In reality, for most
> small requests and responses we are discarding the compressed value as it’d
> be smaller than the uncompressed one - incurring both redundant allocations
> and compressions.
> # For checksumming and CRC32 we pay a high overhead price for small messages.
> 4 bytes extra is *a lot* for an empty write response, for example.
> To address the correctness issue of {{streamId}} not being covered by the
> checksum/CRC32 and the inefficiency in compression and checksumming/CRC32, we
> should switch to a framing protocol with multiple messages in a single frame.
> I suggest we reuse the framing protocol recently implemented for internode
> messaging in CASSANDRA-15066 to the extent that its logic can be borrowed,
> and that we do it before native protocol v5 graduates from beta. See
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderCrc.java
> and
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderLZ4.java.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15789) Rows can get duplicated in mixed major-version clusters and after full upgrade
[
https://issues.apache.org/jira/browse/CASSANDRA-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Yeschenko updated CASSANDRA-15789:
--
Bug Category: Parent values: Correctness(12982)Level 1 values: Recoverable
Corruption / Loss(12986)
Complexity: Normal
Component/s: Local/SSTable
Local/Memtable
Consistency/Coordination
Discovered By: Fuzz Test
Reviewers: Alex Petrov
Severity: Critical
Status: Open (was: Triage Needed)
> Rows can get duplicated in mixed major-version clusters and after full upgrade
> --
>
> Key: CASSANDRA-15789
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15789
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination, Local/Memtable, Local/SSTable
>Reporter: Aleksey Yeschenko
>Assignee: Marcus Eriksson
>Priority: Normal
>
> In a mixed 2.X/3.X major version cluster a sequence of row deletes,
> collection overwrites, paging, and read repair can cause 3.X nodes to split
> individual rows into several rows with identical clustering. This happens due
> to 2.X paging and RT semantics, and a 3.X {{LegacyLayout}} deficiency.
> To reproduce, set up a 2-node mixed major version cluster with the following
> table:
> {code}
> CREATE TABLE distributed_test_keyspace.tlb (
> pk int,
> ck int,
> v map,
> PRIMARY KEY (pk, ck)
> );
> {code}
> 1. Using either node as the coordinator, delete the row with ck=2 using
> timestamp 1
> {code}
> DELETE FROM tbl USING TIMESTAMP 1 WHERE pk = 1 AND ck = 2;
> {code}
> 2. Using either node as the coordinator, insert the following 3 rows:
> {code}
> INSERT INTO tbl (pk, ck, v) VALUES (1, 1, {'e':'f'}) USING TIMESTAMP 3;
> INSERT INTO tbl (pk, ck, v) VALUES (1, 2, {'g':'h'}) USING TIMESTAMP 3;
> INSERT INTO tbl (pk, ck, v) VALUES (1, 3, {'i':'j'}) USING TIMESTAMP 3;
> {code}
> 3. Flush the table on both nodes
> 4. Using the 2.2 node as the coordinator, force read repar by querying the
> table with page size = 2:
>
> {code}
> SELECT * FROM tbl;
> {code}
> 5. Overwrite the row with ck=2 using timestamp 5:
> {code}
> INSERT INTO tbl (pk, ck, v) VALUES (1, 2, {'g':'h'}) USING TIMESTAMP 5;}}
> {code}
> 6. Query the 3.0 node and observe the split row:
> {code}
> cqlsh> select * from distributed_test_keyspace.tlb ;
> pk | ck | v
> ++
> 1 | 1 | {'e': 'f'}
> 1 | 2 | {'g': 'h'}
> 1 | 2 | {'k': 'l'}
> 1 | 3 | {'i': 'j'}
> {code}
> This happens because the read to query the second page ends up generating the
> following mutation for the 3.0 node:
> {code}
> ColumnFamily(tbl -{deletedAt=-9223372036854775808, localDeletion=2147483647,
> ranges=[2:v:_-2:v:!, deletedAt=2, localDeletion=1588588821]
> [2:v:!-2:!, deletedAt=1, localDeletion=1588588821]
> [3:v:_-3:v:!, deletedAt=2, localDeletion=1588588821]}-
> [2:v:63:false:1@3,])
> {code}
> Which on 3.0 side gets incorrectly deserialized as
> {code}
> Mutation(keyspace='distributed_test_keyspace', key='0001', modifications=[
> [distributed_test_keyspace.tbl] key=1
> partition_deletion=deletedAt=-9223372036854775808, localDeletion=2147483647
> columns=[[] | [v]]
> Row[info=[ts=-9223372036854775808] ]: ck=2 | del(v)=deletedAt=2,
> localDeletion=1588588821, [v[c]=d ts=3]
> Row[info=[ts=-9223372036854775808] del=deletedAt=1,
> localDeletion=1588588821 ]: ck=2 |
> Row[info=[ts=-9223372036854775808] ]: ck=3 | del(v)=deletedAt=2,
> localDeletion=1588588821
> ])
> {code}
> {{LegacyLayout}} correctly interprets a range tombstone whose start and
> finish {{collectionName}} values don't match as a wrapping fragment of a
> legacy row deletion that's being interrupted by a collection deletion
> (correctly) - see
> [code|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/LegacyLayout.java#L1874-L1889].
> Quoting the comment inline:
> {code}
> // Because of the way RangeTombstoneList work, we can have a tombstone where
> only one of
> // the bound has a collectionName. That happens if we have a big tombstone A
> (spanning one
> // or multiple rows) and a collection tombstone B. In that case,
> RangeTombstoneList will
> // split this into 3 RTs: the first one from the beginning of A to the
> beginning of B,
> // then B, then a third one from the end of B to the end of A. To make this
> simpler, if
> // we detect that case we transform the 1st and 3rd tombstone so they don't
> end in the middle
> // of a row (which is still correct).
> {code}
> {{LegacyLayout#addRowTombstone()}} method then chokes when it encounters such
> a tombstone in the middle of an existing row - having seen
[jira] [Comment Edited] (CASSANDRA-15733) jvm dtest builder should be provided to the factory and expose state
[ https://issues.apache.org/jira/browse/CASSANDRA-15733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097102#comment-17097102 ] David Capwell edited comment on CASSANDRA-15733 at 5/5/20, 5:05 PM: | Branch | Build | Results | | trunk | [Circle CI|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=feature%2FCASSANDRA-15733] | GREEN | | 3.11 | [Circle CI|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=feature%2FCASSANDRA-15733-3.11] | GREEN | | 3.0 | [Circle CI|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=feature%2FCASSANDRA-15733-3.0] | GREEN | | 2.2 | [Circle CI|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=feature%2FCASSANDRA-15733-2.2] | YELLOW [1] | [1] - org.apache.cassandra.cql3.validation.entities.TypeTest#testDateCompatibility failed in 2.2, this was passing before so assume its a flaky test. The code is jvm dtest so ignoring this error. was (Author: dcapwell): | Branch | Build | Results | | trunk | [Circle CI|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=feature%2FCASSANDRA-15733] | GREEN | | 3.11 | [Circle CI|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=feature%2FCASSANDRA-15733-3.11] | GREEN | | 3.0 | [Circle CI|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=feature%2FCASSANDRA-15733-3.0] | GREEN | | 2.2 | [Circle CI|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=feature%2FCASSANDRA-15733-2.2] | UNKNOWN | > jvm dtest builder should be provided to the factory and expose state > > > Key: CASSANDRA-15733 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15733 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > Currently the builder is rather heavy and creates configs plus call the > factory with specific fields only, this isn’t that flexible and makes it > harder to have custom cluster definitions which require additional fields to > be defined. To solve this we should make the builder be sent to the factory > and expose the state so the factory can get all the fields it needs, the > factory should also be in charge of creating the configs -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15766) NoSpamLogger arguments building objects on hot paths
[
https://issues.apache.org/jira/browse/CASSANDRA-15766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100159#comment-17100159
]
Yifan Cai commented on CASSANDRA-15766:
---
There might not be a big difference between multiple and single lambda. The
lambda is compiled into a static method in the bytecode view. If in the hot
path and JIT kicked in, the static method could be even inlined.
Consider the following code
{code:java}
import java.util.function.Supplier;
class Foo {
public void foo() {
comsume(() -> "foo");
}
public void comsume(Supplier supp) {
supp.get();
}
}
{code}
Running {{javap -v -p Foo.class}} reveals the below.
{code:java}
private static java.lang.String lambda$foo$0();
descriptor: ()Ljava/lang/String;
flags: ACC_PRIVATE, ACC_STATIC, ACC_SYNTHETIC
Code:
stack=1, locals=0, args_size=0
0: ldc #5 // String foo
2: areturn
LineNumberTable:
line 5: 0{code}
> NoSpamLogger arguments building objects on hot paths
>
>
> Key: CASSANDRA-15766
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15766
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Logging
>Reporter: Jon Meredith
>Assignee: Jon Meredith
>Priority: Normal
> Fix For: 4.0-rc
>
>
> NoSpamLogger is used in hot logging paths to prevent logs being overrun. For
> that to be most effective the arguments to the logger need to be cheap to
> construct. During the internode messaging refactor CASSANDRA-15066,
> performance changes to BufferPool for CASSANDRA-14416
> were accidentally reverted in the merge up from 3.11.
> Reviewing other uses since, it looks like there are a few places where the
> arguments require some form of String building.
> org.apache.cassandra.net.InboundSink#accept
> org.apache.cassandra.net.InboundMessageHandler#processCorruptFrame
> org.apache.cassandra.net.InboundMessageHandler.LargeMessage#deserialize
> org.apache.cassandra.net.OutboundConnection#onOverloaded
> org.apache.cassandra.utils.memory.BufferPool.GlobalPool#allocateMoreChunks
> Formatting arguments should either be precomputed, or if expensive they
> should be computed after the decision on whether to noSpamLog has been made.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15789) Rows can get duplicated in mixed major-version clusters and after full upgrade
[
https://issues.apache.org/jira/browse/CASSANDRA-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-15789:
Authors: Aleksey Yeschenko, Marcus Eriksson, Sam Tunnicliffe (was: Marcus
Eriksson)
> Rows can get duplicated in mixed major-version clusters and after full upgrade
> --
>
> Key: CASSANDRA-15789
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15789
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination, Local/Memtable, Local/SSTable
>Reporter: Aleksey Yeschenko
>Assignee: Marcus Eriksson
>Priority: Normal
>
> In a mixed 2.X/3.X major version cluster a sequence of row deletes,
> collection overwrites, paging, and read repair can cause 3.X nodes to split
> individual rows into several rows with identical clustering. This happens due
> to 2.X paging and RT semantics, and a 3.X {{LegacyLayout}} deficiency.
> To reproduce, set up a 2-node mixed major version cluster with the following
> table:
> {code}
> CREATE TABLE distributed_test_keyspace.tlb (
> pk int,
> ck int,
> v map,
> PRIMARY KEY (pk, ck)
> );
> {code}
> 1. Using either node as the coordinator, delete the row with ck=2 using
> timestamp 1
> {code}
> DELETE FROM tbl USING TIMESTAMP 1 WHERE pk = 1 AND ck = 2;
> {code}
> 2. Using either node as the coordinator, insert the following 3 rows:
> {code}
> INSERT INTO tbl (pk, ck, v) VALUES (1, 1, {'e':'f'}) USING TIMESTAMP 3;
> INSERT INTO tbl (pk, ck, v) VALUES (1, 2, {'g':'h'}) USING TIMESTAMP 3;
> INSERT INTO tbl (pk, ck, v) VALUES (1, 3, {'i':'j'}) USING TIMESTAMP 3;
> {code}
> 3. Flush the table on both nodes
> 4. Using the 2.2 node as the coordinator, force read repar by querying the
> table with page size = 2:
>
> {code}
> SELECT * FROM tbl;
> {code}
> 5. Overwrite the row with ck=2 using timestamp 5:
> {code}
> INSERT INTO tbl (pk, ck, v) VALUES (1, 2, {'g':'h'}) USING TIMESTAMP 5;}}
> {code}
> 6. Query the 3.0 node and observe the split row:
> {code}
> cqlsh> select * from distributed_test_keyspace.tlb ;
> pk | ck | v
> ++
> 1 | 1 | {'e': 'f'}
> 1 | 2 | {'g': 'h'}
> 1 | 2 | {'k': 'l'}
> 1 | 3 | {'i': 'j'}
> {code}
> This happens because the read to query the second page ends up generating the
> following mutation for the 3.0 node:
> {code}
> ColumnFamily(tbl -{deletedAt=-9223372036854775808, localDeletion=2147483647,
> ranges=[2:v:_-2:v:!, deletedAt=2, localDeletion=1588588821]
> [2:v:!-2:!, deletedAt=1, localDeletion=1588588821]
> [3:v:_-3:v:!, deletedAt=2, localDeletion=1588588821]}-
> [2:v:63:false:1@3,])
> {code}
> Which on 3.0 side gets incorrectly deserialized as
> {code}
> Mutation(keyspace='distributed_test_keyspace', key='0001', modifications=[
> [distributed_test_keyspace.tbl] key=1
> partition_deletion=deletedAt=-9223372036854775808, localDeletion=2147483647
> columns=[[] | [v]]
> Row[info=[ts=-9223372036854775808] ]: ck=2 | del(v)=deletedAt=2,
> localDeletion=1588588821, [v[c]=d ts=3]
> Row[info=[ts=-9223372036854775808] del=deletedAt=1,
> localDeletion=1588588821 ]: ck=2 |
> Row[info=[ts=-9223372036854775808] ]: ck=3 | del(v)=deletedAt=2,
> localDeletion=1588588821
> ])
> {code}
> {{LegacyLayout}} correctly interprets a range tombstone whose start and
> finish {{collectionName}} values don't match as a wrapping fragment of a
> legacy row deletion that's being interrupted by a collection deletion
> (correctly) - see
> [code|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/LegacyLayout.java#L1874-L1889].
> Quoting the comment inline:
> {code}
> // Because of the way RangeTombstoneList work, we can have a tombstone where
> only one of
> // the bound has a collectionName. That happens if we have a big tombstone A
> (spanning one
> // or multiple rows) and a collection tombstone B. In that case,
> RangeTombstoneList will
> // split this into 3 RTs: the first one from the beginning of A to the
> beginning of B,
> // then B, then a third one from the end of B to the end of A. To make this
> simpler, if
> // we detect that case we transform the 1st and 3rd tombstone so they don't
> end in the middle
> // of a row (which is still correct).
> {code}
> {{LegacyLayout#addRowTombstone()}} method then chokes when it encounters such
> a tombstone in the middle of an existing row - having seen {{v[c]=d}} first,
> and mistakenly starts a new row, while in the middle of an existing one: (see
> [code|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/LegacyLayout.java#L1500-L1501]).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CASSANDRA-15790) EmptyType doesn't override writeValue so could attempt to write bytes when expected not to
[
https://issues.apache.org/jira/browse/CASSANDRA-15790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100176#comment-17100176
]
David Capwell commented on CASSANDRA-15790:
---
This was found with Compact Storage, if the value field is non-empty this
silently corrupts sstables and networking
> EmptyType doesn't override writeValue so could attempt to write bytes when
> expected not to
> --
>
> Key: CASSANDRA-15790
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15790
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Semantics
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>
> EmptyType.writeValues is defined here
> https://github.com/apache/cassandra/blob/e394dc0bb32f612a476269010930c617dd1ed3cb/src/java/org/apache/cassandra/db/marshal/AbstractType.java#L407-L414
> {code}
> public void writeValue(ByteBuffer value, DataOutputPlus out) throws
> IOException
> {
> assert value.hasRemaining();
> if (valueLengthIfFixed() >= 0)
> out.write(value);
> else
> ByteBufferUtil.writeWithVIntLength(value, out);
> }
> {code}
> This is fine when the value is empty as the write of empty no-ops (the
> readValue also noops since the length is 0), but if the value is not empty
> (possible during upgrades or random bugs) then this could silently cause
> corruption; ideally this should throw a exception if the ByteBuffer has data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15790) EmptyType doesn't override writeValue so could attempt to write bytes when expected not to
[
https://issues.apache.org/jira/browse/CASSANDRA-15790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Capwell updated CASSANDRA-15790:
--
Test and Documentation Plan: circle ci + manual verification
Status: Patch Available (was: Open)
> EmptyType doesn't override writeValue so could attempt to write bytes when
> expected not to
> --
>
> Key: CASSANDRA-15790
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15790
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Semantics
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>
> EmptyType.writeValues is defined here
> https://github.com/apache/cassandra/blob/e394dc0bb32f612a476269010930c617dd1ed3cb/src/java/org/apache/cassandra/db/marshal/AbstractType.java#L407-L414
> {code}
> public void writeValue(ByteBuffer value, DataOutputPlus out) throws
> IOException
> {
> assert value.hasRemaining();
> if (valueLengthIfFixed() >= 0)
> out.write(value);
> else
> ByteBufferUtil.writeWithVIntLength(value, out);
> }
> {code}
> This is fine when the value is empty as the write of empty no-ops (the
> readValue also noops since the length is 0), but if the value is not empty
> (possible during upgrades or random bugs) then this could silently cause
> corruption; ideally this should throw a exception if the ByteBuffer has data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15214) OOMs caught and not rethrown
[
https://issues.apache.org/jira/browse/CASSANDRA-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100193#comment-17100193
]
Yifan Cai commented on CASSANDRA-15214:
---
Got it. I did not look closely enough at the discussions in CASSANDRA-13006.
I agree that leaving it to JVM is a more clean and general solution. Also as
you mentioned, "It's relatively easy to ignore the "sacrificial" long array in
a heap dump and we could log clearly what is happening."
Since one should be able to trigger the OOM by looping allocating large chunk
of memory, e.g. array, in the java code. What is the benefit of doing it so
using jvmquake? I can see that in the killer_thread callback function, it also
does long array allocation once notified by the gc callback.
The comment of the callback says
{quote}the only way to reliably trigger OutOfMemory
when we are not actually out of memory (e.g. due to GC behavior) that I
could find was to make JNI calls that allocate large blobs of memory which
can only be done from outside of the GC callbacks.
{quote}
Can you elaborate more about preferring jvmquake?
> OOMs caught and not rethrown
>
>
> Key: CASSANDRA-15214
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15214
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Client, Messaging/Internode
>Reporter: Benedict Elliott Smith
>Priority: Normal
> Fix For: 4.0, 4.0-rc
>
> Attachments: oom-experiments.zip
>
>
> Netty (at least, and perhaps elsewhere in Executors) catches all exceptions,
> so presently there is no way to ensure that an OOM reaches the JVM handler to
> trigger a crash/heapdump.
> It may be that the simplest most consistent way to do this would be to have a
> single thread spawned at startup that waits for any exceptions we must
> propagate to the Runtime.
> We could probably submit a patch upstream to Netty, but for a guaranteed
> future proof approach, it may be worth paying the cost of a single thread.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15789) Rows can get duplicated in mixed major-version clusters and after full upgrade
[
https://issues.apache.org/jira/browse/CASSANDRA-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100070#comment-17100070
]
Aleksey Yeschenko edited comment on CASSANDRA-15789 at 5/5/20, 4:52 PM:
{{nodetool scrub}} can fix thee sstables by collapsing rows with the same
clustering into one, via the logic added in CASSANDRA-12144 to address a
similar corruption.
was (Author: iamaleksey):
{{nodetool scrub}} fixes the issue by collapsing rows with the same clustering
into one, via the logic added in CASSANDRA-12144 to address a similar
corruption.
> Rows can get duplicated in mixed major-version clusters and after full upgrade
> --
>
> Key: CASSANDRA-15789
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15789
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination, Local/Memtable, Local/SSTable
>Reporter: Aleksey Yeschenko
>Assignee: Marcus Eriksson
>Priority: Normal
>
> In a mixed 2.X/3.X major version cluster a sequence of row deletes,
> collection overwrites, paging, and read repair can cause 3.X nodes to split
> individual rows into several rows with identical clustering. This happens due
> to 2.X paging and RT semantics, and a 3.X {{LegacyLayout}} deficiency.
> To reproduce, set up a 2-node mixed major version cluster with the following
> table:
> {code}
> CREATE TABLE distributed_test_keyspace.tlb (
> pk int,
> ck int,
> v map,
> PRIMARY KEY (pk, ck)
> );
> {code}
> 1. Using either node as the coordinator, delete the row with ck=2 using
> timestamp 1
> {code}
> DELETE FROM tbl USING TIMESTAMP 1 WHERE pk = 1 AND ck = 2;
> {code}
> 2. Using either node as the coordinator, insert the following 3 rows:
> {code}
> INSERT INTO tbl (pk, ck, v) VALUES (1, 1, {'e':'f'}) USING TIMESTAMP 3;
> INSERT INTO tbl (pk, ck, v) VALUES (1, 2, {'g':'h'}) USING TIMESTAMP 3;
> INSERT INTO tbl (pk, ck, v) VALUES (1, 3, {'i':'j'}) USING TIMESTAMP 3;
> {code}
> 3. Flush the table on both nodes
> 4. Using the 2.2 node as the coordinator, force read repar by querying the
> table with page size = 2:
>
> {code}
> SELECT * FROM tbl;
> {code}
> 5. Overwrite the row with ck=2 using timestamp 5:
> {code}
> INSERT INTO tbl (pk, ck, v) VALUES (1, 2, {'g':'h'}) USING TIMESTAMP 5;}}
> {code}
> 6. Query the 3.0 node and observe the split row:
> {code}
> cqlsh> select * from distributed_test_keyspace.tlb ;
> pk | ck | v
> ++
> 1 | 1 | {'e': 'f'}
> 1 | 2 | {'g': 'h'}
> 1 | 2 | {'k': 'l'}
> 1 | 3 | {'i': 'j'}
> {code}
> This happens because the read to query the second page ends up generating the
> following mutation for the 3.0 node:
> {code}
> ColumnFamily(tbl -{deletedAt=-9223372036854775808, localDeletion=2147483647,
> ranges=[2:v:_-2:v:!, deletedAt=2, localDeletion=1588588821]
> [2:v:!-2:!, deletedAt=1, localDeletion=1588588821]
> [3:v:_-3:v:!, deletedAt=2, localDeletion=1588588821]}-
> [2:v:63:false:1@3,])
> {code}
> Which on 3.0 side gets incorrectly deserialized as
> {code}
> Mutation(keyspace='distributed_test_keyspace', key='0001', modifications=[
> [distributed_test_keyspace.tbl] key=1
> partition_deletion=deletedAt=-9223372036854775808, localDeletion=2147483647
> columns=[[] | [v]]
> Row[info=[ts=-9223372036854775808] ]: ck=2 | del(v)=deletedAt=2,
> localDeletion=1588588821, [v[c]=d ts=3]
> Row[info=[ts=-9223372036854775808] del=deletedAt=1,
> localDeletion=1588588821 ]: ck=2 |
> Row[info=[ts=-9223372036854775808] ]: ck=3 | del(v)=deletedAt=2,
> localDeletion=1588588821
> ])
> {code}
> {{LegacyLayout}} correctly interprets a range tombstone whose start and
> finish {{collectionName}} values don't match as a wrapping fragment of a
> legacy row deletion that's being interrupted by a collection deletion
> (correctly) - see
> [code|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/LegacyLayout.java#L1874-L1889].
> Quoting the comment inline:
> {code}
> // Because of the way RangeTombstoneList work, we can have a tombstone where
> only one of
> // the bound has a collectionName. That happens if we have a big tombstone A
> (spanning one
> // or multiple rows) and a collection tombstone B. In that case,
> RangeTombstoneList will
> // split this into 3 RTs: the first one from the beginning of A to the
> beginning of B,
> // then B, then a third one from the end of B to the end of A. To make this
> simpler, if
> // we detect that case we transform the 1st and 3rd tombstone so they don't
> end in the middle
> // of a row (which is still correct).
> {code}
> {{LegacyLayout#addRowTombstone()}} method then chokes when it encounters
[jira] [Commented] (CASSANDRA-15789) Rows can get duplicated in mixed major-version clusters and after full upgrade
[
https://issues.apache.org/jira/browse/CASSANDRA-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100070#comment-17100070
]
Aleksey Yeschenko commented on CASSANDRA-15789:
---
{{nodetool scrub}} fixes the issue by collapsing rows with the same clustering
into one, via the logic added in CASSANDRA-12144 to address a similar
corruption.
> Rows can get duplicated in mixed major-version clusters and after full upgrade
> --
>
> Key: CASSANDRA-15789
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15789
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination, Local/Memtable, Local/SSTable
>Reporter: Aleksey Yeschenko
>Assignee: Marcus Eriksson
>Priority: Normal
>
> In a mixed 2.X/3.X major version cluster a sequence of row deletes,
> collection overwrites, paging, and read repair can cause 3.X nodes to split
> individual rows into several rows with identical clustering. This happens due
> to 2.X paging and RT semantics, and a 3.X {{LegacyLayout}} deficiency.
> To reproduce, set up a 2-node mixed major version cluster with the following
> table:
> {code}
> CREATE TABLE distributed_test_keyspace.tlb (
> pk int,
> ck int,
> v map,
> PRIMARY KEY (pk, ck)
> );
> {code}
> 1. Using either node as the coordinator, delete the row with ck=2 using
> timestamp 1
> {code}
> DELETE FROM tbl USING TIMESTAMP 1 WHERE pk = 1 AND ck = 2;
> {code}
> 2. Using either node as the coordinator, insert the following 3 rows:
> {code}
> INSERT INTO tbl (pk, ck, v) VALUES (1, 1, {'e':'f'}) USING TIMESTAMP 3;
> INSERT INTO tbl (pk, ck, v) VALUES (1, 2, {'g':'h'}) USING TIMESTAMP 3;
> INSERT INTO tbl (pk, ck, v) VALUES (1, 3, {'i':'j'}) USING TIMESTAMP 3;
> {code}
> 3. Flush the table on both nodes
> 4. Using the 2.2 node as the coordinator, force read repar by querying the
> table with page size = 2:
>
> {code}
> SELECT * FROM tbl;
> {code}
> 5. Overwrite the row with ck=2 using timestamp 5:
> {code}
> INSERT INTO tbl (pk, ck, v) VALUES (1, 2, {'g':'h'}) USING TIMESTAMP 5;}}
> {code}
> 6. Query the 3.0 node and observe the split row:
> {code}
> cqlsh> select * from distributed_test_keyspace.tlb ;
> pk | ck | v
> ++
> 1 | 1 | {'e': 'f'}
> 1 | 2 | {'g': 'h'}
> 1 | 2 | {'k': 'l'}
> 1 | 3 | {'i': 'j'}
> {code}
> This happens because the read to query the second page ends up generating the
> following mutation for the 3.0 node:
> {code}
> ColumnFamily(tbl -{deletedAt=-9223372036854775808, localDeletion=2147483647,
> ranges=[2:v:_-2:v:!, deletedAt=2, localDeletion=1588588821]
> [2:v:!-2:!, deletedAt=1, localDeletion=1588588821]
> [3:v:_-3:v:!, deletedAt=2, localDeletion=1588588821]}-
> [2:v:63:false:1@3,])
> {code}
> Which on 3.0 side gets incorrectly deserialized as
> {code}
> Mutation(keyspace='distributed_test_keyspace', key='0001', modifications=[
> [distributed_test_keyspace.tbl] key=1
> partition_deletion=deletedAt=-9223372036854775808, localDeletion=2147483647
> columns=[[] | [v]]
> Row[info=[ts=-9223372036854775808] ]: ck=2 | del(v)=deletedAt=2,
> localDeletion=1588588821, [v[c]=d ts=3]
> Row[info=[ts=-9223372036854775808] del=deletedAt=1,
> localDeletion=1588588821 ]: ck=2 |
> Row[info=[ts=-9223372036854775808] ]: ck=3 | del(v)=deletedAt=2,
> localDeletion=1588588821
> ])
> {code}
> {{LegacyLayout}} correctly interprets a range tombstone whose start and
> finish {{collectionName}} values don't match as a wrapping fragment of a
> legacy row deletion that's being interrupted by a collection deletion
> (correctly) - see
> [code|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/LegacyLayout.java#L1874-L1889].
> Quoting the comment inline:
> {code}
> // Because of the way RangeTombstoneList work, we can have a tombstone where
> only one of
> // the bound has a collectionName. That happens if we have a big tombstone A
> (spanning one
> // or multiple rows) and a collection tombstone B. In that case,
> RangeTombstoneList will
> // split this into 3 RTs: the first one from the beginning of A to the
> beginning of B,
> // then B, then a third one from the end of B to the end of A. To make this
> simpler, if
> // we detect that case we transform the 1st and 3rd tombstone so they don't
> end in the middle
> // of a row (which is still correct).
> {code}
> {{LegacyLayout#addRowTombstone()}} method then chokes when it encounters such
> a tombstone in the middle of an existing row - having seen {{v[c]=d}} first,
> and mistakenly starts a new row, while in the middle of an existing one: (see
>
[jira] [Comment Edited] (CASSANDRA-15733) jvm dtest builder should be provided to the factory and expose state
[ https://issues.apache.org/jira/browse/CASSANDRA-15733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097102#comment-17097102 ] David Capwell edited comment on CASSANDRA-15733 at 5/5/20, 5:03 PM: | Branch | Build | Results | | trunk | [Circle CI|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=feature%2FCASSANDRA-15733] | GREEN | | 3.11 | [Circle CI|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=feature%2FCASSANDRA-15733-3.11] | GREEN | | 3.0 | [Circle CI|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=feature%2FCASSANDRA-15733-3.0] | GREEN | | 2.2 | [Circle CI|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=feature%2FCASSANDRA-15733-2.2] | UNKNOWN | was (Author: dcapwell): | Branch | Build | Results | | trunk | [Circle CI|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=feature%2FCASSANDRA-15733] | UNKNOWN | | 3.11 | [Circle CI|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=feature%2FCASSANDRA-15733-3.11] | UNKNOWN | | 3.0 | [Circle CI|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=feature%2FCASSANDRA-15733-3.0] | UNKNOWN | | 2.2 | [Circle CI|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=feature%2FCASSANDRA-15733-2.2] | UNKNOWN | > jvm dtest builder should be provided to the factory and expose state > > > Key: CASSANDRA-15733 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15733 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > Currently the builder is rather heavy and creates configs plus call the > factory with specific fields only, this isn’t that flexible and makes it > harder to have custom cluster definitions which require additional fields to > be defined. To solve this we should make the builder be sent to the factory > and expose the state so the factory can get all the fields it needs, the > factory should also be in charge of creating the configs -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15396) RandomAccessReader does not override skip or skipBytes so will do a lot of disk io when n is large
[ https://issues.apache.org/jira/browse/CASSANDRA-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Joshi updated CASSANDRA-15396: - Reviewers: Dinesh Joshi, Dinesh Joshi (was: Dinesh Joshi) Dinesh Joshi, Dinesh Joshi Status: Review In Progress (was: Patch Available) > RandomAccessReader does not override skip or skipBytes so will do a lot of > disk io when n is large > -- > > Key: CASSANDRA-15396 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15396 > Project: Cassandra > Issue Type: Improvement > Components: Local/SSTable >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Attachments: Histogram-3.png > > > RandomAccessReader does not override skip or skipBytes which becomes a > problem when the size of n (the bytes to skip) is larger than a single > buffer; in these cases we can rely on seek to avoid the extra disk io. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15766) NoSpamLogger arguments building objects on hot paths
[
https://issues.apache.org/jira/browse/CASSANDRA-15766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100139#comment-17100139
]
Jon Meredith commented on CASSANDRA-15766:
--
Thanks for the feedback. I was concerned it wasn't particularly idiomatic.
My original design was to modify the actual log call to check if the statement
should be logged, then checked if any of the format parameters were instances
of {{Supplier and if so rebuild the parameter objects array calling
}}{{get()}} where needed to retrieve the actual object before printing.
I backed off from that as I thought that would be multiple invocations of
lambdas rather than a single invocation for all arguments as well as
copying/updating the objects array (although I suppose it would be safe to
update in place assuming it was always called as a varargs). If you're going to
pay the cost of the lambda, maybe only worth it once. I need to play with JMH
to understand costs better.
I'll have a go at something more idiomatic, but for cases like this it seems
like a single lambda supplying all of the parameters would be more efficient.
{code:java}
private void onOverloaded(Message message)
{
overloadedCountUpdater.incrementAndGet(this);
int serializedSize = canonicalSize(message);
overloadedBytesUpdater.addAndGet(this, serializedSize);
noSpamLogger.warn("{} overloaded; dropping {} message (queue: {} local, {}
endpoint, {} global)",
lazyId,
FBUtilities.prettyPrintMemory(serializedSize),
FBUtilities.prettyPrintMemory(pendingBytes()),
FBUtilities.prettyPrintMemory(reserveCapacityInBytes.endpoint.using()),
FBUtilities.prettyPrintMemory(reserveCapacityInBytes.global.using()));
callbacks.onOverloaded(message, template.to);
} {code}
> NoSpamLogger arguments building objects on hot paths
>
>
> Key: CASSANDRA-15766
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15766
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Logging
>Reporter: Jon Meredith
>Assignee: Jon Meredith
>Priority: Normal
> Fix For: 4.0-rc
>
>
> NoSpamLogger is used in hot logging paths to prevent logs being overrun. For
> that to be most effective the arguments to the logger need to be cheap to
> construct. During the internode messaging refactor CASSANDRA-15066,
> performance changes to BufferPool for CASSANDRA-14416
> were accidentally reverted in the merge up from 3.11.
> Reviewing other uses since, it looks like there are a few places where the
> arguments require some form of String building.
> org.apache.cassandra.net.InboundSink#accept
> org.apache.cassandra.net.InboundMessageHandler#processCorruptFrame
> org.apache.cassandra.net.InboundMessageHandler.LargeMessage#deserialize
> org.apache.cassandra.net.OutboundConnection#onOverloaded
> org.apache.cassandra.utils.memory.BufferPool.GlobalPool#allocateMoreChunks
> Formatting arguments should either be precomputed, or if expensive they
> should be computed after the decision on whether to noSpamLog has been made.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15788) Add tests to cover CacheMetrics
Stephen Mallette created CASSANDRA-15788:
Summary: Add tests to cover CacheMetrics
Key: CASSANDRA-15788
URL: https://issues.apache.org/jira/browse/CASSANDRA-15788
Project: Cassandra
Issue Type: Improvement
Components: Test/unit
Reporter: Stephen Mallette
Assignee: Stephen Mallette
{{CacheMetrics}} and {{ChunkCacheMetrics}} do not have unit tests covering
them. {{CachingBench}} seems to provide some coverage but those tests (which
don't appear to run as part of the standard run of unit tests) are failing and
do not assert against all defined metrics, nor do they seem to assert code in
{{InstrumentingCache}} which also incremented metrics.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15788) Add tests to cover CacheMetrics
[
https://issues.apache.org/jira/browse/CASSANDRA-15788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099949#comment-17099949
]
Stephen Mallette commented on CASSANDRA-15788:
--
Changes can be found here in this branch for review:
https://github.com/apache/cassandra/compare/trunk...spmallette:CASSANDRA-15788-trunk
I wasn't sure what {{CachingBench}} was exactly. It had some assertions against
metrics but, from what I can tell, it doesn't appear to be a standard unit test
so therefore doesn't run as part of that aspect of the build. Some of those
tests were failing. I corrected the failures by better respecting caching
disabling:
https://github.com/apache/cassandra/compare/trunk...spmallette:CASSANDRA-15788-trunk#diff-b384ccc30cb742ba6be29db20028243fR169-R175
though I am uncertain if that change will have ill-effect elsewhere. I suppose
a CircleCI test run would provide confidence there. Note that as a result of
this change {{maybeWrap()}} looks like dead code:
https://github.com/apache/cassandra/compare/trunk...spmallette:CASSANDRA-15788-trunk#diff-b384ccc30cb742ba6be29db20028243fR177-R183
it doesn't appear to have usage so perhaps it can be removed/deprecated.
Anyway, I don't have complete confidence in this approach to making the
assertions pass - please let me know if that's way off.
> Add tests to cover CacheMetrics
> ---
>
> Key: CASSANDRA-15788
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15788
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/unit
>Reporter: Stephen Mallette
>Assignee: Stephen Mallette
>Priority: Normal
>
> {{CacheMetrics}} and {{ChunkCacheMetrics}} do not have unit tests covering
> them. {{CachingBench}} seems to provide some coverage but those tests (which
> don't appear to run as part of the standard run of unit tests) are failing
> and do not assert against all defined metrics, nor do they seem to assert
> code in {{InstrumentingCache}} which also incremented metrics.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15299) CASSANDRA-13304 follow-up: improve checksumming and compression in protocol v5-beta
[
https://issues.apache.org/jira/browse/CASSANDRA-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100037#comment-17100037
]
Olivier Michallat commented on CASSANDRA-15299:
---
I hear your arguments about naming, I'll let others chime in. Thankfully these
are low-level concepts that don't transpire through the driver's public API, so
at least binary compatibility is not a factor.
{quote}I suppose it might be possible to track the stream ids for a given
message
{quote}
That sounds complicated, because the party that checks the CRC is not the one
who assembled the message (client vs server or vice-versa).
For context, how common are flipped bits in your experience? Dropping the whole
connection every time might seem a bit heavy-handed, but if it's a rare
occurrence maybe that's an acceptable tradeoff.
{quote}The first of outer frame of a large message contains the inner frame
header, which specifies the length in bytes of the CQL message body.
{quote}
Got it. That also answers my follow-up question about why a corrupted first
frame is not recoverable.
{quote}we keep accumulating messages into the payload until the max
*uncompressed* size is reached [...] So there is some loss of density when
using compression
{quote}
I see. It might be worth mentioning that in the document as a hint for
implementors.
> CASSANDRA-13304 follow-up: improve checksumming and compression in protocol
> v5-beta
> ---
>
> Key: CASSANDRA-15299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15299
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Aleksey Yeschenko
>Assignee: Sam Tunnicliffe
>Priority: Normal
> Labels: protocolv5
> Fix For: 4.0-beta
>
>
> CASSANDRA-13304 made an important improvement to our native protocol: it
> introduced checksumming/CRC32 to request and response bodies. It’s an
> important step forward, but it doesn’t cover the entire stream. In
> particular, the message header is not covered by a checksum or a crc, which
> poses a correctness issue if, for example, {{streamId}} gets corrupted.
> Additionally, we aren’t quite using CRC32 correctly, in two ways:
> 1. We are calculating the CRC32 of the *decompressed* value instead of
> computing the CRC32 on the bytes written on the wire - losing the properties
> of the CRC32. In some cases, due to this sequencing, attempting to decompress
> a corrupt stream can cause a segfault by LZ4.
> 2. When using CRC32, the CRC32 value is written in the incorrect byte order,
> also losing some of the protections.
> See https://users.ece.cmu.edu/~koopman/pubs/KoopmanCRCWebinar9May2012.pdf for
> explanation for the two points above.
> Separately, there are some long-standing issues with the protocol - since
> *way* before CASSANDRA-13304. Importantly, both checksumming and compression
> operate on individual message bodies rather than frames of multiple complete
> messages. In reality, this has several important additional downsides. To
> name a couple:
> # For compression, we are getting poor compression ratios for smaller
> messages - when operating on tiny sequences of bytes. In reality, for most
> small requests and responses we are discarding the compressed value as it’d
> be smaller than the uncompressed one - incurring both redundant allocations
> and compressions.
> # For checksumming and CRC32 we pay a high overhead price for small messages.
> 4 bytes extra is *a lot* for an empty write response, for example.
> To address the correctness issue of {{streamId}} not being covered by the
> checksum/CRC32 and the inefficiency in compression and checksumming/CRC32, we
> should switch to a framing protocol with multiple messages in a single frame.
> I suggest we reuse the framing protocol recently implemented for internode
> messaging in CASSANDRA-15066 to the extent that its logic can be borrowed,
> and that we do it before native protocol v5 graduates from beta. See
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderCrc.java
> and
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderLZ4.java.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15776) python dtest regression caused by CASSANDRA-15637
[ https://issues.apache.org/jira/browse/CASSANDRA-15776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099980#comment-17099980 ] Josh McKenzie commented on CASSANDRA-15776: --- [~djoshi] / [~dcapwell] - either of you need a hand on this? > python dtest regression caused by CASSANDRA-15637 > - > > Key: CASSANDRA-15776 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15776 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Fix For: 4.0-alpha > > Time Spent: 10m > Remaining Estimate: 0h > > CASSANDRA-15637 deprecated size_estimates in favor of table_estimates to > allow for local primary range estimates (needed for MapReduce). This appears > to have caused a regression in the python dtest nodetool_test.TestNodetool. > test_refresh_size_estimates_clears_invalid_entries (as seen by [Circle > CI|https://app.circleci.com/pipelines/github/dcapwell/cassandra/255/workflows/21907001-93ed-4963-9314-6a0ac6ea0f1d/jobs/1246/tests] > and > [Jenkins|https://ci-cassandra.apache.org/job/Cassandra-trunk-dtest/56/]). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15788) Add tests to cover CacheMetrics
[
https://issues.apache.org/jira/browse/CASSANDRA-15788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Mallette updated CASSANDRA-15788:
-
Change Category: Quality Assurance
Complexity: Low Hanging Fruit
Test and Documentation Plan: additional unit tests
> Add tests to cover CacheMetrics
> ---
>
> Key: CASSANDRA-15788
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15788
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/unit
>Reporter: Stephen Mallette
>Assignee: Stephen Mallette
>Priority: Normal
>
> {{CacheMetrics}} and {{ChunkCacheMetrics}} do not have unit tests covering
> them. {{CachingBench}} seems to provide some coverage but those tests (which
> don't appear to run as part of the standard run of unit tests) are failing
> and do not assert against all defined metrics, nor do they seem to assert
> code in {{InstrumentingCache}} which also incremented metrics.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15789) TBD
Aleksey Yeschenko created CASSANDRA-15789: - Summary: TBD Key: CASSANDRA-15789 URL: https://issues.apache.org/jira/browse/CASSANDRA-15789 Project: Cassandra Issue Type: Bug Reporter: Aleksey Yeschenko Assignee: Marcus Eriksson -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
[
https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100198#comment-17100198
]
David Capwell commented on CASSANDRA-15778:
---
Created a branch to ignore value field for compact storage:
https://github.com/dcapwell/cassandra/tree/corruption/CASSANDRA-15778. Running
through testing now.
> CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
> -
>
> Key: CASSANDRA-15778
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15778
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction, Local/SSTable
>Reporter: Sumanth Pasupuleti
>Priority: Normal
> Fix For: 3.0.x
>
>
> Below is the exception with stack trace. This issue is consistently
> reproduce-able.
> {code:java}
> ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01
> 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception
> on thread
> Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException:
> Corrupted:
> /mnt/data/cassandra/data// at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:180)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:176)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:341)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:47)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_231] at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165)
>
[jira] [Commented] (CASSANDRA-15748) Provide ability to configure IAuditLogger
[ https://issues.apache.org/jira/browse/CASSANDRA-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100234#comment-17100234 ] Jon Meredith commented on CASSANDRA-15748: -- [~stefan.miklosovic] Would you object to me assigning this to a later fix version? I don't think it should prevent us from cutting a first 4.0-beta. > Provide ability to configure IAuditLogger > - > > Key: CASSANDRA-15748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15748 > Project: Cassandra > Issue Type: Improvement > Components: Local/Other >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0-alpha > > Time Spent: 10m > Remaining Estimate: 0h > > Currently there is a parameterless constructor expected for IAuditLogger > instances. There is not any way how to "inject" configuration properties from > cassandra.yaml into these concrete classes. This would be very handy for > custom IAuditLogger implementations. > > PR: [https://github.com/apache/cassandra/pull/555] > [|https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15748] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15790) EmptyType doesn't override writeValue so could attempt to write bytes when expected not to
[
https://issues.apache.org/jira/browse/CASSANDRA-15790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Capwell updated CASSANDRA-15790:
--
Description:
EmptyType.writeValues is defined here
https://github.com/apache/cassandra/blob/e394dc0bb32f612a476269010930c617dd1ed3cb/src/java/org/apache/cassandra/db/marshal/AbstractType.java#L407-L414
{code}
public void writeValue(ByteBuffer value, DataOutputPlus out) throws IOException
{
assert value.hasRemaining();
if (valueLengthIfFixed() >= 0)
out.write(value);
else
ByteBufferUtil.writeWithVIntLength(value, out);
}
{code}
This is fine when the value is empty as the write of empty no-ops (the
readValue also noops since the length is 0), but if the value is not empty
(possible during upgrades or random bugs) then this could silently cause
corruption; ideally this should throw a exception if the ByteBuffer has data.
This was called from
org.apache.cassandra.db.rows.BufferCell.Serializer#serialize, here we check to
see if data is present or not and update the flags. If data is present then
and only then do we call type.writeValue (which requires bytes is not empty).
The problem is that EmptyType never expects writes to happen, but it still
writes them; and does not read them (since it says it is fixed length of 0, so
does read(buffer, 0)).
was:
EmptyType.writeValues is defined here
https://github.com/apache/cassandra/blob/e394dc0bb32f612a476269010930c617dd1ed3cb/src/java/org/apache/cassandra/db/marshal/AbstractType.java#L407-L414
{code}
public void writeValue(ByteBuffer value, DataOutputPlus out) throws IOException
{
assert value.hasRemaining();
if (valueLengthIfFixed() >= 0)
out.write(value);
else
ByteBufferUtil.writeWithVIntLength(value, out);
}
{code}
This is fine when the value is empty as the write of empty no-ops (the
readValue also noops since the length is 0), but if the value is not empty
(possible during upgrades or random bugs) then this could silently cause
corruption; ideally this should throw a exception if the ByteBuffer has data.
> EmptyType doesn't override writeValue so could attempt to write bytes when
> expected not to
> --
>
> Key: CASSANDRA-15790
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15790
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Semantics
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>
> EmptyType.writeValues is defined here
> https://github.com/apache/cassandra/blob/e394dc0bb32f612a476269010930c617dd1ed3cb/src/java/org/apache/cassandra/db/marshal/AbstractType.java#L407-L414
> {code}
> public void writeValue(ByteBuffer value, DataOutputPlus out) throws
> IOException
> {
> assert value.hasRemaining();
> if (valueLengthIfFixed() >= 0)
> out.write(value);
> else
> ByteBufferUtil.writeWithVIntLength(value, out);
> }
> {code}
> This is fine when the value is empty as the write of empty no-ops (the
> readValue also noops since the length is 0), but if the value is not empty
> (possible during upgrades or random bugs) then this could silently cause
> corruption; ideally this should throw a exception if the ByteBuffer has data.
> This was called from
> org.apache.cassandra.db.rows.BufferCell.Serializer#serialize, here we check
> to see if data is present or not and update the flags. If data is present
> then and only then do we call type.writeValue (which requires bytes is not
> empty). The problem is that EmptyType never expects writes to happen, but it
> still writes them; and does not read them (since it says it is fixed length
> of 0, so does read(buffer, 0)).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15790) EmptyType doesn't override writeValue so could attempt to write bytes when expected not to
[
https://issues.apache.org/jira/browse/CASSANDRA-15790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Blake Eggleston updated CASSANDRA-15790:
Authors: Blake Eggleston, David Capwell (was: David Capwell)
> EmptyType doesn't override writeValue so could attempt to write bytes when
> expected not to
> --
>
> Key: CASSANDRA-15790
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15790
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Semantics
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>
> EmptyType.writeValues is defined here
> https://github.com/apache/cassandra/blob/e394dc0bb32f612a476269010930c617dd1ed3cb/src/java/org/apache/cassandra/db/marshal/AbstractType.java#L407-L414
> {code}
> public void writeValue(ByteBuffer value, DataOutputPlus out) throws
> IOException
> {
> assert value.hasRemaining();
> if (valueLengthIfFixed() >= 0)
> out.write(value);
> else
> ByteBufferUtil.writeWithVIntLength(value, out);
> }
> {code}
> This is fine when the value is empty as the write of empty no-ops (the
> readValue also noops since the length is 0), but if the value is not empty
> (possible during upgrades or random bugs) then this could silently cause
> corruption; ideally this should throw a exception if the ByteBuffer has data.
> This was called from
> org.apache.cassandra.db.rows.BufferCell.Serializer#serialize, here we check
> to see if data is present or not and update the flags. If data is present
> then and only then do we call type.writeValue (which requires bytes is not
> empty). The problem is that EmptyType never expects writes to happen, but it
> still writes them; and does not read them (since it says it is fixed length
> of 0, so does read(buffer, 0)).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15773) Add test to cover metrics related to the BufferPool
[
https://issues.apache.org/jira/browse/CASSANDRA-15773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Capwell updated CASSANDRA-15773:
--
Reviewers: David Capwell, Dinesh Joshi (was: Dinesh Joshi)
> Add test to cover metrics related to the BufferPool
> ---
>
> Key: CASSANDRA-15773
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15773
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/unit
>Reporter: Stephen Mallette
>Assignee: Stephen Mallette
>Priority: Normal
>
> At this time there do not appear to be unit tests to validate
> {{BufferPoolMetrics}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15214) OOMs caught and not rethrown
[ https://issues.apache.org/jira/browse/CASSANDRA-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100330#comment-17100330 ] Joey Lynch commented on CASSANDRA-15214: > Since one should be able to trigger the OOM by looping allocating large chunk > of memory, e.g. array, in the java code. What is the benefit of doing it so > using jvmquake? I can see that in the killer_thread callback function, it > also does long array allocation once notified by the gc callback. Ah sorry I was not clear. I think the JVMStabilityDetector (which we call into via inspectThrowable all over the place) should allocate the long array if we see an OutOfMemoryError with message "Direct buffer memory", in turn triggering a Heap OOM (which will trigger the normal resource exhausted mechanism). Since we're not out of _heap_ memory we can trust that JVMStatbilityDetector can run. I guess my proposal is to include jvmquake by default for linux deployments (I can add more architectures if we want more, easy to opt out), and if JVMStabilityDetector sees a "Direct buffer memory" OOM it should force the JVM into a heap OOM, triggering jvmquake's resource exhausted handler. This setup would guarantee that C* dies (and produces a heap dump) if any of the following conditions hold: * The JVM is out of heap memory * The JVM has accumulated 30s of GC debt with 1:5 runtime weight (meaning that we had <85% throughput for at least 30s): aka "GC spirals of death" * The JVM is out of metaspace memory * The JVM is out of threads * (best effort, likely true) The JVM is out of native memory (so basically C* is using 2x the heap size) -> triggers a heap oom -> triggers the first case Unlike the built in JVM options jvmquake really actually works in these edge cases (not only is there a test suite to prove it that the built in Java options don't work but if you run inside the heap you fundamentally can't guarantee you will run, e.g. why the kill -9 approach never really works). > OOMs caught and not rethrown > > > Key: CASSANDRA-15214 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15214 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client, Messaging/Internode >Reporter: Benedict Elliott Smith >Priority: Normal > Fix For: 4.0, 4.0-rc > > Attachments: oom-experiments.zip > > > Netty (at least, and perhaps elsewhere in Executors) catches all exceptions, > so presently there is no way to ensure that an OOM reaches the JVM handler to > trigger a crash/heapdump. > It may be that the simplest most consistent way to do this would be to have a > single thread spawned at startup that waits for any exceptions we must > propagate to the Runtime. > We could probably submit a patch upstream to Netty, but for a guaranteed > future proof approach, it may be worth paying the cost of a single thread. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15790) EmptyType doesn't override writeValue so could attempt to write bytes when expected not to
[
https://issues.apache.org/jira/browse/CASSANDRA-15790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Capwell updated CASSANDRA-15790:
--
Fix Version/s: 4.0-alpha
3.11.x
3.0.x
> EmptyType doesn't override writeValue so could attempt to write bytes when
> expected not to
> --
>
> Key: CASSANDRA-15790
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15790
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Semantics
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-alpha
>
>
> EmptyType.writeValues is defined here
> https://github.com/apache/cassandra/blob/e394dc0bb32f612a476269010930c617dd1ed3cb/src/java/org/apache/cassandra/db/marshal/AbstractType.java#L407-L414
> {code}
> public void writeValue(ByteBuffer value, DataOutputPlus out) throws
> IOException
> {
> assert value.hasRemaining();
> if (valueLengthIfFixed() >= 0)
> out.write(value);
> else
> ByteBufferUtil.writeWithVIntLength(value, out);
> }
> {code}
> This is fine when the value is empty as the write of empty no-ops (the
> readValue also noops since the length is 0), but if the value is not empty
> (possible during upgrades or random bugs) then this could silently cause
> corruption; ideally this should throw a exception if the ByteBuffer has data.
> This was called from
> org.apache.cassandra.db.rows.BufferCell.Serializer#serialize, here we check
> to see if data is present or not and update the flags. If data is present
> then and only then do we call type.writeValue (which requires bytes is not
> empty). The problem is that EmptyType never expects writes to happen, but it
> still writes them; and does not read them (since it says it is fixed length
> of 0, so does read(buffer, 0)).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15748) Provide ability to configure IAuditLogger
[ https://issues.apache.org/jira/browse/CASSANDRA-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Chella updated CASSANDRA-15748: - Reviewers: Marcus Eriksson, Vinay Chella (was: Marcus Eriksson) > Provide ability to configure IAuditLogger > - > > Key: CASSANDRA-15748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15748 > Project: Cassandra > Issue Type: Improvement > Components: Local/Other >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > Currently there is a parameterless constructor expected for IAuditLogger > instances. There is not any way how to "inject" configuration properties from > cassandra.yaml into these concrete classes. This would be very handy for > custom IAuditLogger implementations. > > PR: [https://github.com/apache/cassandra/pull/555] > [|https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15748] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15783) test_optimized_primary_range_repair - transient_replication_test.TestTransientReplication
[ https://issues.apache.org/jira/browse/CASSANDRA-15783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100335#comment-17100335 ] Ekaterina Dimitrova commented on CASSANDRA-15783: - This test started failing after CASSANDRA-15657. Will investigate it further tomorrow. > test_optimized_primary_range_repair - > transient_replication_test.TestTransientReplication > - > > Key: CASSANDRA-15783 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15783 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: Ekaterina Dimitrova >Assignee: Ekaterina Dimitrova >Priority: Normal > Fix For: 4.0-alpha > > > Dtest failure. > Example: > https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/118/workflows/9e57522d-52fa-4d44-88d8-5cec0e87f517/jobs/585/tests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13474) Cassandra pluggable storage engine
[ https://issues.apache.org/jira/browse/CASSANDRA-13474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100356#comment-17100356 ] Jeremiah Jordan commented on CASSANDRA-13474: - No this is not planned for 4.0. > Cassandra pluggable storage engine > -- > > Key: CASSANDRA-13474 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13474 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Core >Reporter: Dikang Gu >Priority: Normal > > Instagram is working on a project to significantly reduce Cassandra's tail > latency, by implementing a new storage engine on top of RocksDB, named > Rocksandra. > We started a prototype of single column (key-value) use case, and then > implemented a full design to support most of the data types and data models > in Cassandra, as well as streaming. > After a year of development and testing, we have rolled out the Rocksandra > project to our internal deployments, and observed 3-4X reduction on P99 read > latency in general, even more than 10 times reduction for some use cases. > We published a blog post about the wins and the benchmark metrics on AWS > environment. > https://engineering.instagram.com/open-sourcing-a-10x-reduction-in-apache-cassandra-tail-latency-d64f86b43589 > I think the biggest performance win comes from we get rid of most Java > garbages created by current read/write path and compactions, which reduces > the JVM overhead and makes the latency to be more predictable. > We are very excited about the potential performance gain. As the next step, I > propose to make the Cassandra storage engine to be pluggable (like Mysql and > MongoDB), and we are very interested in providing RocksDB as one storage > option with more predictable performance, together with community. > Design doc for pluggable storage engine: > https://docs.google.com/document/d/1suZlvhzgB6NIyBNpM9nxoHxz_Ri7qAm-UEO8v8AIFsc/edit -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15783) test_optimized_primary_range_repair - transient_replication_test.TestTransientReplication
[ https://issues.apache.org/jira/browse/CASSANDRA-15783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100360#comment-17100360 ] Dinesh Joshi commented on CASSANDRA-15783: -- I don't see the stack trace. Can you please update the ticket with more information on the failure? > test_optimized_primary_range_repair - > transient_replication_test.TestTransientReplication > - > > Key: CASSANDRA-15783 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15783 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: Ekaterina Dimitrova >Assignee: Ekaterina Dimitrova >Priority: Normal > Fix For: 4.0-alpha > > > Dtest failure. > Example: > https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/118/workflows/9e57522d-52fa-4d44-88d8-5cec0e87f517/jobs/585/tests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15657) Improve zero-copy-streaming containment check by using file sections
[ https://issues.apache.org/jira/browse/CASSANDRA-15657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100332#comment-17100332 ] Ekaterina Dimitrova edited comment on CASSANDRA-15657 at 5/5/20, 11:53 PM: --- [~jasonstack] [~djoshi] transient_replication_test.py::TestTransientReplication::test_optimized_primary_range_repair started failing after this patch, any thoughts? I opened CASSANDRA-15783 and I will be working on it tomorrow was (Author: e.dimitrova): [~jasonstack] [~djoshi] transient_replication_test.py::TestTransientReplication::test_optimized_primary_range_repair started failing after this patch, any thoughts? I opened CASSANDRA-15783 > Improve zero-copy-streaming containment check by using file sections > > > Key: CASSANDRA-15657 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15657 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Streaming and Messaging >Reporter: ZhaoYang >Assignee: ZhaoYang >Priority: Normal > Fix For: 4.0 > > > Currently zero copy streaming is only enabled for leveled-compaction strategy > and it checks if all keys in the sstables are included in the transferred > ranges. > This is very inefficient. The containment check can be improved by checking > if transferred sections (the transferred file positions) cover entire sstable. > I also enabled ZCS for all compaction strategies since the new containment > check is very fast.. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15773) Add test to cover metrics related to the BufferPool
[
https://issues.apache.org/jira/browse/CASSANDRA-15773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Capwell updated CASSANDRA-15773:
--
Status: Changes Suggested (was: Review In Progress)
> Add test to cover metrics related to the BufferPool
> ---
>
> Key: CASSANDRA-15773
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15773
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/unit
>Reporter: Stephen Mallette
>Assignee: Stephen Mallette
>Priority: Normal
>
> At this time there do not appear to be unit tests to validate
> {{BufferPoolMetrics}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15773) Add test to cover metrics related to the BufferPool
[
https://issues.apache.org/jira/browse/CASSANDRA-15773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100342#comment-17100342
]
David Capwell commented on CASSANDRA-15773:
---
*
https://github.com/apache/cassandra/compare/trunk...spmallette:CASSANDRA-15773-trunk#diff-790a43fcd6ed69f4122cd0fe205c847bR72.
leaking, should cleanup after the test is over; same for the other test
*
https://github.com/apache/cassandra/compare/trunk...spmallette:CASSANDRA-15773-trunk#diff-790a43fcd6ed69f4122cd0fe205c847bR75
can this be >= totalBytesRequestedFromPool? Since we don't free, and the
metric is how many chunks are leant out, >= totalBytesRequestedFromPool should
hold true and would be more accurate than > 0
* the test makes an assumption at buffer pool size, which may not be true in
CI, see [1] pool size calculation. For this reason the tests can be flaky
under environments with less resources. One way to deal with this is to define
file_cache_size_in_mb in test/conf/cassandra.yaml, that way memory is constant
for all environments, and the loops can be adjusted to it.
Overall the patch LGTM, mostly small stuff.
[1] -
{code}
if (conf.file_cache_size_in_mb == null)
conf.file_cache_size_in_mb = Math.min(512, (int)
(Runtime.getRuntime().maxMemory() / (4 * 1048576)));
{code}
> Add test to cover metrics related to the BufferPool
> ---
>
> Key: CASSANDRA-15773
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15773
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/unit
>Reporter: Stephen Mallette
>Assignee: Stephen Mallette
>Priority: Normal
>
> At this time there do not appear to be unit tests to validate
> {{BufferPoolMetrics}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15657) Improve zero-copy-streaming containment check by using file sections
[ https://issues.apache.org/jira/browse/CASSANDRA-15657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100332#comment-17100332 ] Ekaterina Dimitrova commented on CASSANDRA-15657: - [~jasonstack] [~djoshi] transient_replication_test.py::TestTransientReplication::test_optimized_primary_range_repair started failing after this patch, any thoughts? I opened CASSANDRA-15783 > Improve zero-copy-streaming containment check by using file sections > > > Key: CASSANDRA-15657 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15657 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Streaming and Messaging >Reporter: ZhaoYang >Assignee: ZhaoYang >Priority: Normal > Fix For: 4.0 > > > Currently zero copy streaming is only enabled for leveled-compaction strategy > and it checks if all keys in the sstables are included in the transferred > ranges. > This is very inefficient. The containment check can be improved by checking > if transferred sections (the transferred file positions) cover entire sstable. > I also enabled ZCS for all compaction strategies since the new containment > check is very fast.. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15772) DOC - Improve tpstats documentation
[
https://issues.apache.org/jira/browse/CASSANDRA-15772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100233#comment-17100233
]
Jon Meredith commented on CASSANDRA-15772:
--
[~Mark Curtis] Would you object to me assigning this to a later fix version? I
don't think it should prevent us from cutting a first 4.0-beta.
> DOC - Improve tpstats documentation
> ---
>
> Key: CASSANDRA-15772
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15772
> Project: Cassandra
> Issue Type: Improvement
> Components: Documentation/Website
>Reporter: Mark Curtis
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> h2. Background
> The docs for {{nodetool tpstats}} gives the user a good overview of the
> correct command format, but lack an explanation of the message types.
> As of Cassandra4.0 and CASSANDRA-15066 the Message types have expanded a lot
> so this becomes quite overwhelming (particularly for new users).
> I'd propose we offer a simple explanation of these grouping them into logical
> groups (see CASSANDRA-15771 which I opened for the formatting output).
> The github code for the {{Verb.java}} class does give the user some idea of
> what these are but not everyone will go looking for that.
> h2. Example
> Code for ref
> [https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/Verb.java]
> I tried to sort some of the tpstats output into categories and made a first
> attempt at an explanation here:
> {code:java}
> BatchesBATCH_REMOVE_REQ
> BatchesBATCH_REMOVE_RSP
> BatchesBATCH_STORE_REQ
> BatchesBATCH_STORE_RSP
> - These are batches, I'd assume the store / removes are as we add and process
> them?
>
> Counters COUNTER_MUTATION_REQ
> Counters COUNTER_MUTATION_RSP
> - Counter writes
> Deprecated INTERNAL_RSP
> Deprecated REQUEST_RSP
> - Some of these are shown as deprecated in the code. Not sure what we should
> add here
>
> Gossip ECHO_REQ
> Gossip ECHO_RSP
> Gossip GOSSIP_DIGEST_ACK
> Gossip GOSSIP_DIGEST_ACK2
> Gossip GOSSIP_DIGEST_SYN
> Gossip GOSSIP_SHUTDOWN
> Gossip PING_REQ
> Gossip PING_RSP
> - Gossip, probably we'd refer to a more detailed page on gossip from here
>
> Hints HINT_REQ
> Hints HINT_RSP
> - Hints!
>
> LWTs PAXOS_COMMIT_REQ
> LWTs PAXOS_COMMIT_RSP
> LWTs PAXOS_PREPARE_REQ
> LWTs PAXOS_PREPARE_RSP
> LWTs PAXOS_PROPOSE_REQ
> LWTs PAXOS_PROPOSE_RSP
> - Paxos phases request / response
>
> Migration SCHEMA_PULL_REQ
> Migration SCHEMA_PULL_RSP
> Migration SCHEMA_PUSH_REQ
> Migration SCHEMA_PUSH_RSP
> Migration SCHEMA_VERSION_REQ
> Migration SCHEMA_VERSION_RSP
> - Schema migration messaging (i.e. adding new tables)
> Misc REPLICATION_DONE_REQ
> Misc REPLICATION_DONE_RSP
> Misc SNAPSHOT_MSG
> Misc SNAPSHOT_REQ
> Misc SNAPSHOT_RSP
> - Unsure of these, could the snapshots be for simply nodetool snapshot
> requests like on truncates?
>
> Reads RANGE_REQ
> Reads RANGE_RSP
> Reads READ_REPAIR_REQ
> Reads READ_REPAIR_RSP
> Reads READ_REQ
> Reads READ_RSP
> - All types of read messages
>
> Repair CLEANUP_MSG
> Repair FAILED_SESSION_MSG
> Repair FAILURE_RSP
> Repair FINALIZE_COMMIT_MSG
> Repair FINALIZE_PROMISE_MSG
> Repair FINALIZE_PROPOSE_MSG
> Repair PREPARE_CONSISTENT_REQ
> Repair PREPARE_CONSISTENT_RSP
> Repair PREPARE_MSG
> Repair REPAIR_RSP
> Repair STATUS_REQ
> Repair STATUS_RSP
> Repair SYNC_REQ
> Repair SYNC_RSP
> Repair VALIDATION_REQ
> Repair VALIDATION_RSP
> Repair ASYMMETRIC_SYNC_REQ
> - Repairs, and phases, I think we'd need a bit more explanation here.
> Validation, Sync etc ok. But unsure on these other types
>
> Writes MUTATION_REQ
> Writes MUTATION_RSP
> Writes TRUNCATE_REQ
> Writes TRUNCATE_RSP
> - All types of write messages{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail:
[jira] [Commented] (CASSANDRA-15790) EmptyType doesn't override writeValue so could attempt to write bytes when expected not to
[
https://issues.apache.org/jira/browse/CASSANDRA-15790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100273#comment-17100273
]
David Capwell commented on CASSANDRA-15790:
---
CI status is YELLOW (py dtest failed with known issues)
> EmptyType doesn't override writeValue so could attempt to write bytes when
> expected not to
> --
>
> Key: CASSANDRA-15790
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15790
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Semantics
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>
> EmptyType.writeValues is defined here
> https://github.com/apache/cassandra/blob/e394dc0bb32f612a476269010930c617dd1ed3cb/src/java/org/apache/cassandra/db/marshal/AbstractType.java#L407-L414
> {code}
> public void writeValue(ByteBuffer value, DataOutputPlus out) throws
> IOException
> {
> assert value.hasRemaining();
> if (valueLengthIfFixed() >= 0)
> out.write(value);
> else
> ByteBufferUtil.writeWithVIntLength(value, out);
> }
> {code}
> This is fine when the value is empty as the write of empty no-ops (the
> readValue also noops since the length is 0), but if the value is not empty
> (possible during upgrades or random bugs) then this could silently cause
> corruption; ideally this should throw a exception if the ByteBuffer has data.
> This was called from
> org.apache.cassandra.db.rows.BufferCell.Serializer#serialize, here we check
> to see if data is present or not and update the flags. If data is present
> then and only then do we call type.writeValue (which requires bytes is not
> empty). The problem is that EmptyType never expects writes to happen, but it
> still writes them; and does not read them (since it says it is fixed length
> of 0, so does read(buffer, 0)).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15766) NoSpamLogger arguments building objects on hot paths
[
https://issues.apache.org/jira/browse/CASSANDRA-15766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100210#comment-17100210
]
Benedict Elliott Smith commented on CASSANDRA-15766:
{quote}The lambda is compiled into a static method in the byte code
{quote}
This is only true of non-capturing lambdas
> NoSpamLogger arguments building objects on hot paths
>
>
> Key: CASSANDRA-15766
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15766
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Logging
>Reporter: Jon Meredith
>Assignee: Jon Meredith
>Priority: Normal
> Fix For: 4.0-rc
>
>
> NoSpamLogger is used in hot logging paths to prevent logs being overrun. For
> that to be most effective the arguments to the logger need to be cheap to
> construct. During the internode messaging refactor CASSANDRA-15066,
> performance changes to BufferPool for CASSANDRA-14416
> were accidentally reverted in the merge up from 3.11.
> Reviewing other uses since, it looks like there are a few places where the
> arguments require some form of String building.
> org.apache.cassandra.net.InboundSink#accept
> org.apache.cassandra.net.InboundMessageHandler#processCorruptFrame
> org.apache.cassandra.net.InboundMessageHandler.LargeMessage#deserialize
> org.apache.cassandra.net.OutboundConnection#onOverloaded
> org.apache.cassandra.utils.memory.BufferPool.GlobalPool#allocateMoreChunks
> Formatting arguments should either be precomputed, or if expensive they
> should be computed after the decision on whether to noSpamLog has been made.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15766) NoSpamLogger arguments building objects on hot paths
[
https://issues.apache.org/jira/browse/CASSANDRA-15766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100210#comment-17100210
]
Benedict Elliott Smith edited comment on CASSANDRA-15766 at 5/5/20, 7:34 PM:
-
{quote}The lambda is compiled into a static method in the byte code
{quote}
This is only true of non-capturing lambdas
edit: actually, I'm not sure what you mean here - the method _body_ will be
compiled to a static method, but the lambda will also behave like a static
_property_ if it captures no variables.
In many cases, though, even a lambda that must be allocated may be elided
entirely if the method you pass it to is inlined, and escape analysis can do
its magic - but there are a lot of 'ifs' there.
was (Author: benedict):
{quote}The lambda is compiled into a static method in the byte code
{quote}
This is only true of non-capturing lambdas
> NoSpamLogger arguments building objects on hot paths
>
>
> Key: CASSANDRA-15766
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15766
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Logging
>Reporter: Jon Meredith
>Assignee: Jon Meredith
>Priority: Normal
> Fix For: 4.0-rc
>
>
> NoSpamLogger is used in hot logging paths to prevent logs being overrun. For
> that to be most effective the arguments to the logger need to be cheap to
> construct. During the internode messaging refactor CASSANDRA-15066,
> performance changes to BufferPool for CASSANDRA-14416
> were accidentally reverted in the merge up from 3.11.
> Reviewing other uses since, it looks like there are a few places where the
> arguments require some form of String building.
> org.apache.cassandra.net.InboundSink#accept
> org.apache.cassandra.net.InboundMessageHandler#processCorruptFrame
> org.apache.cassandra.net.InboundMessageHandler.LargeMessage#deserialize
> org.apache.cassandra.net.OutboundConnection#onOverloaded
> org.apache.cassandra.utils.memory.BufferPool.GlobalPool#allocateMoreChunks
> Formatting arguments should either be precomputed, or if expensive they
> should be computed after the decision on whether to noSpamLog has been made.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15782) Compression test failure
[ https://issues.apache.org/jira/browse/CASSANDRA-15782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Lynch updated CASSANDRA-15782: --- Since Version: 4.0-alpha Source Control Link: https://github.com/apache/cassandra-dtest/commit/da7fcefb16d16af8924cda35c0a6a63ad553694f Resolution: Fixed Status: Resolved (was: Ready to Commit) Committed as [da7fcefb|https://github.com/apache/cassandra-dtest/commit/da7fcefb16d16af8924cda35c0a6a63ad553694f] Note that I considered this might impact pre trunk dtest runs but this fix should be safe since compacting in 2.x/3.x should also produce a deflate table (as flush does). If I'm wrong I'll fix it. Thanks [~Bereng] for the report, sorry for missing that. > Compression test failure > > > Key: CASSANDRA-15782 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15782 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: Berenguer Blasi >Assignee: Joey Lynch >Priority: Normal > Fix For: 4.0-alpha > > > On CASSANDRA-15560 compression test failed. This was bisected to > [9c1bbf3|https://github.com/apache/cassandra/commit/9c1bbf3ac913f9bdf7a0e0922106804af42d2c1e] > from CASSANDRA-15379. > Full details here > CC/ [~jolynch] in case he can spot it quick. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15685) flaky testWithMismatchingPending - org.apache.cassandra.distributed.test.PreviewRepairTest
[
https://issues.apache.org/jira/browse/CASSANDRA-15685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie updated CASSANDRA-15685:
--
Fix Version/s: (was: 4.0-beta)
4.0-alpha
> flaky testWithMismatchingPending -
> org.apache.cassandra.distributed.test.PreviewRepairTest
> --
>
> Key: CASSANDRA-15685
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15685
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Kevin Gallardo
>Assignee: David Capwell
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-alpha
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Observed in:
> https://app.circleci.com/pipelines/github/newkek/cassandra/34/workflows/1c6b157d-13c3-48a9-85fb-9fe8c153256b/jobs/191/tests
> Failure:
> {noformat}
> testWithMismatchingPending -
> org.apache.cassandra.distributed.test.PreviewRepairTest
> junit.framework.AssertionFailedError
> at
> org.apache.cassandra.distributed.test.PreviewRepairTest.testWithMismatchingPending(PreviewRepairTest.java:97)
> {noformat}
> [Circle
> CI|https://circleci.com/gh/dcapwell/cassandra/tree/bug%2FCASSANDRA-15685]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15783) test_optimized_primary_range_repair - transient_replication_test.TestTransientReplication
[ https://issues.apache.org/jira/browse/CASSANDRA-15783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-15783: -- Fix Version/s: (was: 4.0-beta) (was: 4.0) 4.0-alpha > test_optimized_primary_range_repair - > transient_replication_test.TestTransientReplication > - > > Key: CASSANDRA-15783 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15783 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: Ekaterina Dimitrova >Assignee: Ekaterina Dimitrova >Priority: Normal > Fix For: 4.0-alpha > > > Dtest failure. > Example: > https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/118/workflows/9e57522d-52fa-4d44-88d8-5cec0e87f517/jobs/585/tests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15676) flaky test testWriteUnknownResult- org.apache.cassandra.distributed.test.CasWriteTest
[
https://issues.apache.org/jira/browse/CASSANDRA-15676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie updated CASSANDRA-15676:
--
Fix Version/s: (was: 4.0-beta)
4.0-alpha
> flaky test testWriteUnknownResult-
> org.apache.cassandra.distributed.test.CasWriteTest
> -
>
> Key: CASSANDRA-15676
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15676
> Project: Cassandra
> Issue Type: Task
> Components: Test/dtest
>Reporter: Kevin Gallardo
>Assignee: Gianluca Righetto
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> Failure observed in:
> https://app.circleci.com/pipelines/github/newkek/cassandra/33/workflows/54007cf7-4424-4ec1-9655-665f6044e6d1/jobs/187/tests
> {noformat}
> testWriteUnknownResult - org.apache.cassandra.distributed.test.CasWriteTest
> junit.framework.AssertionFailedError: Expecting cause to be
> CasWriteUncertainException
> at
> org.apache.cassandra.distributed.test.CasWriteTest.testWriteUnknownResult(CasWriteTest.java:257)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15313) Fix flaky - ChecksummingTransformerTest - org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
[
https://issues.apache.org/jira/browse/CASSANDRA-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie updated CASSANDRA-15313:
--
Fix Version/s: (was: 4.0-beta)
4.0-alpha
> Fix flaky - ChecksummingTransformerTest -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> ---
>
> Key: CASSANDRA-15313
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15313
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Vinay Chella
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0-alpha
>
> Attachments: CASSANDRA-15313-hack.patch
>
>
> During the recent runs, this test appears to be flaky.
> Example failure:
> [https://circleci.com/gh/vinaykumarchella/cassandra/459#tests/containers/94]
> corruptionCausesFailure-compression -
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest
> {code:java}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> at org.quicktheories.impl.Precursor.(Precursor.java:17)
> at
> org.quicktheories.impl.ConcreteDetachedSource.(ConcreteDetachedSource.java:8)
> at
> org.quicktheories.impl.ConcreteDetachedSource.detach(ConcreteDetachedSource.java:23)
> at org.quicktheories.generators.Retry.generate(CodePoints.java:51)
> at
> org.quicktheories.generators.Generate.lambda$intArrays$10(Generate.java:190)
> at
> org.quicktheories.generators.Generate$$Lambda$17/1847008471.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$mix$10(Gen.java:184)
> at org.quicktheories.core.Gen$$Lambda$45/802243390.generate(Unknown
> Source)
> at org.quicktheories.core.Gen.lambda$flatMap$5(Gen.java:93)
> at org.quicktheories.core.Gen$$Lambda$48/363509958.generate(Unknown
> Source)
> at
> org.quicktheories.dsl.TheoryBuilder4.lambda$prgnToTuple$12(TheoryBuilder4.java:188)
> at
> org.quicktheories.dsl.TheoryBuilder4$$Lambda$40/2003496028.generate(Unknown
> Source)
> at org.quicktheories.core.DescribingGenerator.generate(Gen.java:255)
> at org.quicktheories.core.FilteredGenerator.generate(Gen.java:225)
> at org.quicktheories.core.Gen.lambda$map$0(Gen.java:36)
> at org.quicktheories.core.Gen$$Lambda$20/71399214.generate(Unknown
> Source)
> at org.quicktheories.impl.Core.generate(Core.java:150)
> at org.quicktheories.impl.Core.shrink(Core.java:103)
> at org.quicktheories.impl.Core.run(Core.java:39)
> at org.quicktheories.impl.TheoryRunner.check(TheoryRunner.java:35)
> at org.quicktheories.dsl.TheoryBuilder4.check(TheoryBuilder4.java:150)
> at
> org.quicktheories.dsl.TheoryBuilder4.checkAssert(TheoryBuilder4.java:162)
> at
> org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure(ChecksummingTransformerTest.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-dtest] branch master updated: Fix compression_test.TestCompression regression
This is an automated email from the ASF dual-hosted git repository.
jolynch pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-dtest.git
The following commit(s) were added to refs/heads/master by this push:
new da7fcef Fix compression_test.TestCompression regression
da7fcef is described below
commit da7fcefb16d16af8924cda35c0a6a63ad553694f
Author: Joseph Lynch
AuthorDate: Mon May 4 10:59:35 2020 -0700
Fix compression_test.TestCompression regression
CASSANDRA-15379 changed behavior of how we flush for deflate, namely we
flush LZ4 and then compact to deflate off the hot path. Updated this
test to take this into account.
Patch by Joseph Lynch; Reviewed by Berenguer Blasi for CASSANDRA-15782
---
compression_test.py | 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/compression_test.py b/compression_test.py
index d865ba2..4688b72 100644
--- a/compression_test.py
+++ b/compression_test.py
@@ -103,7 +103,11 @@ class TestCompression(TestHelper):
for n in range(0, 100):
session.execute("insert into compression_opts_table (id) values
(uuid());")
-sstables = self.flush('compression_opts_table')
+self.flush('compression_opts_table')
+# Due to CASSANDRA-15379 we have to compact to get the actual table
+# compression to take effect since deflate is a slow compressor
+self.perform_node_tool_cmd(cmd='compact',
table='compression_opts_table', indexes=list())
+sstables = self.get_sstables(table='compression_opts_table',
indexes=list())
sstable_paths = self.get_table_paths('compression_opts_table')
found = False
for sstable_path in sstable_paths:
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15748) Provide ability to configure IAuditLogger
[ https://issues.apache.org/jira/browse/CASSANDRA-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Meredith updated CASSANDRA-15748: - Fix Version/s: (was: 4.0-alpha) 4.0-beta > Provide ability to configure IAuditLogger > - > > Key: CASSANDRA-15748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15748 > Project: Cassandra > Issue Type: Improvement > Components: Local/Other >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > Currently there is a parameterless constructor expected for IAuditLogger > instances. There is not any way how to "inject" configuration properties from > cassandra.yaml into these concrete classes. This would be very handy for > custom IAuditLogger implementations. > > PR: [https://github.com/apache/cassandra/pull/555] > [|https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15748] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15782) Compression test failure
[ https://issues.apache.org/jira/browse/CASSANDRA-15782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-15782: -- Fix Version/s: (was: 4.0-beta) (was: 4.0) 4.0-alpha > Compression test failure > > > Key: CASSANDRA-15782 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15782 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: Berenguer Blasi >Assignee: Joey Lynch >Priority: Normal > Fix For: 4.0-alpha > > > On CASSANDRA-15560 compression test failed. This was bisected to > [9c1bbf3|https://github.com/apache/cassandra/commit/9c1bbf3ac913f9bdf7a0e0922106804af42d2c1e] > from CASSANDRA-15379. > Full details here > CC/ [~jolynch] in case he can spot it quick. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15757) CustomIndexTest.indexBuildingPagesLargePartitions is flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-15757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie updated CASSANDRA-15757:
--
Fix Version/s: (was: 4.0-beta)
4.0-alpha
> CustomIndexTest.indexBuildingPagesLargePartitions is flaky
> --
>
> Key: CASSANDRA-15757
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15757
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Jon Meredith
>Assignee: Andres de la Peña
>Priority: Normal
> Fix For: 4.0-alpha
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> CustomIndexTest.indexBuildingPagesLargePartitions is flaky. Failed in CI and
> was able to reproduce failure inside IntelliJ by setting test Repeat to ‘Run
> Until Failure’. Failed after 459 iterations.
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.cassandra.index.CustomIndexTest.lambda$indexBuildingPagesLargePartitions$1(CustomIndexTest.java:687)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
> at
> org.apache.cassandra.index.CustomIndexTest.indexBuildingPagesLargePartitions(CustomIndexTest.java:687)
> at jdk.internal.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:53)
> at
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
> at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14280) Fix timeout test - org.apache.cassandra.cql3.ViewTest
[ https://issues.apache.org/jira/browse/CASSANDRA-14280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-14280: -- Fix Version/s: (was: 4.0) 4.0-alpha > Fix timeout test - org.apache.cassandra.cql3.ViewTest > - > > Key: CASSANDRA-14280 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14280 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Testing >Reporter: Dikang Gu >Assignee: Dikang Gu >Priority: Normal > Fix For: 4.0-alpha > > > The test timeout very often, it seems too big, try to split it into multiple > tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15766) NoSpamLogger arguments building objects on hot paths
[ https://issues.apache.org/jira/browse/CASSANDRA-15766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100214#comment-17100214 ] Yifan Cai commented on CASSANDRA-15766: --- Agree that there are many 'if's. What a lambda is compiled into depends on the java compiler. For a capturing lambda, it can still be a static method with the expanded param list. (off topic) At the end, we need to run benchmark to quantitively measure the difference. > NoSpamLogger arguments building objects on hot paths > > > Key: CASSANDRA-15766 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15766 > Project: Cassandra > Issue Type: Bug > Components: Observability/Logging >Reporter: Jon Meredith >Assignee: Jon Meredith >Priority: Normal > Fix For: 4.0-rc > > > NoSpamLogger is used in hot logging paths to prevent logs being overrun. For > that to be most effective the arguments to the logger need to be cheap to > construct. During the internode messaging refactor CASSANDRA-15066, > performance changes to BufferPool for CASSANDRA-14416 > were accidentally reverted in the merge up from 3.11. > Reviewing other uses since, it looks like there are a few places where the > arguments require some form of String building. > org.apache.cassandra.net.InboundSink#accept > org.apache.cassandra.net.InboundMessageHandler#processCorruptFrame > org.apache.cassandra.net.InboundMessageHandler.LargeMessage#deserialize > org.apache.cassandra.net.OutboundConnection#onOverloaded > org.apache.cassandra.utils.memory.BufferPool.GlobalPool#allocateMoreChunks > Formatting arguments should either be precomputed, or if expensive they > should be computed after the decision on whether to noSpamLog has been made. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15790) EmptyType doesn't override writeValue so could attempt to write bytes when expected not to
[
https://issues.apache.org/jira/browse/CASSANDRA-15790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West updated CASSANDRA-15790:
Reviewers: Jordan West, Jordan West (was: Jordan West)
Jordan West, Jordan West
Status: Review In Progress (was: Patch Available)
> EmptyType doesn't override writeValue so could attempt to write bytes when
> expected not to
> --
>
> Key: CASSANDRA-15790
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15790
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Semantics
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>
> EmptyType.writeValues is defined here
> https://github.com/apache/cassandra/blob/e394dc0bb32f612a476269010930c617dd1ed3cb/src/java/org/apache/cassandra/db/marshal/AbstractType.java#L407-L414
> {code}
> public void writeValue(ByteBuffer value, DataOutputPlus out) throws
> IOException
> {
> assert value.hasRemaining();
> if (valueLengthIfFixed() >= 0)
> out.write(value);
> else
> ByteBufferUtil.writeWithVIntLength(value, out);
> }
> {code}
> This is fine when the value is empty as the write of empty no-ops (the
> readValue also noops since the length is 0), but if the value is not empty
> (possible during upgrades or random bugs) then this could silently cause
> corruption; ideally this should throw a exception if the ByteBuffer has data.
> This was called from
> org.apache.cassandra.db.rows.BufferCell.Serializer#serialize, here we check
> to see if data is present or not and update the flags. If data is present
> then and only then do we call type.writeValue (which requires bytes is not
> empty). The problem is that EmptyType never expects writes to happen, but it
> still writes them; and does not read them (since it says it is fixed length
> of 0, so does read(buffer, 0)).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15748) Provide ability to configure IAuditLogger
[ https://issues.apache.org/jira/browse/CASSANDRA-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100246#comment-17100246 ] Stefan Miklosovic commented on CASSANDRA-15748: --- [~jmeredithco] no worries, as long as we do not forget, all good :) > Provide ability to configure IAuditLogger > - > > Key: CASSANDRA-15748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15748 > Project: Cassandra > Issue Type: Improvement > Components: Local/Other >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0-alpha > > Time Spent: 10m > Remaining Estimate: 0h > > Currently there is a parameterless constructor expected for IAuditLogger > instances. There is not any way how to "inject" configuration properties from > cassandra.yaml into these concrete classes. This would be very handy for > custom IAuditLogger implementations. > > PR: [https://github.com/apache/cassandra/pull/555] > [|https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15748] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15783) test_optimized_primary_range_repair - transient_replication_test.TestTransientReplication
[
https://issues.apache.org/jira/browse/CASSANDRA-15783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100449#comment-17100449
]
ZhaoYang edited comment on CASSANDRA-15783 at 5/6/20, 4:35 AM:
---
Error trace:
{code:java}
=== FAILURES ===
_ TestTransientReplication.test_optimized_primary_range_repair _
self =
@pytest.mark.no_vnodes
def test_optimized_primary_range_repair(self):
""" optimized primary range incremental repair from full replica should
remove data on node3 """
self._test_speculative_write_repair_cycle(primary_range=True,
optimized_repair=True,
repair_coordinator=self.node1,
> expect_node3_data=False)
transient_replication_test.py:463:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
transient_replication_test.py:435: in _test_speculative_write_repair_cycle
self.assert_has_sstables(self.node2, compact=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self =
node = , flush = False
compact = True
def assert_has_sstables(self, node, flush=False, compact=False):
if flush:
node.flush()
if compact:
node.nodetool(' '.join(['compact', self.keyspace, self.table]))
sstables = node.get_sstables(self.keyspace, self.table)
> assert sstables
E assert []
{code}
copying my comment from 15657:
{quote}This patch allows ZCS for size-tiered compaction which is used by the
test and the test log showed that the file is indeed received via ZCS...
{code:java}
INFO [CompactionExecutor:2] 2020-05-06 12:28:10,848
PendingRepairManager.java:446 - Obsoleting transient repaired ssatbles
DEBUG [CompactionExecutor:2] 2020-05-06 12:28:10,852
PendingRepairManager.java:144 - Removing compaction strategy for pending repair
fc89be70-8f51-11ea-8232-f78d8bc3c1f4 on ks.tblINFO [NonPeriodicTasks:1]
2020-05-06 12:28:10,855 SSTable.java:111 - Deleting sstable:
/private/var/folders/w0/m4svxry56h56g_42mx0rwbcrgn/T/dtest-9fybyxp7/test/node2/data0/ks/tbl-f26130408f5111ea8232f78d8bc3c1f4/na-1-big
{code}
Transient replica repair is removing the received file and it's doing it
earlier on ZCS sstable than non-ZCS sstable..
{quote}
Let me know if there is anything I can help
was (Author: jasonstack):
Error trace:
{code:java}
=== FAILURES ===
_ TestTransientReplication.test_optimized_primary_range_repair _
self =
@pytest.mark.no_vnodes
def test_optimized_primary_range_repair(self):
""" optimized primary range incremental repair from full replica should
remove data on node3 """
self._test_speculative_write_repair_cycle(primary_range=True,
optimized_repair=True,
repair_coordinator=self.node1,
> expect_node3_data=False)
transient_replication_test.py:463:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
transient_replication_test.py:435: in _test_speculative_write_repair_cycle
self.assert_has_sstables(self.node2, compact=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self =
node = , flush = False
compact = True
def assert_has_sstables(self, node, flush=False, compact=False):
if flush:
node.flush()
if compact:
node.nodetool(' '.join(['compact', self.keyspace, self.table]))
sstables = node.get_sstables(self.keyspace, self.table)
> assert sstables
E assert []
{code}
copying my comment from 15657:
{quote}This patch allows ZCS for size-tiered compaction which is used by the
test and the test log showed that the file is indeed received via ZCS...
{code:java}
INFO [CompactionExecutor:2] 2020-05-06 12:28:10,848
PendingRepairManager.java:446 - Obsoleting transient repaired ssatbles
DEBUG [CompactionExecutor:2] 2020-05-06 12:28:10,852
PendingRepairManager.java:144 - Removing compaction strategy for pending repair
fc89be70-8f51-11ea-8232-f78d8bc3c1f4 on ks.tblINFO [NonPeriodicTasks:1]
2020-05-06 12:28:10,855 SSTable.java:111 - Deleting sstable:
/private/var/folders/w0/m4svxry56h56g_42mx0rwbcrgn/T/dtest-9fybyxp7/test/node2/data0/ks/tbl-f26130408f5111ea8232f78d8bc3c1f4/na-1-big
{code}
Transient replica repair is removing the received file and it's doing it faster
on ZCS sstable than non-ZCS sstable..
{quote}
Let me know if there is anything I can help
> test_optimized_primary_range_repair -
>
[jira] [Comment Edited] (CASSANDRA-15657) Improve zero-copy-streaming containment check by using file sections
[
https://issues.apache.org/jira/browse/CASSANDRA-15657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100447#comment-17100447
]
ZhaoYang edited comment on CASSANDRA-15657 at 5/6/20, 4:35 AM:
---
[~e.dimitrova] Thanks for the report. This patch allows ZCS for size-tiered
compaction which is used by the test and the test log showed that the file is
indeed received via ZCS...
{code:java}
INFO [CompactionExecutor:2] 2020-05-06 12:28:10,848
PendingRepairManager.java:446 - Obsoleting transient repaired ssatbles
DEBUG [CompactionExecutor:2] 2020-05-06 12:28:10,852
PendingRepairManager.java:144 - Removing compaction strategy for pending repair
fc89be70-8f51-11ea-8232-f78d8bc3c1f4 on ks.tblINFO [NonPeriodicTasks:1]
2020-05-06 12:28:10,855 SSTable.java:111 - Deleting sstable:
/private/var/folders/w0/m4svxry56h56g_42mx0rwbcrgn/T/dtest-9fybyxp7/test/node2/data0/ks/tbl-f26130408f5111ea8232f78d8bc3c1f4/na-1-big
{code}
Transient replica repair is removing the received file and it's doing it
earlier on ZCS sstable than non-ZCS sstable..
was (Author: jasonstack):
[~e.dimitrova] Thanks for the report. This patch allows ZCS for size-tiered
compaction which is used by the test and the test log showed that the file is
indeed received via ZCS...
{code:java}
INFO [CompactionExecutor:2] 2020-05-06 12:28:10,848
PendingRepairManager.java:446 - Obsoleting transient repaired ssatbles
DEBUG [CompactionExecutor:2] 2020-05-06 12:28:10,852
PendingRepairManager.java:144 - Removing compaction strategy for pending repair
fc89be70-8f51-11ea-8232-f78d8bc3c1f4 on ks.tblINFO [NonPeriodicTasks:1]
2020-05-06 12:28:10,855 SSTable.java:111 - Deleting sstable:
/private/var/folders/w0/m4svxry56h56g_42mx0rwbcrgn/T/dtest-9fybyxp7/test/node2/data0/ks/tbl-f26130408f5111ea8232f78d8bc3c1f4/na-1-big
{code}
Transient replica repair is removing the received file and it's doing it faster
on ZCS sstable than non-ZCS sstable..
> Improve zero-copy-streaming containment check by using file sections
>
>
> Key: CASSANDRA-15657
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15657
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Streaming and Messaging
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 4.0
>
>
> Currently zero copy streaming is only enabled for leveled-compaction strategy
> and it checks if all keys in the sstables are included in the transferred
> ranges.
> This is very inefficient. The containment check can be improved by checking
> if transferred sections (the transferred file positions) cover entire sstable.
> I also enabled ZCS for all compaction strategies since the new containment
> check is very fast..
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14965) Write the Data Modeling Documentation Section
[ https://issues.apache.org/jira/browse/CASSANDRA-14965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Lynch updated CASSANDRA-14965: --- Resolution: Duplicate Status: Resolved (was: Open) Looks like this got done elsewhere :yay: > Write the Data Modeling Documentation Section > - > > Key: CASSANDRA-14965 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14965 > Project: Cassandra > Issue Type: Improvement > Components: Documentation/Website >Reporter: Joey Lynch >Assignee: Joey Lynch >Priority: Low > > It would be good to fill out the data modeling > [section|http://cassandra.apache.org/doc/latest/data_modeling/index.html?highlight=todo] > of the website. I plan to fill it out with a discussion of the following: > 1. CQL Query Modeling == Data Model > 2. Enough storage explanation to justify partition/clustering column choices > 3. Data modeling best practices (theoretical and practical limits, partition > sizes, column sizes, etc ... > 4. Examples of different common data models (e.g. forward index, reverse > index, time series, json blob storage, etc ...) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15783) test_optimized_primary_range_repair - transient_replication_test.TestTransientReplication
[
https://issues.apache.org/jira/browse/CASSANDRA-15783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100449#comment-17100449
]
ZhaoYang commented on CASSANDRA-15783:
--
copying my comment from 15657:
{quote}This patch allows ZCS for size-tiered compaction which is used by the
test and the test log showed that the file is indeed received via
ZCS...Probably some lifecycle issue that the received sstable isn't added to
ColumnFamilyStore.
{quote}
Let me know if there is anything I can help
> test_optimized_primary_range_repair -
> transient_replication_test.TestTransientReplication
> -
>
> Key: CASSANDRA-15783
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15783
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> Dtest failure.
> Example:
> https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/118/workflows/9e57522d-52fa-4d44-88d8-5cec0e87f517/jobs/585/tests
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15783) test_optimized_primary_range_repair - transient_replication_test.TestTransientReplication
[
https://issues.apache.org/jira/browse/CASSANDRA-15783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100449#comment-17100449
]
ZhaoYang edited comment on CASSANDRA-15783 at 5/6/20, 4:17 AM:
---
Error trace:
{code:java}
=== FAILURES ===
_ TestTransientReplication.test_optimized_primary_range_repair _
self =
@pytest.mark.no_vnodes
def test_optimized_primary_range_repair(self):
""" optimized primary range incremental repair from full replica should
remove data on node3 """
self._test_speculative_write_repair_cycle(primary_range=True,
optimized_repair=True,
repair_coordinator=self.node1,
> expect_node3_data=False)
transient_replication_test.py:463:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
transient_replication_test.py:435: in _test_speculative_write_repair_cycle
self.assert_has_sstables(self.node2, compact=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self =
node = , flush = False
compact = True
def assert_has_sstables(self, node, flush=False, compact=False):
if flush:
node.flush()
if compact:
node.nodetool(' '.join(['compact', self.keyspace, self.table]))
sstables = node.get_sstables(self.keyspace, self.table)
> assert sstables
E assert []
{code}
copying my comment from 15657:
{quote}This patch allows ZCS for size-tiered compaction which is used by the
test and the test log showed that the file is indeed received via
ZCS...Probably some lifecycle issue that the received sstable isn't added to
ColumnFamilyStore.
{quote}
Let me know if there is anything I can help
was (Author: jasonstack):
copying my comment from 15657:
{quote}This patch allows ZCS for size-tiered compaction which is used by the
test and the test log showed that the file is indeed received via
ZCS...Probably some lifecycle issue that the received sstable isn't added to
ColumnFamilyStore.
{quote}
Let me know if there is anything I can help
> test_optimized_primary_range_repair -
> transient_replication_test.TestTransientReplication
> -
>
> Key: CASSANDRA-15783
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15783
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> Dtest failure.
> Example:
> https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/118/workflows/9e57522d-52fa-4d44-88d8-5cec0e87f517/jobs/585/tests
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15783) test_optimized_primary_range_repair - transient_replication_test.TestTransientReplication
[
https://issues.apache.org/jira/browse/CASSANDRA-15783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100449#comment-17100449
]
ZhaoYang edited comment on CASSANDRA-15783 at 5/6/20, 4:33 AM:
---
Error trace:
{code:java}
=== FAILURES ===
_ TestTransientReplication.test_optimized_primary_range_repair _
self =
@pytest.mark.no_vnodes
def test_optimized_primary_range_repair(self):
""" optimized primary range incremental repair from full replica should
remove data on node3 """
self._test_speculative_write_repair_cycle(primary_range=True,
optimized_repair=True,
repair_coordinator=self.node1,
> expect_node3_data=False)
transient_replication_test.py:463:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
transient_replication_test.py:435: in _test_speculative_write_repair_cycle
self.assert_has_sstables(self.node2, compact=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self =
node = , flush = False
compact = True
def assert_has_sstables(self, node, flush=False, compact=False):
if flush:
node.flush()
if compact:
node.nodetool(' '.join(['compact', self.keyspace, self.table]))
sstables = node.get_sstables(self.keyspace, self.table)
> assert sstables
E assert []
{code}
copying my comment from 15657:
{quote}This patch allows ZCS for size-tiered compaction which is used by the
test and the test log showed that the file is indeed received via ZCS...
{code:java}
INFO [CompactionExecutor:2] 2020-05-06 12:28:10,848
PendingRepairManager.java:446 - Obsoleting transient repaired ssatbles
DEBUG [CompactionExecutor:2] 2020-05-06 12:28:10,852
PendingRepairManager.java:144 - Removing compaction strategy for pending repair
fc89be70-8f51-11ea-8232-f78d8bc3c1f4 on ks.tblINFO [NonPeriodicTasks:1]
2020-05-06 12:28:10,855 SSTable.java:111 - Deleting sstable:
/private/var/folders/w0/m4svxry56h56g_42mx0rwbcrgn/T/dtest-9fybyxp7/test/node2/data0/ks/tbl-f26130408f5111ea8232f78d8bc3c1f4/na-1-big
{code}
Transient replica repair is removing the received file and it's doing it faster
on ZCS sstable than non-ZCS sstable..
{quote}
Let me know if there is anything I can help
was (Author: jasonstack):
Error trace:
{code:java}
=== FAILURES ===
_ TestTransientReplication.test_optimized_primary_range_repair _
self =
@pytest.mark.no_vnodes
def test_optimized_primary_range_repair(self):
""" optimized primary range incremental repair from full replica should
remove data on node3 """
self._test_speculative_write_repair_cycle(primary_range=True,
optimized_repair=True,
repair_coordinator=self.node1,
> expect_node3_data=False)
transient_replication_test.py:463:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
transient_replication_test.py:435: in _test_speculative_write_repair_cycle
self.assert_has_sstables(self.node2, compact=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self =
node = , flush = False
compact = True
def assert_has_sstables(self, node, flush=False, compact=False):
if flush:
node.flush()
if compact:
node.nodetool(' '.join(['compact', self.keyspace, self.table]))
sstables = node.get_sstables(self.keyspace, self.table)
> assert sstables
E assert []
{code}
copying my comment from 15657:
{quote}This patch allows ZCS for size-tiered compaction which is used by the
test and the test log showed that the file is indeed received via
ZCS...Probably some lifecycle issue that the received sstable isn't added to
ColumnFamilyStore.
{quote}
Let me know if there is anything I can help
> test_optimized_primary_range_repair -
> transient_replication_test.TestTransientReplication
> -
>
> Key: CASSANDRA-15783
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15783
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> Dtest failure.
> Example:
>
[jira] [Commented] (CASSANDRA-15775) Configuration to disallow queries with "allow filtering"
[ https://issues.apache.org/jira/browse/CASSANDRA-15775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100430#comment-17100430 ] maxwellguo commented on CASSANDRA-15775: we have just disallow the use of "allow filtering" every time user use this words. I think it is ok to open or close this feature of "allow filtering" under table level for some people may just use this to do some test. we can set it to close the feature by default and people can open it by using "ALTER TABLE"? > Configuration to disallow queries with "allow filtering" > > > Key: CASSANDRA-15775 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15775 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Local Write-Read Paths >Reporter: Christian Fredriksson >Priority: Normal > > Problem: We have inexperienced developers not following guidelines or best > pratices who do queries with "allow filtering" which have negative impact on > performance on other queries and developers. > It would be beneficial to have a (server side) configuration to disallow > these queries altogether. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15657) Improve zero-copy-streaming containment check by using file sections
[ https://issues.apache.org/jira/browse/CASSANDRA-15657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100447#comment-17100447 ] ZhaoYang commented on CASSANDRA-15657: -- [~e.dimitrova] Thanks for the report. This patch allows ZCS for size-tiered compaction which is used by the test and the test log showed that the file is indeed received via ZCS...Probably some lifecycle issue that the received sstable isn't added to ColumnFamilyStore. > Improve zero-copy-streaming containment check by using file sections > > > Key: CASSANDRA-15657 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15657 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Streaming and Messaging >Reporter: ZhaoYang >Assignee: ZhaoYang >Priority: Normal > Fix For: 4.0 > > > Currently zero copy streaming is only enabled for leveled-compaction strategy > and it checks if all keys in the sstables are included in the transferred > ranges. > This is very inefficient. The containment check can be improved by checking > if transferred sections (the transferred file positions) cover entire sstable. > I also enabled ZCS for all compaction strategies since the new containment > check is very fast.. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15657) Improve zero-copy-streaming containment check by using file sections
[
https://issues.apache.org/jira/browse/CASSANDRA-15657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100447#comment-17100447
]
ZhaoYang edited comment on CASSANDRA-15657 at 5/6/20, 4:32 AM:
---
[~e.dimitrova] Thanks for the report. This patch allows ZCS for size-tiered
compaction which is used by the test and the test log showed that the file is
indeed received via ZCS...
{code:java}
INFO [CompactionExecutor:2] 2020-05-06 12:28:10,848
PendingRepairManager.java:446 - Obsoleting transient repaired ssatblesDEBUG
[CompactionExecutor:2] 2020-05-06 12:28:10,852 PendingRepairManager.java:144 -
Removing compaction strategy for pending repair
fc89be70-8f51-11ea-8232-f78d8bc3c1f4 on ks.tblINFO [NonPeriodicTasks:1]
2020-05-06 12:28:10,855 SSTable.java:111 - Deleting sstable:
/private/var/folders/w0/m4svxry56h56g_42mx0rwbcrgn/T/dtest-9fybyxp7/test/node2/data0/ks/tbl-f26130408f5111ea8232f78d8bc3c1f4/na-1-big
{code}
Transient replica repair is removing the received file and it's doing it faster
on ZCS sstable than non-ZCS sstable.
was (Author: jasonstack):
[~e.dimitrova] Thanks for the report. This patch allows ZCS for size-tiered
compaction which is used by the test and the test log showed that the file is
indeed received via ZCS...Probably some lifecycle issue that the received
sstable isn't added to ColumnFamilyStore.
> Improve zero-copy-streaming containment check by using file sections
>
>
> Key: CASSANDRA-15657
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15657
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Streaming and Messaging
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 4.0
>
>
> Currently zero copy streaming is only enabled for leveled-compaction strategy
> and it checks if all keys in the sstables are included in the transferred
> ranges.
> This is very inefficient. The containment check can be improved by checking
> if transferred sections (the transferred file positions) cover entire sstable.
> I also enabled ZCS for all compaction strategies since the new containment
> check is very fast..
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15657) Improve zero-copy-streaming containment check by using file sections
[
https://issues.apache.org/jira/browse/CASSANDRA-15657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100447#comment-17100447
]
ZhaoYang edited comment on CASSANDRA-15657 at 5/6/20, 4:32 AM:
---
[~e.dimitrova] Thanks for the report. This patch allows ZCS for size-tiered
compaction which is used by the test and the test log showed that the file is
indeed received via ZCS...
{code:java}
INFO [CompactionExecutor:2] 2020-05-06 12:28:10,848
PendingRepairManager.java:446 - Obsoleting transient repaired ssatbles
DEBUG [CompactionExecutor:2] 2020-05-06 12:28:10,852
PendingRepairManager.java:144 - Removing compaction strategy for pending repair
fc89be70-8f51-11ea-8232-f78d8bc3c1f4 on ks.tblINFO [NonPeriodicTasks:1]
2020-05-06 12:28:10,855 SSTable.java:111 - Deleting sstable:
/private/var/folders/w0/m4svxry56h56g_42mx0rwbcrgn/T/dtest-9fybyxp7/test/node2/data0/ks/tbl-f26130408f5111ea8232f78d8bc3c1f4/na-1-big
{code}
Transient replica repair is removing the received file and it's doing it faster
on ZCS sstable than non-ZCS sstable..
was (Author: jasonstack):
[~e.dimitrova] Thanks for the report. This patch allows ZCS for size-tiered
compaction which is used by the test and the test log showed that the file is
indeed received via ZCS...
{code:java}
INFO [CompactionExecutor:2] 2020-05-06 12:28:10,848
PendingRepairManager.java:446 - Obsoleting transient repaired ssatblesDEBUG
[CompactionExecutor:2] 2020-05-06 12:28:10,852 PendingRepairManager.java:144 -
Removing compaction strategy for pending repair
fc89be70-8f51-11ea-8232-f78d8bc3c1f4 on ks.tblINFO [NonPeriodicTasks:1]
2020-05-06 12:28:10,855 SSTable.java:111 - Deleting sstable:
/private/var/folders/w0/m4svxry56h56g_42mx0rwbcrgn/T/dtest-9fybyxp7/test/node2/data0/ks/tbl-f26130408f5111ea8232f78d8bc3c1f4/na-1-big
{code}
Transient replica repair is removing the received file and it's doing it faster
on ZCS sstable than non-ZCS sstable.
> Improve zero-copy-streaming containment check by using file sections
>
>
> Key: CASSANDRA-15657
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15657
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Streaming and Messaging
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 4.0
>
>
> Currently zero copy streaming is only enabled for leveled-compaction strategy
> and it checks if all keys in the sstables are included in the transferred
> ranges.
> This is very inefficient. The containment check can be improved by checking
> if transferred sections (the transferred file positions) cover entire sstable.
> I also enabled ZCS for all compaction strategies since the new containment
> check is very fast..
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15657) Improve zero-copy-streaming containment check by using file sections
[
https://issues.apache.org/jira/browse/CASSANDRA-15657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100447#comment-17100447
]
ZhaoYang edited comment on CASSANDRA-15657 at 5/6/20, 4:38 AM:
---
[~e.dimitrova] Thanks for the report. This patch allows ZCS for size-tiered
compaction which is used by the test and the test log showed that the file is
indeed received via ZCS...
{code:java}
INFO [CompactionExecutor:2] 2020-05-06 12:28:10,848
PendingRepairManager.java:446 - Obsoleting transient repaired ssatbles
DEBUG [CompactionExecutor:2] 2020-05-06 12:28:10,852
PendingRepairManager.java:144 - Removing compaction strategy for pending repair
fc89be70-8f51-11ea-8232-f78d8bc3c1f4 on ks.tblINFO [NonPeriodicTasks:1]
2020-05-06 12:28:10,855 SSTable.java:111 - Deleting sstable:
/private/var/folders/w0/m4svxry56h56g_42mx0rwbcrgn/T/dtest-9fybyxp7/test/node2/data0/ks/tbl-f26130408f5111ea8232f78d8bc3c1f4/na-1-big
{code}
Transient replica \{{PendingRepairManager}} is removing the received file and
it's doing it earlier on ZCS sstable than non-ZCS sstable because ZCS sstable
carries "repairedAt" metadata.
was (Author: jasonstack):
[~e.dimitrova] Thanks for the report. This patch allows ZCS for size-tiered
compaction which is used by the test and the test log showed that the file is
indeed received via ZCS...
{code:java}
INFO [CompactionExecutor:2] 2020-05-06 12:28:10,848
PendingRepairManager.java:446 - Obsoleting transient repaired ssatbles
DEBUG [CompactionExecutor:2] 2020-05-06 12:28:10,852
PendingRepairManager.java:144 - Removing compaction strategy for pending repair
fc89be70-8f51-11ea-8232-f78d8bc3c1f4 on ks.tblINFO [NonPeriodicTasks:1]
2020-05-06 12:28:10,855 SSTable.java:111 - Deleting sstable:
/private/var/folders/w0/m4svxry56h56g_42mx0rwbcrgn/T/dtest-9fybyxp7/test/node2/data0/ks/tbl-f26130408f5111ea8232f78d8bc3c1f4/na-1-big
{code}
Transient replica repair is removing the received file and it's doing it
earlier on ZCS sstable than non-ZCS sstable..
> Improve zero-copy-streaming containment check by using file sections
>
>
> Key: CASSANDRA-15657
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15657
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Streaming and Messaging
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 4.0
>
>
> Currently zero copy streaming is only enabled for leveled-compaction strategy
> and it checks if all keys in the sstables are included in the transferred
> ranges.
> This is very inefficient. The containment check can be improved by checking
> if transferred sections (the transferred file positions) cover entire sstable.
> I also enabled ZCS for all compaction strategies since the new containment
> check is very fast..
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15783) test_optimized_primary_range_repair - transient_replication_test.TestTransientReplication
[
https://issues.apache.org/jira/browse/CASSANDRA-15783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100449#comment-17100449
]
ZhaoYang edited comment on CASSANDRA-15783 at 5/6/20, 4:38 AM:
---
Error trace:
{code:java}
=== FAILURES ===
_ TestTransientReplication.test_optimized_primary_range_repair _
self =
@pytest.mark.no_vnodes
def test_optimized_primary_range_repair(self):
""" optimized primary range incremental repair from full replica should
remove data on node3 """
self._test_speculative_write_repair_cycle(primary_range=True,
optimized_repair=True,
repair_coordinator=self.node1,
> expect_node3_data=False)
transient_replication_test.py:463:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
transient_replication_test.py:435: in _test_speculative_write_repair_cycle
self.assert_has_sstables(self.node2, compact=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self =
node = , flush = False
compact = True
def assert_has_sstables(self, node, flush=False, compact=False):
if flush:
node.flush()
if compact:
node.nodetool(' '.join(['compact', self.keyspace, self.table]))
sstables = node.get_sstables(self.keyspace, self.table)
> assert sstables
E assert []
{code}
copying my comment from 15657:
{quote}This patch allows ZCS for size-tiered compaction which is used by the
test and the test log showed that the file is indeed received via ZCS...
{code:java}
INFO [CompactionExecutor:2] 2020-05-06 12:28:10,848
PendingRepairManager.java:446 - Obsoleting transient repaired ssatbles
DEBUG [CompactionExecutor:2] 2020-05-06 12:28:10,852
PendingRepairManager.java:144 - Removing compaction strategy for pending repair
fc89be70-8f51-11ea-8232-f78d8bc3c1f4 on ks.tblINFO [NonPeriodicTasks:1]
2020-05-06 12:28:10,855 SSTable.java:111 - Deleting sstable:
/private/var/folders/w0/m4svxry56h56g_42mx0rwbcrgn/T/dtest-9fybyxp7/test/node2/data0/ks/tbl-f26130408f5111ea8232f78d8bc3c1f4/na-1-big
{code}
Transient replica \{{PendingRepairManager}} is removing the received file and
it's doing it earlier on ZCS sstable than non-ZCS sstable because ZCS sstable
carries "repairedAt" metadata.
{quote}
Let me know if there is anything I can help
was (Author: jasonstack):
Error trace:
{code:java}
=== FAILURES ===
_ TestTransientReplication.test_optimized_primary_range_repair _
self =
@pytest.mark.no_vnodes
def test_optimized_primary_range_repair(self):
""" optimized primary range incremental repair from full replica should
remove data on node3 """
self._test_speculative_write_repair_cycle(primary_range=True,
optimized_repair=True,
repair_coordinator=self.node1,
> expect_node3_data=False)
transient_replication_test.py:463:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
transient_replication_test.py:435: in _test_speculative_write_repair_cycle
self.assert_has_sstables(self.node2, compact=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self =
node = , flush = False
compact = True
def assert_has_sstables(self, node, flush=False, compact=False):
if flush:
node.flush()
if compact:
node.nodetool(' '.join(['compact', self.keyspace, self.table]))
sstables = node.get_sstables(self.keyspace, self.table)
> assert sstables
E assert []
{code}
copying my comment from 15657:
{quote}This patch allows ZCS for size-tiered compaction which is used by the
test and the test log showed that the file is indeed received via ZCS...
{code:java}
INFO [CompactionExecutor:2] 2020-05-06 12:28:10,848
PendingRepairManager.java:446 - Obsoleting transient repaired ssatbles
DEBUG [CompactionExecutor:2] 2020-05-06 12:28:10,852
PendingRepairManager.java:144 - Removing compaction strategy for pending repair
fc89be70-8f51-11ea-8232-f78d8bc3c1f4 on ks.tblINFO [NonPeriodicTasks:1]
2020-05-06 12:28:10,855 SSTable.java:111 - Deleting sstable:
/private/var/folders/w0/m4svxry56h56g_42mx0rwbcrgn/T/dtest-9fybyxp7/test/node2/data0/ks/tbl-f26130408f5111ea8232f78d8bc3c1f4/na-1-big
{code}
Transient replica repair is removing the received file and it's doing it
earlier on ZCS sstable than non-ZCS sstable..
{quote}
Let me know if there is anything I can help
>
[jira] [Comment Edited] (CASSANDRA-15783) test_optimized_primary_range_repair - transient_replication_test.TestTransientReplication
[
https://issues.apache.org/jira/browse/CASSANDRA-15783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100449#comment-17100449
]
ZhaoYang edited comment on CASSANDRA-15783 at 5/6/20, 4:46 AM:
---
Error trace:
{code:java}
=== FAILURES ===
_ TestTransientReplication.test_optimized_primary_range_repair _
self =
@pytest.mark.no_vnodes
def test_optimized_primary_range_repair(self):
""" optimized primary range incremental repair from full replica should
remove data on node3 """
self._test_speculative_write_repair_cycle(primary_range=True,
optimized_repair=True,
repair_coordinator=self.node1,
> expect_node3_data=False)
transient_replication_test.py:463:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
transient_replication_test.py:435: in _test_speculative_write_repair_cycle
self.assert_has_sstables(self.node2, compact=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self =
node = , flush = False
compact = True
def assert_has_sstables(self, node, flush=False, compact=False):
if flush:
node.flush()
if compact:
node.nodetool(' '.join(['compact', self.keyspace, self.table]))
sstables = node.get_sstables(self.keyspace, self.table)
> assert sstables
E assert []
{code}
copying my comment from 15657:
{quote}This patch allows ZCS for size-tiered compaction which is used by the
test and the test log showed that the file is indeed received via ZCS...
{code:java}
INFO [CompactionExecutor:2] 2020-05-06 12:28:10,848
PendingRepairManager.java:446 - Obsoleting transient repaired ssatbles
DEBUG [CompactionExecutor:2] 2020-05-06 12:28:10,852
PendingRepairManager.java:144 - Removing compaction strategy for pending repair
fc89be70-8f51-11ea-8232-f78d8bc3c1f4 on ks.tblINFO [NonPeriodicTasks:1]
2020-05-06 12:28:10,855 SSTable.java:111 - Deleting sstable:
/private/var/folders/w0/m4svxry56h56g_42mx0rwbcrgn/T/dtest-9fybyxp7/test/node2/data0/ks/tbl-f26130408f5111ea8232f78d8bc3c1f4/na-1-big
{code}
Transient replica {{PendingRepairManager}} is removing the received file and
it's doing it earlier on ZCS sstable than non-ZCS sstable because ZCS sstable
carries "repairedAt" metadata.
{quote}
[~e.dimitrova] [~djoshi] not sure if it's an expected behavior from transient
replica...
Let me know if there is anything I can help..
was (Author: jasonstack):
Error trace:
{code:java}
=== FAILURES ===
_ TestTransientReplication.test_optimized_primary_range_repair _
self =
@pytest.mark.no_vnodes
def test_optimized_primary_range_repair(self):
""" optimized primary range incremental repair from full replica should
remove data on node3 """
self._test_speculative_write_repair_cycle(primary_range=True,
optimized_repair=True,
repair_coordinator=self.node1,
> expect_node3_data=False)
transient_replication_test.py:463:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
transient_replication_test.py:435: in _test_speculative_write_repair_cycle
self.assert_has_sstables(self.node2, compact=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self =
node = , flush = False
compact = True
def assert_has_sstables(self, node, flush=False, compact=False):
if flush:
node.flush()
if compact:
node.nodetool(' '.join(['compact', self.keyspace, self.table]))
sstables = node.get_sstables(self.keyspace, self.table)
> assert sstables
E assert []
{code}
copying my comment from 15657:
{quote}This patch allows ZCS for size-tiered compaction which is used by the
test and the test log showed that the file is indeed received via ZCS...
{code:java}
INFO [CompactionExecutor:2] 2020-05-06 12:28:10,848
PendingRepairManager.java:446 - Obsoleting transient repaired ssatbles
DEBUG [CompactionExecutor:2] 2020-05-06 12:28:10,852
PendingRepairManager.java:144 - Removing compaction strategy for pending repair
fc89be70-8f51-11ea-8232-f78d8bc3c1f4 on ks.tblINFO [NonPeriodicTasks:1]
2020-05-06 12:28:10,855 SSTable.java:111 - Deleting sstable:
/private/var/folders/w0/m4svxry56h56g_42mx0rwbcrgn/T/dtest-9fybyxp7/test/node2/data0/ks/tbl-f26130408f5111ea8232f78d8bc3c1f4/na-1-big
{code}
Transient replica \{{PendingRepairManager}} is removing the received file and
it's doing it
[jira] [Commented] (CASSANDRA-13606) Improve handling of 2i initialization failures
[
https://issues.apache.org/jira/browse/CASSANDRA-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099849#comment-17099849
]
Berenguer Blasi commented on CASSANDRA-13606:
-
[~adelapena]
- If I understood correctly your first point, I'll log writable (and queryable
while I am at it) index set
[here|https://github.com/apache/cassandra/pull/570/files#diff-3f2c8994c4ff8748c3faf7e70958520dR658]
and we silently ignore writes to a failed index. +1 on that.
- Correct, I left the 'recovery path' unused as I saw it more as a future
proofing thing than an actual replacement to the rebuild. I am -1 on adding a
{{recover_index}}. What if we add to {{rebuild_index}} a {{recover}} param that
defaults to false? That would be less 'noisy' and backwards compatible etc.
wdyt?
> Improve handling of 2i initialization failures
> --
>
> Key: CASSANDRA-13606
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13606
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/2i Index
>Reporter: Sergio Bossa
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> CASSANDRA-10130 fixes the 2i build management, but initialization failures
> are still not properly handled, most notably because:
> * Initialization failures make the index non-queryable, but it can still be
> written to.
> * Initialization failures can be recovered via full rebuilds.
> Both points above are probably suboptimal because the initialization logic
> could be more complex than just an index build, hence it shouldn't be made
> recoverable via a simple rebuild, and could cause the index to be fully
> unavailable not just for reads, but for writes as well.
> So, we should better handle initialization failures by:
> * Allowing the index implementation to specify if unavailable for reads,
> writes, or both.
> * Providing a proper method to recover, distinct from index rebuilds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15299) CASSANDRA-13304 follow-up: improve checksumming and compression in protocol v5-beta
[
https://issues.apache.org/jira/browse/CASSANDRA-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099833#comment-17099833
]
Sam Tunnicliffe commented on CASSANDRA-15299:
-
Hi [~omichallat] , thanks for the feedback this is really useful.
{quote}You're hinting at renaming at a later point. I think that will be most
welcome, the names as they are now are pretty confusing. And IMHO you should
rename the outer container: even though "frame" is better suited for it, it's
more important to preserve familiarity for people coming from the legacy code.
{quote}
I was absolutely planning on renaming, I just imagined that there might be
several opinions on this, so was holding off to give the discussion a chance.
My personal preference would be to rename the inner container; I totally
appreciate your point about maintaining continuity for people coming from the
legacy code, but there a few points that sway me in the opposite direction:
1) Internal consistency and compatibility. The proposal here is to reuse both
the concepts and, where possible, the implementation from the internode su
bsystem.
2) At some point legacy is legacy and we ought to pick the right names for
things and concepts rather than be forever bound by previous decisions. This
can definitely be painful, but is probably necessary at times. c.f. the use of
{{Row}} in 2.2 and earlier and how that's changed wrt to {{Partition}} and CQL
rows with 3.0.
3) I wonder if it's actually more important to aim for clarity in naming in
order to better accomodate people without pre-existing familiarity with the
codebase. Those of us who have been working on the code for some time already
have a mental model and for sure, re-mapping the terms therein is annoying,
somewhat detrimental to productivity and in some cases, potentially dangerous
(though I don't think that's such a concern here). For those coming to the code
for the first time though, abiguous and/or outdated naming surely has an even
greater impact in terms of aggravating the learning curve.
{quote}From a client point of view, a dropped frame will result in request
timeouts. We have no way of providing a better error, since the stream ids of
the failed requests are in the corrupt payload. I'm wondering if it might not
be better to drop the connection all the time: at least the client gets
immediate feedback (we could try to propagate a cause), instead of a bunch of
requests timing out for no apparent reason.
{quote}
Yes, I agree. This is the major difference between the internode and
client/server messaging schemes. Failing hard and fast whenever a corrupt frame
is encountered seems the right thing to do at the moment. I suppose it might be
possible to track the stream ids for a given message on the client side and
have the server return an error, rather than closing the connection. But I'm
not sure just how feasible that is on the client side.
{quote}When two large messages are sent in a row, how will you tell the last
outer frame of the first message from the first outer frame of the second?
Don't you need some sort of is_last_frame flag in the header?
{quote}
The first of outer frame of a large message contains the inner frame header,
which specifies the length in bytes of the CQL message body. The processor
accumulates outer frames until that message has been completely received. The
single CQL message per large message constraint is required for this to work.
{quote}When compression is enabled, how are you going to test when the payload
reaches the max size?...all the messages in the payload are compressed
together, so you can't check the size as you go like you do for the
uncompressed version
{quote}
That's correct, we keep accumulating messages into the payload until the max
*uncompressed* size is reached, or we run out of messages. So there is some
loss of density when using compression, but this is an implementation detail
and could be improved later without requiring further changes to the protocol.
> CASSANDRA-13304 follow-up: improve checksumming and compression in protocol
> v5-beta
> ---
>
> Key: CASSANDRA-15299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15299
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Aleksey Yeschenko
>Assignee: Sam Tunnicliffe
>Priority: Normal
> Labels: protocolv5
> Fix For: 4.0-beta
>
>
> CASSANDRA-13304 made an important improvement to our native protocol: it
> introduced checksumming/CRC32 to request and response bodies. It’s an
> important step forward, but it doesn’t cover the entire stream. In
> particular, the message header is not covered by a checksum or a crc, which
>
[jira] [Commented] (CASSANDRA-13606) Improve handling of 2i initialization failures
[
https://issues.apache.org/jira/browse/CASSANDRA-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099809#comment-17099809
]
Andres de la Peña commented on CASSANDRA-13606:
---
The index initialization failure is already logged, so I'm not sure whether it
would be useful to log every write to the base table that can't go into the
not-writable index. Perhaps we could the {{SIM.writableIndexes}} set instead of
{{SIM#indexes}} when we decide which indexes are going to be affected be a
write transaction, so we would just silently exclude not-writable indexes from
every write until we they have recovered.
As for providing a method to recover distinct from index rebuilds, and not
missing initialization work, I see two problems:
* The new {{Index#getRecoveryTaskSupport}} method is only called from
{{SIM#recoverIndexesBlocking}}, and I think the later is never called.
* Doing a full index rebuild with {{nodetool rebuild_index}} invokes
{{SIM#rebuildIndexesBlocking}}, that makes the index available for both reads
and writes without calling neither {{Index#getInitializationTask}} nor
{{Index#getRecoveryTaskSupport}}.
I can think of two different ways of differentiating the rebuild of a properly
initialized index from recovering from an initialization failure:
- Having a separate nodetool command, for example {{nodetool recover_index}},
that calls {{SIM#recoverIndexesBlocking}}, that calls the new
{{Index#getRecoveryTaskSupport}}.
- Keep the current {{nodetool rebuild_index}} command for both rebuild and
recovery, but use {{Index#getRecoveryTaskSupport}} under the hood if we know
that the index has failed during its initialization.
WDYT?
> Improve handling of 2i initialization failures
> --
>
> Key: CASSANDRA-13606
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13606
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/2i Index
>Reporter: Sergio Bossa
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> CASSANDRA-10130 fixes the 2i build management, but initialization failures
> are still not properly handled, most notably because:
> * Initialization failures make the index non-queryable, but it can still be
> written to.
> * Initialization failures can be recovered via full rebuilds.
> Both points above are probably suboptimal because the initialization logic
> could be more complex than just an index build, hence it shouldn't be made
> recoverable via a simple rebuild, and could cause the index to be fully
> unavailable not just for reads, but for writes as well.
> So, we should better handle initialization failures by:
> * Allowing the index implementation to specify if unavailable for reads,
> writes, or both.
> * Providing a proper method to recover, distinct from index rebuilds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15745) Conflicting LWT transactions may be committed during topology change
[
https://issues.apache.org/jira/browse/CASSANDRA-15745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099825#comment-17099825
]
Benedict Elliott Smith edited comment on CASSANDRA-15745 at 5/5/20, 12:10 PM:
--
Thanks for the report [~Osipov]. Just to reformulate (mostly just removing
"Topology change is advertised on A", as I don't believe this is a necessary
step):
# Topology change starts on C, replacing A with D, only visible on C
# CAS1 starts on C with \{A, B, C, D}
# CAS2 (ballot > CAS1) starts on A with \{A, B, C}
# CAS1 prepares on \{B, C, D} (timeout on A)
# CAS2 prepares and accepts on \{A, B} (timeout on C); commits on A; terminates
# CAS1 accepts on D; terminates
# Topology change finishes (A is removed), visible globally
# CAS3 prepares with \{C, D}, sees accept of CAS1 and re-proposes it (with a
newer ballot)
Unfortunately this isn't trivial to fix, though there is more than one
approach. I happen to have an incomplete piece of work that should be able to
address this issue, but I have no timeline on when I may be able to propose it
here as a patch.
was (Author: benedict):
Thanks for the report [~Osipov]. Just to reformulate (mostly just removing
"Topology change is advertised on A", as I don't believe this is a necessary
step):
# Topology change starts on C, replacing A with D, only visible on C
# CAS1 starts on C with \{A, B, C, D}
# CAS2 (ballot > CAS1) starts on A with {A, B, C}
# CAS1 prepares on {B, C, D} (timeout on A)
# CAS2 prepares and accepts on \{A, B} (timeout on C); commits on A; terminates
# CAS1 accepts on D; terminates
# Topology change finishes (A is removed), visible globally
# CAS3 prepares with \{C, D}, sees accept of CAS1 and re-proposes it (with a
newer ballot)
Unfortunately this isn't trivial to fix, though there is more than one
approach. I happen to have an incomplete piece of work that should be able to
address this issue, but I have no timeline on when I may be able to propose it
here as a patch.
> Conflicting LWT transactions may be committed during topology change
>
>
> Key: CASSANDRA-15745
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15745
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Lightweight Transactions
>Reporter: Konstantin
>Priority: Normal
>
> Let's consider a cluster which consists of replicas A, B and C.
> We're adding replica D which replaces A.
> A scenario is possible when two conflicting transactions, CAS1 and CAS2, may
> be committed during replace:
> CAS2 ballot > CAS1 ballot
> CAS2 and CAS1 conflict on LWT condition, yet both of them may be committed
> in case of the following sequence of events:
> Topology change starts, advertises on C
> CAS1 starts on node C, uses {A, B, C, D}
> CAS2 starts on node A, still uses {A, B, C}
> Topology change is advertised on A
> CAS1 prepares on {B, C, D}
> CAS2 prepares and accepts on {A, B}, commits on A
> CAS1 accepts on D, then stops
> Streaming starts, topology change finishes, A is removed
> CAS3 prepares using C and D. It sees the accept of CAS1 and replays it
> Both CAS1 and CAS2 are committed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
