[jira] [Commented] (CASSANDRA-19556) Add guardrail to block DDL/DCL queries and replace alter_table_enabled guardrail
[
https://issues.apache.org/jira/browse/CASSANDRA-19556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845918#comment-17845918
]
Josh McKenzie commented on CASSANDRA-19556:
---
{quote}where Josh said that he would make an exception
{quote}
For the record, I'm one voice among {_}many{_}. That said, lazy consensus is a
thing. Looks like [~mck] would prefer it in 5.0.1 vs. this late in the release
cycle. Personally I hold a pretty strong opinion we shouldn't deprecate
features or guardrails in point releases and those kinds of changes need to
happen in a major.
So as I said before and indicated on the ML, I'm ambivalent. I think we'd be
fine with individual guardrails for each group of DDL in our next major, and we
could even create a superset guardrail (as we have on the ticket here) that
effectively overrides all of them to keep backwards compatibility.
> Add guardrail to block DDL/DCL queries and replace alter_table_enabled
> guardrail
>
>
> Key: CASSANDRA-19556
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19556
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Guardrails
>Reporter: Yuqi Yan
>Assignee: Yuqi Yan
>Priority: Normal
> Fix For: 5.0-rc, 5.x
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Sometimes we want to block DDL/DCL queries to stop new schemas being created
> or roles created. (e.g. when doing live-upgrade)
> For DDL guardrail current implementation won't block the query if it's no-op
> (e.g. CREATE TABLE...IF NOT EXISTS, but table already exists, etc. The
> guardrail check is added in apply() right after all the existence check)
> I don't have preference on either block every DDL query or check whether if
> it's no-op here. Just we have some users always run CREATE..IF NOT EXISTS..
> at startup, which is no-op but will be blocked by this guardrail and failed
> to start.
>
> 4.1 PR: [https://github.com/apache/cassandra/pull/3248]
> trunk PR: [https://github.com/apache/cassandra/pull/3275]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19556) Guardrail to block DDL/DCL queries
[ https://issues.apache.org/jira/browse/CASSANDRA-19556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843709#comment-17843709 ] Josh McKenzie commented on CASSANDRA-19556: --- bq. I think that having it more granular is just overkill Maybe. But there's a lot of users with a lot of different experiences on the project; please hit up the dev list w/a [DISCUSS] thread and see if there's a clear consensus. > Guardrail to block DDL/DCL queries > -- > > Key: CASSANDRA-19556 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19556 > Project: Cassandra > Issue Type: New Feature > Components: Feature/Guardrails >Reporter: Yuqi Yan >Assignee: Yuqi Yan >Priority: Normal > Fix For: 5.x > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Sometimes we want to block DDL/DCL queries to stop new schemas being created > or roles created. (e.g. when doing live-upgrade) > For DDL guardrail current implementation won't block the query if it's no-op > (e.g. CREATE TABLE...IF NOT EXISTS, but table already exists, etc. The > guardrail check is added in apply() right after all the existence check) > I don't have preference on either block every DDL query or check whether if > it's no-op here. Just we have some users always run CREATE..IF NOT EXISTS.. > at startup, which is no-op but will be blocked by this guardrail and failed > to start. > > 4.1 PR: [https://github.com/apache/cassandra/pull/3248] > trunk PR: [https://github.com/apache/cassandra/pull/3275] > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19556) Guardrail to block DDL/DCL queries
[
https://issues.apache.org/jira/browse/CASSANDRA-19556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843322#comment-17843322
]
Josh McKenzie commented on CASSANDRA-19556:
---
Hm. So two paths become immediately clear going from where we are (table only
DDL guardrail) to a more broadly encompassing set of guardrails.
* *Path 1:* we have a single DDL guardrail. On or off.
* {*}Path 2{*}: we have separate guardrails for categories of DDL, so
Keyspaces, roles, Tables, etc. Can be toggled on or off individually
I don't think I have an opinion either way right now. I can't think of cases in
which it makes sense to have DDL enabled for some but not other operations
off-hand, but I also favor flexibility and composability in general. So for me
it's a wash.
Given this is sort of API-ish, it'd probably be useful to hit the dev ML with a
[DISCUSS] thread and see what other people think?
> Guardrail to block DDL/DCL queries
> --
>
> Key: CASSANDRA-19556
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19556
> Project: Cassandra
> Issue Type: New Feature
> Components: Feature/Guardrails
>Reporter: Yuqi Yan
>Assignee: Yuqi Yan
>Priority: Normal
> Fix For: 5.x
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Sometimes we want to block DDL/DCL queries to stop new schemas being created
> or roles created. (e.g. when doing live-upgrade)
> For DDL guardrail current implementation won't block the query if it's no-op
> (e.g. CREATE TABLE...IF NOT EXISTS, but table already exists, etc. The
> guardrail check is added in apply() right after all the existence check)
> I don't have preference on either block every DDL query or check whether if
> it's no-op here. Just we have some users always run CREATE..IF NOT EXISTS..
> at startup, which is no-op but will be blocked by this guardrail and failed
> to start.
>
> 4.1 PR: [https://github.com/apache/cassandra/pull/3248]
> trunk PR: [https://github.com/apache/cassandra/pull/3275]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17298) Test Failure: org.apache.cassandra.cql3.MemtableSizeTest
[
https://issues.apache.org/jira/browse/CASSANDRA-17298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842927#comment-17842927
]
Josh McKenzie commented on CASSANDRA-17298:
---
{quote}when I run locally
{quote}
Locally in intellij or locally from console? I ask because on my JDK21 work, I
have a failure in `MemtableQuickTest` that's passing in IDE and failing from
console. Since it looks like they source their runtime env params from
different locations I'm starting to suspect drift there.
Also seen some other jamm related issues but they're known issues on other JIRA
tickets as well, so I wouldn't be surprised if you were wrestling with or
seeing something there as well - matches w/what you're seeing in CI.
> Test Failure: org.apache.cassandra.cql3.MemtableSizeTest
>
>
> Key: CASSANDRA-17298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17298
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Josh McKenzie
>Priority: Normal
> Fix For: 4.0.x, 4.1.x
>
>
> [https://ci-cassandra.apache.org/job/Cassandra-4.0/313/testReport/org.apache.cassandra.cql3/MemtableSizeTest/testTruncationReleasesLogSpace_2/]
> Failed 4 times in the last 30 runs. Flakiness: 27%, Stability: 86%
> Error Message
> Expected heap usage close to 49.930MiB, got 41.542MiB.
> {code}
> Stacktrace
> junit.framework.AssertionFailedError: Expected heap usage close to 49.930MiB,
> got 41.542MiB.
> at
> org.apache.cassandra.cql3.MemtableSizeTest.testSize(MemtableSizeTest.java:130)
> at org.apache.cassandra.Util.runCatchingAssertionError(Util.java:644)
> at org.apache.cassandra.Util.flakyTest(Util.java:669)
> at
> org.apache.cassandra.cql3.MemtableSizeTest.testTruncationReleasesLogSpace(MemtableSizeTest.java:61)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
> *UPDATE:* It was discovered that unit tests were running with
> memtable_allocation_type: offheap_objects when we ship C* with heap_buffers.
> So we changed that in CASSANDRA-19326, now we test with
> memtable_allocation_type: heap_buffers. As a result, this test now fails all
> the time on 4.0 and 4.1.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19490) Add foundation for independent parsing of junit based output for CI reporting to cassandra-builds
[ https://issues.apache.org/jira/browse/CASSANDRA-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842087#comment-17842087 ] Josh McKenzie commented on CASSANDRA-19490: --- Nothing here. Was swamped just getting all the things together and working; right now it's honor system. Once we have CASSANDRA-18731 this _should_ be moot since the config would indicate resource limits. The resource allocation in the system I cobbled together on top of David's work is actually significantly more constrained in both CPU and RAM compared to what we discussed and what's available in ASF CI, but it's all in bespoke .yml files for the parallelizer David wrote. > Add foundation for independent parsing of junit based output for CI reporting > to cassandra-builds > - > > Key: CASSANDRA-19490 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19490 > Project: Cassandra > Issue Type: New Feature > Components: CI >Reporter: Josh McKenzie >Assignee: Josh McKenzie >Priority: Normal > > PR attached. > For doing CI ourselves, it's useful to have a single pane of glass report > where you have a summary of results for all your suites as well as inlined > failures. This should be agnostic to any xunit based output; so long as we > co-locate the xunit data in directories adjacent to one another, the script > in the PR will generate an in-memory representation of the xunit results as > well as inline failures to an existing .html file. > The contents will need to be tweaked a bit to generate the top level branch + > sha + checkstyle + summaries information, but the vast majority of that is > already parsed and easily available within the script and can be extended > pretty trivially. > Opening up a pr to pull this into > [cassandra-builds](https://github.com/apache/cassandra-builds) since [~mck] > is actively working on that and needs these primitives. I'd expect the > contents in ci_parser to be massaged to become a more finalized, full > solution before we start to use it but no harm in the incremental merge. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19490) Add foundation for independent parsing of junit based output for CI reporting to cassandra-builds
[ https://issues.apache.org/jira/browse/CASSANDRA-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842083#comment-17842083 ] Josh McKenzie commented on CASSANDRA-19490: --- You've integrated this now right [~mck] ? i.e. can we close this out? > Add foundation for independent parsing of junit based output for CI reporting > to cassandra-builds > - > > Key: CASSANDRA-19490 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19490 > Project: Cassandra > Issue Type: New Feature > Components: CI >Reporter: Josh McKenzie >Assignee: Josh McKenzie >Priority: Normal > > PR attached. > For doing CI ourselves, it's useful to have a single pane of glass report > where you have a summary of results for all your suites as well as inlined > failures. This should be agnostic to any xunit based output; so long as we > co-locate the xunit data in directories adjacent to one another, the script > in the PR will generate an in-memory representation of the xunit results as > well as inline failures to an existing .html file. > The contents will need to be tweaked a bit to generate the top level branch + > sha + checkstyle + summaries information, but the vast majority of that is > already parsed and easily available within the script and can be extended > pretty trivially. > Opening up a pr to pull this into > [cassandra-builds](https://github.com/apache/cassandra-builds) since [~mck] > is actively working on that and needs these primitives. I'd expect the > contents in ci_parser to be massaged to become a more finalized, full > solution before we start to use it but no harm in the incremental merge. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-19101) Test Failure: org.apache.cassandra.db.commitlog.CommitlogShutdownTest failed on trunk
[
https://issues.apache.org/jira/browse/CASSANDRA-19101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie reassigned CASSANDRA-19101:
-
Assignee: Josh McKenzie
> Test Failure: org.apache.cassandra.db.commitlog.CommitlogShutdownTest failed
> on trunk
> -
>
> Key: CASSANDRA-19101
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19101
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Jacek Lewandowski
>Assignee: Josh McKenzie
>Priority: Normal
> Fix For: 5.x
>
>
> {noformat}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.cassandra.db.commitlog.CommitlogShutdownTest.testShutdownWithPendingTasks(CommitlogShutdownTest.java:96)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.jboss.byteman.contrib.bmunit.BMUnitRunner$10.evaluate(BMUnitRunner.java:393)
> at
> org.jboss.byteman.contrib.bmunit.BMUnitRunner$6.evaluate(BMUnitRunner.java:263)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.jboss.byteman.contrib.bmunit.BMUnitRunner$1.evaluate(BMUnitRunner.java:97)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
> at
> com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
> at
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
> at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
> {noformat}
> Manual testing to confirm issues found by CircleCI when testing
> CASSANDRA-18464. Run with Java 11 / IntelliJ
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18661) Update cassandra-stress to use Apache Commons CLI
[
https://issues.apache.org/jira/browse/CASSANDRA-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837738#comment-17837738
]
Josh McKenzie commented on CASSANDRA-18661:
---
{quote}Perhaps that is a topic for the ML
{quote}
+1. [~rustyrazorblade] - you able to / willing to open up a [DISCUSS] thread on
this? I've just started more aggressively using tlp-stress internally and I
think the time is ripe for us to talk about this as a project.
> Update cassandra-stress to use Apache Commons CLI
> -
>
> Key: CASSANDRA-18661
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18661
> Project: Cassandra
> Issue Type: Improvement
> Components: Tool/stress
>Reporter: Brad Schoening
>Assignee: Claude Warren
>Priority: Normal
> Labels: lhf
>
> The Apache Commons CLI library provides an API for parsing command line
> options with the package org.apache.commons.cli and this is already used by a
> dozen of existing Cassandra utilities including:
> {quote}SSTableMetadataViewer, StandaloneScrubber, StandaloneSplitter,
> SSTableExport, BulkLoader, and others.
> {quote}
> However, cassandra-stress is an outlier which uses its own custom classes to
> parse command line options with classes such as OptionsSimple. In addition,
> the options syntax for username, password, and others are not aligned with
> the format used by CQLSH.
> Currently, there are > 5K lines of code in 'settings' which appears to just
> process command line args.
> This suggestion is to:
>
> a) Upgrade cassandra-stress to use Apache Commons CLI (no new dependencies
> are required as this library is already used by the project)
>
> b) Align the cassandra-stress CLI options with those in CQLSH,
>
> {quote}For example, using the new syntax like CQLSH:
> {quote}
>
> cassandra-stress -username foo -password bar
> {quote}and replacing the old syntax:
> {quote}
> cassandra-stress -mode username=foo and password=bar
>
> This will simplify and unify the code base, eliminate code and reduce the
> confusion between similar named classes such as
> org.apache.cassandra.stress.settings.\{Option, OptionsMulti, OptionsSimple}
> and org.apache.commons.cli.{Option, OptionGroup, Options)
>
> Note: documentation will need to be updated as well
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18831) Enable Cassandra to build and run under JDK21
[ https://issues.apache.org/jira/browse/CASSANDRA-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837735#comment-17837735 ] Josh McKenzie commented on CASSANDRA-18831: --- Sorry, think this is me abusing colloquialisms of US english. Basically, I'm only reactively updating dependencies where required to get them to work on JDK21. I'm not proactively going to update any dependencies unless required by the JDK upgrade and environment. The 3 heuristics I use to determine whether to advocate for a dependency bump: # Is there a CVE? # Do we need to bump the dep to support some other change (another lib bump, JDK version change, etc)? # Is there new functionality we need from either a new dep or a version update? In the case of things working on JDK21, I'm limiting my considerations to #2 above; only updating where required to have things work on JDK21. My guess is the gap from 17->21 is smaller than the gap from 11->17 was so I'm not running into as many breakages as you did w/that work, for which I am very grateful to you. :) > Enable Cassandra to build and run under JDK21 > - > > Key: CASSANDRA-18831 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18831 > Project: Cassandra > Issue Type: Task > Components: Build >Reporter: Achilles Benetopoulos >Assignee: Josh McKenzie >Priority: Normal > Fix For: 5.x > > Attachments: jdk21-patch > > > This patch builds on the work in CASSANDRA-16895 that added JDK17 to the list > of supported Java versions, and extends that work to enable building and > running Cassandra under JDK21. > The following commits comprise the changes included in the attached patch: > - > [https://github.com/apache/cassandra/commit/b15d4d6980e787ab5f3405ca8cb17a9c92a4aa47] > - > [https://github.com/apache/cassandra/commit/0c5df38dafe58bfec8924e81507bb604e1543897] > - > [https://github.com/apache/cassandra/commit/6506b7279d98eed4b2b65b71e0c6f41eb78c6913] > - > [https://github.com/apache/cassandra/commit/564cbd534c5a975cda0c629c14c68c8745b41451] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19551) CCM nodes share the same environment variable map breaking upgrade tests
[
https://issues.apache.org/jira/browse/CASSANDRA-19551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie updated CASSANDRA-19551:
--
Status: Ready to Commit (was: Review In Progress)
> CCM nodes share the same environment variable map breaking upgrade tests
>
>
> Key: CASSANDRA-19551
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19551
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest/python
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Normal
> Fix For: 5.x
>
> Attachments: ci_summary.html
>
>
> In {{node.py}} {{__environment_variables}} is generally always set with a map
> that is passed in from {{cluster.py}} so it is [shared between
> nodes](https://github.com/riptano/ccm/blob/ac264706c8ca007cc584871ce907d48db334d36d/ccmlib/node.py#L151)
> and if nodes modify the map, such as in {{start}} when [updating the Java
> version](https://github.com/riptano/ccm/blob/ac264706c8ca007cc584871ce907d48db334d36d/ccmlib/node.py#L860)
> then when {{get_env}} runs it will [overwrite the Java
> version](https://github.com/riptano/ccm/blob/ac264706c8ca007cc584871ce907d48db334d36d/ccmlib/node.py#L244)
> that is selected by {{update_java_version}}.
> This results in {{nodetool drain}} failing when upgrading from 3.11 to 4.0 in
> some of the upgrade tests because after the first node upgrades to 4.0 it's
> not longer possible for the subsequent nodes to select a Java version that
> isn't 11 because it's overridden by {{__environment_variables}}.
> I'm not even 100% clear on why the code in {{start}} should update
> {{__environment_variables}} at all if we calculate the correct java version
> on every invocation of other tools.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19551) CCM nodes share the same environment variable map breaking upgrade tests
[
https://issues.apache.org/jira/browse/CASSANDRA-19551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie updated CASSANDRA-19551:
--
Reviewers: Joshua McKenzie, Josh McKenzie (was: Joshua McKenzie)
Joshua McKenzie, Josh McKenzie (was: Josh McKenzie, Joshua
McKenzie)
Status: Review In Progress (was: Patch Available)
> CCM nodes share the same environment variable map breaking upgrade tests
>
>
> Key: CASSANDRA-19551
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19551
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest/python
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Normal
> Fix For: 5.x
>
> Attachments: ci_summary.html
>
>
> In {{node.py}} {{__environment_variables}} is generally always set with a map
> that is passed in from {{cluster.py}} so it is [shared between
> nodes](https://github.com/riptano/ccm/blob/ac264706c8ca007cc584871ce907d48db334d36d/ccmlib/node.py#L151)
> and if nodes modify the map, such as in {{start}} when [updating the Java
> version](https://github.com/riptano/ccm/blob/ac264706c8ca007cc584871ce907d48db334d36d/ccmlib/node.py#L860)
> then when {{get_env}} runs it will [overwrite the Java
> version](https://github.com/riptano/ccm/blob/ac264706c8ca007cc584871ce907d48db334d36d/ccmlib/node.py#L244)
> that is selected by {{update_java_version}}.
> This results in {{nodetool drain}} failing when upgrading from 3.11 to 4.0 in
> some of the upgrade tests because after the first node upgrades to 4.0 it's
> not longer possible for the subsequent nodes to select a Java version that
> isn't 11 because it's overridden by {{__environment_variables}}.
> I'm not even 100% clear on why the code in {{start}} should update
> {{__environment_variables}} at all if we calculate the correct java version
> on every invocation of other tools.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19551) CCM nodes share the same environment variable map breaking upgrade tests
[
https://issues.apache.org/jira/browse/CASSANDRA-19551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837721#comment-17837721
]
Josh McKenzie commented on CASSANDRA-19551:
---
+1 on the PR. As I mentioned in the PR notes, looks like the cluster object is
the only other mutable thing we assign by ref but I can't think of a situation
in which that'd be mutated externally so should be fine.
> CCM nodes share the same environment variable map breaking upgrade tests
>
>
> Key: CASSANDRA-19551
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19551
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest/python
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Normal
> Fix For: 5.x
>
> Attachments: ci_summary.html
>
>
> In {{node.py}} {{__environment_variables}} is generally always set with a map
> that is passed in from {{cluster.py}} so it is [shared between
> nodes](https://github.com/riptano/ccm/blob/ac264706c8ca007cc584871ce907d48db334d36d/ccmlib/node.py#L151)
> and if nodes modify the map, such as in {{start}} when [updating the Java
> version](https://github.com/riptano/ccm/blob/ac264706c8ca007cc584871ce907d48db334d36d/ccmlib/node.py#L860)
> then when {{get_env}} runs it will [overwrite the Java
> version](https://github.com/riptano/ccm/blob/ac264706c8ca007cc584871ce907d48db334d36d/ccmlib/node.py#L244)
> that is selected by {{update_java_version}}.
> This results in {{nodetool drain}} failing when upgrading from 3.11 to 4.0 in
> some of the upgrade tests because after the first node upgrades to 4.0 it's
> not longer possible for the subsequent nodes to select a Java version that
> isn't 11 because it's overridden by {{__environment_variables}}.
> I'm not even 100% clear on why the code in {{start}} should update
> {{__environment_variables}} at all if we calculate the correct java version
> on every invocation of other tools.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19478) Add CI for python 3.6 back to 5.0
[ https://issues.apache.org/jira/browse/CASSANDRA-19478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837395#comment-17837395 ] Josh McKenzie commented on CASSANDRA-19478: --- We're talking python version on what, cqlshlib tests only? Makes sense to me for pre-commit smoke suites to check that as well if we still support it, since those suites are lightweight. > Add CI for python 3.6 back to 5.0 > - > > Key: CASSANDRA-19478 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19478 > Project: Cassandra > Issue Type: Task > Components: CI >Reporter: Brandon Williams >Priority: Normal > Fix For: 5.0.x > > > 5.0 doesn't currently run python 3.6 (but branches below it do) however in > CASSANDRA-19467 we have restored python 3.6 support so we should have test > coverage for it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18831) Enable Cassandra to build and run under JDK21
[ https://issues.apache.org/jira/browse/CASSANDRA-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836255#comment-17836255 ] Josh McKenzie commented on CASSANDRA-18831: --- Can provide a brief update on this. We have some dependencies that need to be bumped (chronicle*, mockito, bytebuddy), some startup args that need to be tweaked, and some various and sundry other things. I'm working through test failures now but when I have something that builds and passes on CI my plan is to attach it here even if we don't yet have things configured on CI. I'll pursue getting that set up formally in the ASF space after we have general support merged, at the risk of said support atrophying between when we merge it in and when we get it integrated into CI. > Enable Cassandra to build and run under JDK21 > - > > Key: CASSANDRA-18831 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18831 > Project: Cassandra > Issue Type: Task > Components: Build >Reporter: Achilles Benetopoulos >Assignee: Josh McKenzie >Priority: Normal > Fix For: 5.x > > Attachments: jdk21-patch > > > This patch builds on the work in CASSANDRA-16895 that added JDK17 to the list > of supported Java versions, and extends that work to enable building and > running Cassandra under JDK21. > The following commits comprise the changes included in the attached patch: > - > [https://github.com/apache/cassandra/commit/b15d4d6980e787ab5f3405ca8cb17a9c92a4aa47] > - > [https://github.com/apache/cassandra/commit/0c5df38dafe58bfec8924e81507bb604e1543897] > - > [https://github.com/apache/cassandra/commit/6506b7279d98eed4b2b65b71e0c6f41eb78c6913] > - > [https://github.com/apache/cassandra/commit/564cbd534c5a975cda0c629c14c68c8745b41451] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-19490) Add foundation for independent parsing of junit based output for CI reporting to cassandra-builds
Josh McKenzie created CASSANDRA-19490: - Summary: Add foundation for independent parsing of junit based output for CI reporting to cassandra-builds Key: CASSANDRA-19490 URL: https://issues.apache.org/jira/browse/CASSANDRA-19490 Project: Cassandra Issue Type: New Feature Components: CI Reporter: Josh McKenzie Assignee: Josh McKenzie PR attached. For doing CI ourselves, it's useful to have a single pane of glass report where you have a summary of results for all your suites as well as inlined failures. This should be agnostic to any xunit based output; so long as we co-locate the xunit data in directories adjacent to one another, the script in the PR will generate an in-memory representation of the xunit results as well as inline failures to an existing .html file. The contents will need to be tweaked a bit to generate the top level branch + sha + checkstyle + summaries information, but the vast majority of that is already parsed and easily available within the script and can be extended pretty trivially. Opening up a pr to pull this into [cassandra-builds](https://github.com/apache/cassandra-builds) since [~mck] is actively working on that and needs these primitives. I'd expect the contents in ci_parser to be massaged to become a more finalized, full solution before we start to use it but no harm in the incremental merge. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18661) Update cassandra-stress to use Apache Commons CLI
[
https://issues.apache.org/jira/browse/CASSANDRA-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17827783#comment-17827783
]
Josh McKenzie commented on CASSANDRA-18661:
---
So 1: this looks much improved. Great work!
And 2: In the grand tradition of this project, introducing something late in
the conversation that's a 90 degree right turn, has anyone considered
deprecating and removing cassandra-stress in favor of
[tlp-stress|https://github.com/thelastpickle/tlp-stress] or some variant
thereof? I believe there are some forks of it out and about that folks might be
willing to donate to the foundation, and making that the de facto in-tree
stress tool seems like it'd be a boon for our users.
[~rustyrazorblade] - got any thoughts here?
> Update cassandra-stress to use Apache Commons CLI
> -
>
> Key: CASSANDRA-18661
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18661
> Project: Cassandra
> Issue Type: Improvement
> Components: Tool/stress
>Reporter: Brad Schoening
>Assignee: Claude Warren
>Priority: Normal
> Labels: lhf
>
> The Apache Commons CLI library provides an API for parsing command line
> options with the package org.apache.commons.cli and this is already used by a
> dozen of existing Cassandra utilities including:
> {quote}SSTableMetadataViewer, StandaloneScrubber, StandaloneSplitter,
> SSTableExport, BulkLoader, and others.
> {quote}
> However, cassandra-stress is an outlier which uses its own custom classes to
> parse command line options with classes such as OptionsSimple. In addition,
> the options syntax for username, password, and others are not aligned with
> the format used by CQLSH.
> Currently, there are > 5K lines of code in 'settings' which appears to just
> process command line args.
> This suggestion is to:
>
> a) Upgrade cassandra-stress to use Apache Commons CLI (no new dependencies
> are required as this library is already used by the project)
>
> b) Align the cassandra-stress CLI options with those in CQLSH,
>
> {quote}For example, using the new syntax like CQLSH:
> {quote}
>
> cassandra-stress -username foo -password bar
> {quote}and replacing the old syntax:
> {quote}
> cassandra-stress -mode username=foo and password=bar
>
> This will simplify and unify the code base, eliminate code and reduce the
> confusion between similar named classes such as
> org.apache.cassandra.stress.settings.\{Option, OptionsMulti, OptionsSimple}
> and org.apache.commons.cli.{Option, OptionGroup, Options)
>
> Note: documentation will need to be updated as well
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18831) Enable Cassandra to build and run under JDK21
[
https://issues.apache.org/jira/browse/CASSANDRA-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17827782#comment-17827782
]
Josh McKenzie commented on CASSANDRA-18831:
---
{quote}I suspect we should have a new epic - similar to the one we have for
introducing JDK17 (CASSANDRA-16895). However, I am optimistic that the jump
between 17 and 21 will be less work as we have caught up on maintenance.
{quote}
Got my fingers crossed here as well. I'll be getting this together for internal
CI infra (which closely mirrors upstream, both circle and the new standalone
jenkinsfile scripts) and circle back here once I have some insight into how
this looks w/regards to CI. All this work will be reflected in CASSANDRA-18731
later this year.
My hope is that we don't run into as many pain points as we did from the 11->17
transition, but if we do I have all the context primed in my head to burn
through it in CI at this point. I'll keep you posted [~e.dimitrova]
> Enable Cassandra to build and run under JDK21
> -
>
> Key: CASSANDRA-18831
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18831
> Project: Cassandra
> Issue Type: Task
> Components: Build
>Reporter: Achilles Benetopoulos
>Assignee: Josh McKenzie
>Priority: Normal
> Fix For: 5.x
>
> Attachments: jdk21-patch
>
>
> This patch builds on the work in CASSANDRA-16895 that added JDK17 to the list
> of supported Java versions, and extends that work to enable building and
> running Cassandra under JDK21.
> The following commits comprise the changes included in the attached patch:
> -
> [https://github.com/apache/cassandra/commit/b15d4d6980e787ab5f3405ca8cb17a9c92a4aa47]
> -
> [https://github.com/apache/cassandra/commit/0c5df38dafe58bfec8924e81507bb604e1543897]
> -
> [https://github.com/apache/cassandra/commit/6506b7279d98eed4b2b65b71e0c6f41eb78c6913]
> -
> [https://github.com/apache/cassandra/commit/564cbd534c5a975cda0c629c14c68c8745b41451]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-18831) Enable Cassandra to build and run under JDK21
[ https://issues.apache.org/jira/browse/CASSANDRA-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie reassigned CASSANDRA-18831: - Assignee: Josh McKenzie > Enable Cassandra to build and run under JDK21 > - > > Key: CASSANDRA-18831 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18831 > Project: Cassandra > Issue Type: Task > Components: Build >Reporter: Achilles Benetopoulos >Assignee: Josh McKenzie >Priority: Normal > Fix For: 5.x > > Attachments: jdk21-patch > > > This patch builds on the work in CASSANDRA-16895 that added JDK17 to the list > of supported Java versions, and extends that work to enable building and > running Cassandra under JDK21. > The following commits comprise the changes included in the attached patch: > - > [https://github.com/apache/cassandra/commit/b15d4d6980e787ab5f3405ca8cb17a9c92a4aa47] > - > [https://github.com/apache/cassandra/commit/0c5df38dafe58bfec8924e81507bb604e1543897] > - > [https://github.com/apache/cassandra/commit/6506b7279d98eed4b2b65b71e0c6f41eb78c6913] > - > [https://github.com/apache/cassandra/commit/564cbd534c5a975cda0c629c14c68c8745b41451] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19406) Optimize pre-commit test suite
[ https://issues.apache.org/jira/browse/CASSANDRA-19406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-19406: -- Description: See the following ML threads: * Branimir discussing having 2 supported configs which included a lot of back and forth about what we're testing: [link|https://lists.apache.org/thread/qf5c3hhz6qkpyqvbd3sppzlmftlc0bw0] * Follow up DISCUSS thread notifying about this ticket: [link|https://lists.apache.org/thread/r433j1mk111v102z4nr1m9l1fqf41fbr] The pre-commit test suite has a lot of redundancy in our current approach while also leaving certain configurations untested or undertested. We need to address both sides of this. > Optimize pre-commit test suite > -- > > Key: CASSANDRA-19406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19406 > Project: Cassandra > Issue Type: Epic >Reporter: Jacek Lewandowski >Priority: Normal > > See the following ML threads: > * Branimir discussing having 2 supported configs which included a lot of > back and forth about what we're testing: > [link|https://lists.apache.org/thread/qf5c3hhz6qkpyqvbd3sppzlmftlc0bw0] > * Follow up DISCUSS thread notifying about this ticket: > [link|https://lists.apache.org/thread/r433j1mk111v102z4nr1m9l1fqf41fbr] > The pre-commit test suite has a lot of redundancy in our current approach > while also leaving certain configurations untested or undertested. We need to > address both sides of this. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19406) Optimize pre-commit test suite
[ https://issues.apache.org/jira/browse/CASSANDRA-19406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17819350#comment-17819350 ] Josh McKenzie commented on CASSANDRA-19406: --- I don't think we really came to a consensus on the dev ML threads did we? Or do we have a lazy consensus on this and are good to move forward? > Optimize pre-commit test suite > -- > > Key: CASSANDRA-19406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19406 > Project: Cassandra > Issue Type: Epic >Reporter: Jacek Lewandowski >Priority: Normal > > See the following ML threads: > * Branimir discussing having 2 supported configs which included a lot of > back and forth about what we're testing: > [link|https://lists.apache.org/thread/qf5c3hhz6qkpyqvbd3sppzlmftlc0bw0] > * Follow up DISCUSS thread notifying about this ticket: > [link|https://lists.apache.org/thread/r433j1mk111v102z4nr1m9l1fqf41fbr] > The pre-commit test suite has a lot of redundancy in our current approach > while also leaving certain configurations untested or undertested. We need to > address both sides of this. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18998) Expose CDC via CQL
[ https://issues.apache.org/jira/browse/CASSANDRA-18998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17805802#comment-17805802 ] Josh McKenzie commented on CASSANDRA-18998: --- I can speak to some of the thinking "Back in the Day". The heap and compute pressure of doing CDC into another table (ser/deser, LSM tree maintenance, write amplification, compaction, etc) would make the solution pretty unattractive to most use-cases. Further, leaning on coordinator-level push of a mutation after write opens up pandora's box w/replica failure and orphaned data in edge cases (i.e. you get write acks from 3 replicas, all 3 die, but you've acked the CDC data downstream and the source data that prompted it is now lost w/out any meaningful way to reconcile that you now have orphaned data in your downstream system - very similar problem to what we've run across with materialized views...) Having the sidecar parse local files, a separate CDC-specific file from the CommitLog w/CDC only information, and deduping before feeding downstream would all be pretty viable things to do from the sidecar space IMO. > Expose CDC via CQL > -- > > Key: CASSANDRA-18998 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18998 > Project: Cassandra > Issue Type: Improvement > Components: Feature/Change Data Capture >Reporter: Stefen Hillman >Assignee: Dinesh Joshi >Priority: Normal > > It looks like there was originally a plan to expose change data externally > via CQL for any client to consume, as noted under Design Options #4 on the > original design document: > [https://docs.google.com/document/d/1ZxCWYkeZTquxsvf5hdPc0fiUnUHna8POvgt6TIzML4Y/edit?usp=sharing.] > > It appears that this was ultimately not implemented. Is there a plan to > improve CDC in the future with this functionality? The current implementation > is tedious to use - you have to write a local agent that lives on every > single c* node and then implement the Apache-provided CommitLogReader in > order to parse the files. On top of that, it makes deduplication and ordering > basically impossible if there isn't an inherently ordered property. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17812) Rate-limit new client connection setup to avoid overwhelming bcrypt
[ https://issues.apache.org/jira/browse/CASSANDRA-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17804390#comment-17804390 ] Josh McKenzie commented on CASSANDRA-17812: --- Since this is an improvement we only target unreleased versions with it; while this patch helps with stabilization in general it's also not a bugfix in and of itself. The code-changes for it aren't that extensive so creating a one-off patch to apply to 4.1 and doing a local release shouldn't be _too_ painful for you to do if that's something you desperately need. > Rate-limit new client connection setup to avoid overwhelming bcrypt > --- > > Key: CASSANDRA-17812 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17812 > Project: Cassandra > Issue Type: Improvement > Components: Feature/Encryption >Reporter: Josh McKenzie >Assignee: Josh McKenzie >Priority: Normal > Fix For: 5.0-alpha1, 5.0 > > > A flood of reconnects can cause a ton of pain at the bcrypt phase of > validating incoming connections. While this shouldn't happen during normal > operations, we need a rate limit server side - if there's a bad client out > there (version and/or configuration) that misbehaves, a way to cap the pain > on a server is quite useful. Right now the only option is to cap the total > connections which has other issues that aren't an ideal tradeoff (inability > to connect, etc). > Moving authentication requests to a small, separate pool will prevent > starvation handling all other requests. If the authExecutor pool backs up it > may cause authentication timeouts, but the clients should back off and retry > while the rest of the system continues to make progress. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18731) Add declarative root CI structure
[
https://issues.apache.org/jira/browse/CASSANDRA-18731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17790270#comment-17790270
]
Josh McKenzie commented on CASSANDRA-18731:
---
{quote}Today, our 5.0+ requirements to the CI pre-commit pipeline are...
{quote}
I think we should revisit this on the dev ML; I don't fully agree w/what you've
outlined above and this is something we should discuss as a project community,
not mandate on a comment in a JIRA.
[~e.dimitrova] made the excellent point that as the JDK authors keep tightening
down what's allowed in terms of unsafe and reflection-based access, we're more
likely to see failures on the highest supported JDK version rather than the
lowest for example, meaning the python dtest suites and jvm-dtest-upgrade
should likely be calibrated to "highest supported and down", vs. "lowest
supported and up".
Further, I don't agree with (and we've discussed) the need to run test-cdc (or
it existing at all vs. it just being enabled for all tests), the compression,
oa, or trie suites as pre-commit smoke suites, etc. I have a strong belief that
we should expect very infrequent test failures on non-default configurations
vs. the base; that functionality is supposed to be API compatible and if it
works on one configuration, it works on all. While there will no doubt be
flakes, the amount of compute required to run all those pre-commit as a smoke
suite effectively removes the "smoke" aspect of it and just makes pre-commit a
vast majority of the cost of a full run.
Same for testing everything in no-vnode and vnode combination; that's so
terribly wasteful that I just can't agree with doing that for everything
pre-commit (in any env, circle, new, or ASF-CI). I would very much prefer we
take the approach you enumerated on the dev ML:
{quote}Can we get these tests temporarily annotated as skipped while all the
subtickets to 19055 are being worked on ?
{quote}
We have the current example of the TCM merge on CASSANDRA-19055 to immediately
inspect as to whether this approach is sufficient or not. With an _incredibly
large_ patch in the form of TCM (88k+ LoC, 900+ files touched), we have less
than a .002% test failure injection rate using the above restricted smoke
heuristic, and many of them look to be circle ci env specific and not asf ci.
(Of note - the artifact building bit definitely needs to be resolved; we should
consider extending an "artifact build only" target on ASF CI people can
parameterize with feature branches to make sure they haven't broken the various
builds?)
And for repeated runs, we discussed the fact that not having support for
repeated runs in ASF CI meant we also didn't want to put down a hard blocker on
requiring those runs on other pre-commit environments. Has your position
changed on that? Regardless, that's just a question of prioritization; it's
relatively trivial to finish up support for that if we deem it an immediate
priority.
> Add declarative root CI structure
> -
>
> Key: CASSANDRA-18731
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18731
> Project: Cassandra
> Issue Type: Improvement
> Components: CI
>Reporter: Josh McKenzie
>Assignee: Josh McKenzie
>Priority: Normal
> Fix For: 5.x
>
>
> Currently we have a somewhat declarative structure in .circleci, however
> there's still quite a bit that's baked in (resource limitations, parallelism,
> which suites qualify for which pipelines (pre-commit vs. post-commit vs.
> ???), etc). Further, while CASSANDRA-18133 brings the build scripts in-tree,
> all these parameters (pipelines, env vars, job definitions, resource
> constraints) are all still scattered throughout the shell scripts and/or
> reliant on the {{JenkinsFile}} to determine what suites comprise what
> pipelines.
> This ticket aims to decouple the definition of pipelines and jobs for CI from
> the implementations themselves. The goal here is to define, establish, and
> test both the base config and some helper methods to provide _other_
> configurations (circle, ASF CI, etc) the tools they need to programmatically
> inherit base CI config from a purely declarative structure.
> Follow-up tickets will involve rewriting the in-tree build scripts and
> JenkinsFile generation to rely on this structure, as well as integrating the
> config parsing unit tests into our CI.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRASC-83) Require gossip to be enabled for ring and token ranges mapping endpoints
[ https://issues.apache.org/jira/browse/CASSANDRASC-83?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786086#comment-17786086 ] Josh McKenzie commented on CASSANDRASC-83: -- +1 here as well - very clean! > Require gossip to be enabled for ring and token ranges mapping endpoints > > > Key: CASSANDRASC-83 > URL: https://issues.apache.org/jira/browse/CASSANDRASC-83 > Project: Sidecar for Apache Cassandra > Issue Type: Improvement > Components: Rest API >Reporter: Francisco Guerrero >Assignee: Francisco Guerrero >Priority: Normal > Labels: pull-request-available > > The ring and token ranges mapping endpoints leverage JMX to produce the > payload. Ring and token ranges mapping information is derived from gossip > information available to the Cassandra instance. If gossip has been disabled, > the information might provide an inconsistent view of the cluster which is > undesirable. We should instead return a 503 (Service Unavailable) to the > client whenever the client tries to access these endpoints and gossip has > been disabled. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19003) Get Sidecar port through CassandraContext for more flexibility
[ https://issues.apache.org/jira/browse/CASSANDRA-19003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784556#comment-17784556 ] Josh McKenzie commented on CASSANDRA-19003: --- +1 - LGTM. > Get Sidecar port through CassandraContext for more flexibility > --- > > Key: CASSANDRA-19003 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19003 > Project: Cassandra > Issue Type: Task > Components: Analytics Library >Reporter: Saranya Krishnakumar >Assignee: Saranya Krishnakumar >Priority: Normal > Time Spent: 10m > Remaining Estimate: 0h > > We get sidecar port from `BulkSparkConf` but it will be better if we get it > from `CassandraContext` provides more flexibility -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18635) Test failure: org.apache.cassandra.distributed.test.UpgradeSSTablesTest
[
https://issues.apache.org/jira/browse/CASSANDRA-18635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17783775#comment-17783775
]
Josh McKenzie commented on CASSANDRA-18635:
---
I'll try and carve out time to take a closer look later this week.
> Test failure: org.apache.cassandra.distributed.test.UpgradeSSTablesTest
> ---
>
> Key: CASSANDRA-18635
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18635
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest/java
>Reporter: Brandon Williams
>Assignee: Josh McKenzie
>Priority: Normal
> Fix For: 5.0-rc, 5.x
>
>
> Seen here:
> https://app.circleci.com/pipelines/github/driftx/cassandra/1095/workflows/6114e2e3-8dcc-4bb0-b664-ae7d82c3349f/jobs/33405/tests
> {noformat}
> junit.framework.AssertionFailedError: expected:<0> but was:<2>
> at
> org.apache.cassandra.distributed.test.UpgradeSSTablesTest.upgradeSSTablesInterruptsOngoingCompaction(UpgradeSSTablesTest.java:86)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-18635) Test failure: org.apache.cassandra.distributed.test.UpgradeSSTablesTest
[
https://issues.apache.org/jira/browse/CASSANDRA-18635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie reassigned CASSANDRA-18635:
-
Assignee: Josh McKenzie
> Test failure: org.apache.cassandra.distributed.test.UpgradeSSTablesTest
> ---
>
> Key: CASSANDRA-18635
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18635
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest/java
>Reporter: Brandon Williams
>Assignee: Josh McKenzie
>Priority: Normal
> Fix For: 5.0-rc, 5.x
>
>
> Seen here:
> https://app.circleci.com/pipelines/github/driftx/cassandra/1095/workflows/6114e2e3-8dcc-4bb0-b664-ae7d82c3349f/jobs/33405/tests
> {noformat}
> junit.framework.AssertionFailedError: expected:<0> but was:<2>
> at
> org.apache.cassandra.distributed.test.UpgradeSSTablesTest.upgradeSSTablesInterruptsOngoingCompaction(UpgradeSSTablesTest.java:86)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18731) Add declarative root CI structure
[
https://issues.apache.org/jira/browse/CASSANDRA-18731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie updated CASSANDRA-18731:
--
Status: Open (was: Patch Available)
> Add declarative root CI structure
> -
>
> Key: CASSANDRA-18731
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18731
> Project: Cassandra
> Issue Type: Improvement
> Components: CI
>Reporter: Josh McKenzie
>Assignee: Josh McKenzie
>Priority: Normal
> Fix For: 5.x
>
>
> Currently we have a somewhat declarative structure in .circleci, however
> there's still quite a bit that's baked in (resource limitations, parallelism,
> which suites qualify for which pipelines (pre-commit vs. post-commit vs.
> ???), etc). Further, while CASSANDRA-18133 brings the build scripts in-tree,
> all these parameters (pipelines, env vars, job definitions, resource
> constraints) are all still scattered throughout the shell scripts and/or
> reliant on the {{JenkinsFile}} to determine what suites comprise what
> pipelines.
> This ticket aims to decouple the definition of pipelines and jobs for CI from
> the implementations themselves. The goal here is to define, establish, and
> test both the base config and some helper methods to provide _other_
> configurations (circle, ASF CI, etc) the tools they need to programmatically
> inherit base CI config from a purely declarative structure.
> Follow-up tickets will involve rewriting the in-tree build scripts and
> JenkinsFile generation to rely on this structure, as well as integrating the
> config parsing unit tests into our CI.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18665) Update jenkins groovy dsl to use in-tree scripts
[
https://issues.apache.org/jira/browse/CASSANDRA-18665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17771583#comment-17771583
]
Josh McKenzie commented on CASSANDRA-18665:
---
Looks like you get pre-github-PR style JIRA comments from me on this one. :D
- Agree w/Maxim; I think we should drop support for anything not on the
maintained list here: [https://cassandra.apache.org/_/download.html]. There's
a non-trivial burden associated with us carrying these scripts, support, and
artifacts along.
- Consider removing duplication of supported versions between
{{cassandra-artifacts.sh}} and {{{}cassandra-dtest-pytest.sh{}}}; maintenance
error-prone.
- Same for regx_version in {{cassandra-test.sh}} (i.e. having a shared .sh
they all
source w/a function that returns the regx would simplify)
- Consider {{mkdir -p}} instead of {{mkdir}} in execute call in
{{prepare_release.sh}}
to be a bit more fault tolerant. Or nuking and re-creating. /shrug
- Line 288: wildcard {{cp build/cassandra*}} may end up catching things we
don't
want, either now or in the future. Is that the right scope for the glob?
h3. *{{cassandra_job_dsl_seed.groovy}}*
- Is {{dtest-upgrade-upgrade}} a typo on line 74? Only occurrance I could find
in
either this repo or C* proper. Maybe it's supposed to be
{{{}dtest-upgrade-large{}}}?
- Why 600 for the timeout on artifact building? Does that include queue time as
well or just execution? i.e. if the latter... 10 hours is a long time. :)
- In our buildSteps in {{{}matrixPostBuildScript{}}}, it might be nice to get
some kind
of echo of a) how much data we pruned from tmp and /tmp, and b) how much data
we're still using. Something like: \{{find $1 -type f -exec du -ah {} + | sort
-rh | head -n 20}} (or whatever # suits your fancy) has been helpful for some
of my testing to try and find what big consumers are.
- Kind of nitty, but is there a reason we \{{cloneOption { shallow(false) }}}?
Seems we'd save some space by doing shallow of branch only.
- Why are we testing on python3.6? (i.e. we're defaulting to 3.7 in
{{{}docker/run-tests.sh{}}}?)
- Have a diff between {{HEAD}} and {{c2aa282}} camping out around line 823
- Is there any way to get away from the significant redundancy of specifying
the clean commands on each job?
- nit: indentation on Branch Pipelines header comment
h3. *{{.build/docker/ubuntu2004_test.docker}}*
- So the \{1..11}seq for ccm creation - what's the failure rate and reason look
like there? Why 11 attempts?
- If we initialize the ccm repos w/cassandra builds at docker image build time,
we're getting a pretty baked in version of things that needs to be updated
each time we cut a new release right? Why do this at docker image compile
time rather than runtime, and/or just rely on
{{{}cassandra-dtest/upgrade_tests/upgrade_manifes.py{}}}'s list of latest
versions to
correctly pull down and build cassandra instances then?
- Along with that, we pre-cache C* versions on python 3.7 env right? Not on 3.6
or 3.11 which we pull down on the image?
h3. *{{upgrade_base.py}}*
- What's the story with the df and mount debugging on cluster install_dir?
> Update jenkins groovy dsl to use in-tree scripts
>
>
> Key: CASSANDRA-18665
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18665
> Project: Cassandra
> Issue Type: Task
> Components: CI
>Reporter: Michael Semb Wever
>Assignee: Michael Semb Wever
>Priority: Normal
> Fix For: 5.0.x, 5.x
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15284) AssertionError while scrubbing sstable
[
https://issues.apache.org/jira/browse/CASSANDRA-15284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764627#comment-17764627
]
Josh McKenzie commented on CASSANDRA-15284:
---
Probably going to be another week or two before I can come up for air and check
this out.
> AssertionError while scrubbing sstable
> --
>
> Key: CASSANDRA-15284
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15284
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Compression
>Reporter: Gianluigi Tiesi
>Priority: Normal
> Fix For: 3.11.x, 4.0.x, 4.1.x
>
> Attachments: assert-comp-meta.diff
>
>
> I've got a damaged data file but while trying to run scrub (online or
> offline) I always get this
> error:
>
> {code:java}
> -- StackTrace --
> java.lang.AssertionError
> at
> org.apache.cassandra.io.compress.CompressionMetadata$Chunk.(CompressionMetadata.java:474)
> at
> org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:239)
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:163)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:331)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:336)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinalEarly(BigTableWriter.java:318)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.switchWriter(SSTableRewriter.java:322)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:370)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:357)
> at
> org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:291)
> at
> org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:1010)
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$200(CompactionManager.java:83)
> at
> org.apache.cassandra.db.compaction.CompactionManager$3.execute(CompactionManager.java:391)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:312)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> At the moment I've moved away the corrupted file, If you need more info fell
> free to ask
>
> According to the source
> [https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java#L474]
> looks like the requested chung length is <= 0
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18065) Flaky test org.apache.cassandra.tools.TopPartitionsTest#testStartAndStopScheduledSampling
[
https://issues.apache.org/jira/browse/CASSANDRA-18065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759023#comment-17759023
]
Josh McKenzie commented on CASSANDRA-18065:
---
bq. This fails on its initial commit from CASSANDRA-17821, here:
Happy to take a look but it'll be a couple weeks before I can get to it.
> Flaky test
> org.apache.cassandra.tools.TopPartitionsTest#testStartAndStopScheduledSampling
> -
>
> Key: CASSANDRA-18065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18065
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Andres de la Peña
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 5.0.x, 5.x
>
>
> The test
> {{org.apache.cassandra.tools.TopPartitionsTest#testStartAndStopScheduledSampling}}
> is fails intermittently on trunk with CircleCI:
> *
> https://app.circleci.com/pipelines/github/adelapena/cassandra/2508/workflows/92f054d7-9386-498f-9ba4-330181cd4782/jobs/24692
> *
> https://app.circleci.com/pipelines/github/adelapena/cassandra/2511/workflows/7aba8baa-0a6d-404a-b08b-c6a8078caca3/jobs/24706/tests
> The failure looks like:
> {code}
> junit.framework.AssertionFailedError: Scheduled sampled tasks should be
> removed expected:<[]> but was:<[*.*]>
> at
> org.apache.cassandra.tools.TopPartitionsTest.testStartAndStopScheduledSampling(TopPartitionsTest.java:116)
> {code}
> I haven't seen this failure on Butler/Jenkins yet, but it can be reproduced
> with the CircleCI multiplexer:
> {code}
> .circleci/generate.sh -m \
> -e REPEATED_UTESTS_COUNT=2000 \
> -e REPEATED_UTESTS=org.apache.cassandra.tools.TopPartitionsTest
> {code}
> It seems to fail 11 times on 2000 runs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18534) Make sstable format configurable per table
[
https://issues.apache.org/jira/browse/CASSANDRA-18534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759022#comment-17759022
]
Josh McKenzie commented on CASSANDRA-18534:
---
bq. Because it is a good idea to permit the configuration to change between
nodes. E.g. if we want to do a gradual rollout of something
So subtle distinction here. I'm referring to a situation where we have the
aliased group of parameters in a {{system_distributed}} table, but we configure
_which_ of those we're using on a per-node basis. _In theory_ this could give
us a situation where users can centrally manage configuration groups and then
roll them out gradually.
Suppose it's only a marginal benefit since you have to touch local .yaml files
to flip the config anyway.
> Make sstable format configurable per table
> --
>
> Key: CASSANDRA-18534
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18534
> Project: Cassandra
> Issue Type: Improvement
> Components: Cluster/Schema, Local/SSTable
>Reporter: Branimir Lambov
>Assignee: Maxwell Guo
>Priority: Normal
> Fix For: 5.x
>
>
> Some SSTable format settings need to be configurable per table for better
> efficiency. This includes:
> - {{row_index_granularity}}
> - {{bloom_filter_fp_chance}}
> - {{crc_check_chance}}
> - {{min/max_index_interval}}
> Some of these are currently configurable using direct properties of tables.
> Having them as format properties makes better sense and should also support
> specifying useable combinations of settings, e.g.
> {code:java}
> CREATE TABLE ... WITH sstable_format = "bti-fast";
> CREATE TABLE ... WITH sstable_format = "bti-small";
> {code}
> where {{bti-fast}} and {{bti-small}} can be defined in {{cassandra.yaml}}
> e.g. as
> {code:java}
> sstable.format.options:
> - bti-fast:
> row_index_granularity: 1kiB
> bloom_filter_fp_chance: 0.01
> - bti-small:
> row_index_granularity: 32kiB
> bloom_filter_fp_chance: 0.1
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18534) Make sstable format configurable per table
[
https://issues.apache.org/jira/browse/CASSANDRA-18534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758068#comment-17758068
]
Josh McKenzie commented on CASSANDRA-18534:
---
bq. where bti-fast and bti-small can be defined in cassandra.yaml e.g. as
I believe we covered this on a previous ticket but I forgot. Is there a reason
we're relying primarily on the .yaml rather than a system_distributed table
accessible via vtables for aliased groups of sstable configuration params?
> Make sstable format configurable per table
> --
>
> Key: CASSANDRA-18534
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18534
> Project: Cassandra
> Issue Type: Improvement
> Components: Cluster/Schema, Local/SSTable
>Reporter: Branimir Lambov
>Assignee: Maxwell Guo
>Priority: Normal
> Fix For: 5.x
>
>
> Some SSTable format settings need to be configurable per table for better
> efficiency. This includes:
> - {{row_index_granularity}}
> - {{bloom_filter_fp_chance}}
> - {{crc_check_chance}}
> - {{min/max_index_interval}}
> Some of these are currently configurable using direct properties of tables.
> Having them as format properties makes better sense and should also support
> specifying useable combinations of settings, e.g.
> {code:java}
> CREATE TABLE ... WITH sstable_format = "bti-fast";
> CREATE TABLE ... WITH sstable_format = "bti-small";
> {code}
> where {{bti-fast}} and {{bti-small}} can be defined in {{cassandra.yaml}}
> e.g. as
> {code:java}
> sstable.format.options:
> - bti-fast:
> row_index_granularity: 1kiB
> bloom_filter_fp_chance: 0.01
> - bti-small:
> row_index_granularity: 32kiB
> bloom_filter_fp_chance: 0.1
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18731) Add declarative root CI structure
[
https://issues.apache.org/jira/browse/CASSANDRA-18731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751725#comment-17751725
]
Josh McKenzie commented on CASSANDRA-18731:
---
ping [~edimitrova], [~adelapena], [~mck], and [~dcapwell]. I know all of you
have done quite a bit of work in the CI space and probably have great insight /
opinions to share here. Any of you have cycles for review / collab on this?
> Add declarative root CI structure
> -
>
> Key: CASSANDRA-18731
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18731
> Project: Cassandra
> Issue Type: Improvement
> Components: CI
>Reporter: Josh McKenzie
>Assignee: Josh McKenzie
>Priority: Normal
> Fix For: 5.x
>
>
> Currently we have a somewhat declarative structure in .circleci, however
> there's still quite a bit that's baked in (resource limitations, parallelism,
> which suites qualify for which pipelines (pre-commit vs. post-commit vs.
> ???), etc). Further, while CASSANDRA-18133 brings the build scripts in-tree,
> all these parameters (pipelines, env vars, job definitions, resource
> constraints) are all still scattered throughout the shell scripts and/or
> reliant on the {{JenkinsFile}} to determine what suites comprise what
> pipelines.
> This ticket aims to decouple the definition of pipelines and jobs for CI from
> the implementations themselves. The goal here is to define, establish, and
> test both the base config and some helper methods to provide _other_
> configurations (circle, ASF CI, etc) the tools they need to programmatically
> inherit base CI config from a purely declarative structure.
> Follow-up tickets will involve rewriting the in-tree build scripts and
> JenkinsFile generation to rely on this structure, as well as integrating the
> config parsing unit tests into our CI.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18731) Add declarative root CI structure
[
https://issues.apache.org/jira/browse/CASSANDRA-18731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie updated CASSANDRA-18731:
--
Test and Documentation Plan: New tests provided in \{.build/config/test} to
be run manually for this review. Will need to integrate into CI in the near
future after changes here are stabilized.
Status: Patch Available (was: Open)
> Add declarative root CI structure
> -
>
> Key: CASSANDRA-18731
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18731
> Project: Cassandra
> Issue Type: Improvement
> Components: CI
>Reporter: Josh McKenzie
>Assignee: Josh McKenzie
>Priority: Normal
> Fix For: 5.x
>
>
> Currently we have a somewhat declarative structure in .circleci, however
> there's still quite a bit that's baked in (resource limitations, parallelism,
> which suites qualify for which pipelines (pre-commit vs. post-commit vs.
> ???), etc). Further, while CASSANDRA-18133 brings the build scripts in-tree,
> all these parameters (pipelines, env vars, job definitions, resource
> constraints) are all still scattered throughout the shell scripts and/or
> reliant on the {{JenkinsFile}} to determine what suites comprise what
> pipelines.
> This ticket aims to decouple the definition of pipelines and jobs for CI from
> the implementations themselves. The goal here is to define, establish, and
> test both the base config and some helper methods to provide _other_
> configurations (circle, ASF CI, etc) the tools they need to programmatically
> inherit base CI config from a purely declarative structure.
> Follow-up tickets will involve rewriting the in-tree build scripts and
> JenkinsFile generation to rely on this structure, as well as integrating the
> config parsing unit tests into our CI.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18731) Add declarative root CI structure
[
https://issues.apache.org/jira/browse/CASSANDRA-18731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie updated CASSANDRA-18731:
--
Change Category: Quality Assurance
Complexity: Normal
Fix Version/s: 5.x
Status: Open (was: Triage Needed)
> Add declarative root CI structure
> -
>
> Key: CASSANDRA-18731
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18731
> Project: Cassandra
> Issue Type: Improvement
> Components: CI
>Reporter: Josh McKenzie
>Assignee: Josh McKenzie
>Priority: Normal
> Fix For: 5.x
>
>
> Currently we have a somewhat declarative structure in .circleci, however
> there's still quite a bit that's baked in (resource limitations, parallelism,
> which suites qualify for which pipelines (pre-commit vs. post-commit vs.
> ???), etc). Further, while CASSANDRA-18133 brings the build scripts in-tree,
> all these parameters (pipelines, env vars, job definitions, resource
> constraints) are all still scattered throughout the shell scripts and/or
> reliant on the {{JenkinsFile}} to determine what suites comprise what
> pipelines.
> This ticket aims to decouple the definition of pipelines and jobs for CI from
> the implementations themselves. The goal here is to define, establish, and
> test both the base config and some helper methods to provide _other_
> configurations (circle, ASF CI, etc) the tools they need to programmatically
> inherit base CI config from a purely declarative structure.
> Follow-up tickets will involve rewriting the in-tree build scripts and
> JenkinsFile generation to rely on this structure, as well as integrating the
> config parsing unit tests into our CI.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18731) Add declarative root CI structure
[
https://issues.apache.org/jira/browse/CASSANDRA-18731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751716#comment-17751716
]
Josh McKenzie commented on CASSANDRA-18731:
---
PR available [here.|https://github.com/apache/cassandra/pull/2554]
There's implications to what we do in CASSANDRA-18594 and other CI work from
this here; in my opinion our CI structure should be based on a decoupled root
like this so new job additions and env changes can be done in one place and
these changes cascade down to all potential CI implementations in the ecosystem
(ASF CI w/JenkinsFile, .circleci, etc).
> Add declarative root CI structure
> -
>
> Key: CASSANDRA-18731
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18731
> Project: Cassandra
> Issue Type: Improvement
> Components: CI
>Reporter: Josh McKenzie
>Assignee: Josh McKenzie
>Priority: Normal
>
> Currently we have a somewhat declarative structure in .circleci, however
> there's still quite a bit that's baked in (resource limitations, parallelism,
> which suites qualify for which pipelines (pre-commit vs. post-commit vs.
> ???), etc). Further, while CASSANDRA-18133 brings the build scripts in-tree,
> all these parameters (pipelines, env vars, job definitions, resource
> constraints) are all still scattered throughout the shell scripts and/or
> reliant on the {{JenkinsFile}} to determine what suites comprise what
> pipelines.
> This ticket aims to decouple the definition of pipelines and jobs for CI from
> the implementations themselves. The goal here is to define, establish, and
> test both the base config and some helper methods to provide _other_
> configurations (circle, ASF CI, etc) the tools they need to programmatically
> inherit base CI config from a purely declarative structure.
> Follow-up tickets will involve rewriting the in-tree build scripts and
> JenkinsFile generation to rely on this structure, as well as integrating the
> config parsing unit tests into our CI.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-18731) Add declarative root CI structure
Josh McKenzie created CASSANDRA-18731:
-
Summary: Add declarative root CI structure
Key: CASSANDRA-18731
URL: https://issues.apache.org/jira/browse/CASSANDRA-18731
Project: Cassandra
Issue Type: Improvement
Components: CI
Reporter: Josh McKenzie
Assignee: Josh McKenzie
Currently we have a somewhat declarative structure in .circleci, however
there's still quite a bit that's baked in (resource limitations, parallelism,
which suites qualify for which pipelines (pre-commit vs. post-commit vs. ???),
etc). Further, while CASSANDRA-18133 brings the build scripts in-tree, all
these parameters (pipelines, env vars, job definitions, resource constraints)
are all still scattered throughout the shell scripts and/or reliant on the
{{JenkinsFile}} to determine what suites comprise what pipelines.
This ticket aims to decouple the definition of pipelines and jobs for CI from
the implementations themselves. The goal here is to define, establish, and test
both the base config and some helper methods to provide _other_ configurations
(circle, ASF CI, etc) the tools they need to programmatically inherit base CI
config from a purely declarative structure.
Follow-up tickets will involve rewriting the in-tree build scripts and
JenkinsFile generation to rely on this structure, as well as integrating the
config parsing unit tests into our CI.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-18627) Tidy up vestigial code left over after CASSANDRA-17931
[
https://issues.apache.org/jira/browse/CASSANDRA-18627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737322#comment-17737322
]
Josh McKenzie edited comment on CASSANDRA-18627 at 6/26/23 9:24 PM:
-Hm. To anyone wandering past and curious, I'm not convinced this actually
_fixes_ anything since the only call sites in the {{PendingRepairManager}}
actually don't pass in any additional sstables or bytes (just 0), so the
compaction strategies don't have any estimated extra sstables or bytes to
calculate anything from. I'll need to dig a bit into where those values make
their way into this scope to then be passed on to compaction to determine where
and why they're vestigial in this way.-
Restructured ticket to clean this code up since it's clearly confusing. :)
was (Author: jmckenzie):
Hm. To anyone wandering past and curious, I'm not convinced this actually
_fixes_ anything since the only call sites in the {{PendingRepairManager}}
actually don't pass in any additional sstables or bytes (just 0), so the
compaction strategies don't have any estimated extra sstables or bytes to
calculate anything from. I'll need to dig a bit into where those values make
their way into this scope to then be passed on to compaction to determine where
and why they're vestigial in this way.
> Tidy up vestigial code left over after CASSANDRA-17931
> --
>
> Key: CASSANDRA-18627
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18627
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Repair, Local/Compaction
>Reporter: Josh McKenzie
>Assignee: Josh McKenzie
>Priority: Normal
> Fix For: 5.x
>
>
> In CASSANDRA-17931, some code was left over (\{{additionalBytes}},
> {{{}additionalSSTables{}}}) inside the {{PendingRepairManager}} which, while
> not explicitly hurting anything, provides for some confusion around when we
> do or don't take additional values into account when adjusting for pending
> operations when deciding what to do with upcoming repairs etc.
>
> We should tidy that up and consider making the role of those methods / their
> structure more clear between the \{{PendingRepairManager]} hierarchy and the
> {{AbstractCompactionStrategy}} hierarchy to prevent further confusion.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] (CASSANDRA-18627) Tidy up vestigial code left over after CASSANDRA-17931
[ https://issues.apache.org/jira/browse/CASSANDRA-18627 ]
Josh McKenzie deleted comment on CASSANDRA-18627:
---
was (Author: jmckenzie):
||Item|Link||
|PR|[link|https://github.com/apache/cassandra/pull/2450]|
|JDK8
CI|[link|https://app.circleci.com/pipelines/github/jmckenzie-dev/cassandra/337/workflows/4a26c29b-dca6-4b14-bc7c-8b7c9977a977]|
|JDK11
CI|[link|https://app.circleci.com/pipelines/github/jmckenzie-dev/cassandra/337/workflows/2eb8052f-4f63-4099-a96c-e19fafe6634b]|
> Tidy up vestigial code left over after CASSANDRA-17931
> --
>
> Key: CASSANDRA-18627
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18627
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Repair, Local/Compaction
>Reporter: Josh McKenzie
>Assignee: Josh McKenzie
>Priority: Normal
> Fix For: 5.x
>
>
> In CASSANDRA-17931, some code was left over (\{{additionalBytes}},
> {{{}additionalSSTables{}}}) inside the {{PendingRepairManager}} which, while
> not explicitly hurting anything, provides for some confusion around when we
> do or don't take additional values into account when adjusting for pending
> operations when deciding what to do with upcoming repairs etc.
>
> We should tidy that up and consider making the role of those methods / their
> structure more clear between the \{{PendingRepairManager]} hierarchy and the
> {{AbstractCompactionStrategy}} hierarchy to prevent further confusion.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18627) Tidy up vestigial code left over after CASSANDRA-17931
[
https://issues.apache.org/jira/browse/CASSANDRA-18627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie updated CASSANDRA-18627:
--
Description:
In CASSANDRA-17931, some code was left over (\{{additionalBytes}},
{{{}additionalSSTables{}}}) inside the {{PendingRepairManager}} which, while
not explicitly hurting anything, provides for some confusion around when we do
or don't take additional values into account when adjusting for pending
operations when deciding what to do with upcoming repairs etc.
We should tidy that up and consider making the role of those methods / their
structure more clear between the \{{PendingRepairManager]} hierarchy and the
{{AbstractCompactionStrategy}} hierarchy to prevent further confusion.
was:
In CASSANDRA-17931, we failed to correctly take into account the
additionalSSTables and additionalBytes passed in in one of our calls to
{code}PendingRepairManager.getEstimatedRemainingTasks{code}
> Tidy up vestigial code left over after CASSANDRA-17931
> --
>
> Key: CASSANDRA-18627
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18627
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Repair, Local/Compaction
>Reporter: Josh McKenzie
>Assignee: Josh McKenzie
>Priority: Normal
> Fix For: 5.x
>
>
> In CASSANDRA-17931, some code was left over (\{{additionalBytes}},
> {{{}additionalSSTables{}}}) inside the {{PendingRepairManager}} which, while
> not explicitly hurting anything, provides for some confusion around when we
> do or don't take additional values into account when adjusting for pending
> operations when deciding what to do with upcoming repairs etc.
>
> We should tidy that up and consider making the role of those methods / their
> structure more clear between the \{{PendingRepairManager]} hierarchy and the
> {{AbstractCompactionStrategy}} hierarchy to prevent further confusion.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18627) Tidy up vestigial code left over after CASSANDRA-17931
[
https://issues.apache.org/jira/browse/CASSANDRA-18627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie updated CASSANDRA-18627:
--
Summary: Tidy up vestigial code left over after CASSANDRA-17931 (was: Fix
calculation of pending compactions in pending repair manager)
> Tidy up vestigial code left over after CASSANDRA-17931
> --
>
> Key: CASSANDRA-18627
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18627
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Repair, Local/Compaction
>Reporter: Josh McKenzie
>Assignee: Josh McKenzie
>Priority: Normal
> Fix For: 5.x
>
>
> In CASSANDRA-17931, we failed to correctly take into account the
> additionalSSTables and additionalBytes passed in in one of our calls to
> {code}PendingRepairManager.getEstimatedRemainingTasks{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18627) Fix calculation of pending compactions in pending repair manager
[
https://issues.apache.org/jira/browse/CASSANDRA-18627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737322#comment-17737322
]
Josh McKenzie commented on CASSANDRA-18627:
---
Hm. To anyone wandering past and curious, I'm not convinced this actually
_fixes_ anything since the only call sites in the {{PendingRepairManager}}
actually don't pass in any additional sstables or bytes (just 0), so the
compaction strategies don't have any estimated extra sstables or bytes to
calculate anything from. I'll need to dig a bit into where those values make
their way into this scope to then be passed on to compaction to determine where
and why they're vestigial in this way.
> Fix calculation of pending compactions in pending repair manager
>
>
> Key: CASSANDRA-18627
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18627
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Repair, Local/Compaction
>Reporter: Josh McKenzie
>Assignee: Josh McKenzie
>Priority: Normal
> Fix For: 5.x
>
>
> In CASSANDRA-17931, we failed to correctly take into account the
> additionalSSTables and additionalBytes passed in in one of our calls to
> {code}PendingRepairManager.getEstimatedRemainingTasks{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18627) Fix calculation of pending compactions in pending repair manager
[
https://issues.apache.org/jira/browse/CASSANDRA-18627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie updated CASSANDRA-18627:
--
Status: Open (was: Patch Available)
> Fix calculation of pending compactions in pending repair manager
>
>
> Key: CASSANDRA-18627
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18627
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Repair, Local/Compaction
>Reporter: Josh McKenzie
>Assignee: Josh McKenzie
>Priority: Normal
> Fix For: 5.x
>
>
> In CASSANDRA-17931, we failed to correctly take into account the
> additionalSSTables and additionalBytes passed in in one of our calls to
> {code}PendingRepairManager.getEstimatedRemainingTasks{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18627) Fix calculation of pending compactions in pending repair manager
[
https://issues.apache.org/jira/browse/CASSANDRA-18627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737310#comment-17737310
]
Josh McKenzie commented on CASSANDRA-18627:
---
||Item|Link||
|PR|[link|https://github.com/apache/cassandra/pull/2450]|
|JDK8
CI|[link|https://app.circleci.com/pipelines/github/jmckenzie-dev/cassandra/337/workflows/4a26c29b-dca6-4b14-bc7c-8b7c9977a977]|
|JDK11
CI|[link|https://app.circleci.com/pipelines/github/jmckenzie-dev/cassandra/337/workflows/2eb8052f-4f63-4099-a96c-e19fafe6634b]|
> Fix calculation of pending compactions in pending repair manager
>
>
> Key: CASSANDRA-18627
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18627
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Repair, Local/Compaction
>Reporter: Josh McKenzie
>Assignee: Josh McKenzie
>Priority: Normal
>
> In CASSANDRA-17931, we failed to correctly take into account the
> additionalSSTables and additionalBytes passed in in one of our calls to
> {code}PendingRepairManager.getEstimatedRemainingTasks{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18627) Fix calculation of pending compactions in pending repair manager
[
https://issues.apache.org/jira/browse/CASSANDRA-18627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie updated CASSANDRA-18627:
--
Bug Category: Parent values: Degradation(12984)
Complexity: Low Hanging Fruit
Discovered By: User Report
Fix Version/s: 5.x
Severity: Normal
Status: Open (was: Triage Needed)
> Fix calculation of pending compactions in pending repair manager
>
>
> Key: CASSANDRA-18627
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18627
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Repair, Local/Compaction
>Reporter: Josh McKenzie
>Assignee: Josh McKenzie
>Priority: Normal
> Fix For: 5.x
>
>
> In CASSANDRA-17931, we failed to correctly take into account the
> additionalSSTables and additionalBytes passed in in one of our calls to
> {code}PendingRepairManager.getEstimatedRemainingTasks{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18627) Fix calculation of pending compactions in pending repair manager
[
https://issues.apache.org/jira/browse/CASSANDRA-18627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie updated CASSANDRA-18627:
--
Test and Documentation Plan: No new testing; confirm existing tests pass.
Status: Patch Available (was: Open)
> Fix calculation of pending compactions in pending repair manager
>
>
> Key: CASSANDRA-18627
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18627
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Repair, Local/Compaction
>Reporter: Josh McKenzie
>Assignee: Josh McKenzie
>Priority: Normal
> Fix For: 5.x
>
>
> In CASSANDRA-17931, we failed to correctly take into account the
> additionalSSTables and additionalBytes passed in in one of our calls to
> {code}PendingRepairManager.getEstimatedRemainingTasks{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-18627) Fix calculation of pending compactions in pending repair manager
Josh McKenzie created CASSANDRA-18627:
-
Summary: Fix calculation of pending compactions in pending repair
manager
Key: CASSANDRA-18627
URL: https://issues.apache.org/jira/browse/CASSANDRA-18627
Project: Cassandra
Issue Type: Bug
Components: Consistency/Repair, Local/Compaction
Reporter: Josh McKenzie
Assignee: Josh McKenzie
In CASSANDRA-17931, we failed to correctly take into account the
additionalSSTables and additionalBytes passed in in one of our calls to
{code}PendingRepairManager.getEstimatedRemainingTasks{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18599) Cassandra Analytics - Upgrade to use JUnit 5
[ https://issues.apache.org/jira/browse/CASSANDRA-18599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17732645#comment-17732645 ] Josh McKenzie commented on CASSANDRA-18599: --- Not to be pedantic (which is always how the best pedantry starts), but does that same bar hold for _upgrading_ an existing library vs. adding? We're pretty slow on the draw upgrading deps and also pretty unstructured. Might be worth revisiting and thinking about in general. > Cassandra Analytics - Upgrade to use JUnit 5 > > > Key: CASSANDRA-18599 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18599 > Project: Cassandra > Issue Type: Improvement >Reporter: Doug Rohrer >Priority: Normal > > For future work, we’d like to use some features only available in JUnit 5 > (TestTemplates being the immediate need). > Upgrade cassandra-analytics dependencies to JUnit 5 and fix any test issues. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11745) Add bytes limit to queries and paging
[ https://issues.apache.org/jira/browse/CASSANDRA-11745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17730103#comment-17730103 ] Josh McKenzie commented on CASSANDRA-11745: --- Still valid; I'll take a look by end of week barring any catastrophe. > Add bytes limit to queries and paging > - > > Key: CASSANDRA-11745 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11745 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Core >Reporter: Richard Low >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 5.0 > > Time Spent: 10m > Remaining Estimate: 0h > > For some data models, values may be of very different sizes. When querying > data, limit by count doesn’t work well and leads to timeouts. It would be > much better to limit by size of the response, probably by stopping at the > first row that goes above the limit. This applies to paging too so you can > safely page through such data without timeout worries. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18567) Add jobs for resource-intensive Python upgrade tests
[ https://issues.apache.org/jira/browse/CASSANDRA-18567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729743#comment-17729743 ] Josh McKenzie commented on CASSANDRA-18567: --- bq. create new jobs dedicated to the resource-intensive upgrade tests Do you know off the top of your head if / when we run these in an automated fashion? Is that per-commit, or on a particular cadence, or before releases? (ping [~mck]) > Add jobs for resource-intensive Python upgrade tests > > > Key: CASSANDRA-18567 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18567 > Project: Cassandra > Issue Type: Task > Components: CI >Reporter: Ekaterina Dimitrova >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.0.x, 4.1.x, 5.x > > > As it was noticed in CASSANDRA-18499, Python upgrade tests should include > resource-intensive ones. Some are run in CircleCI but skipped in Jenkins (see > CASSANDRA-18499 for reference). > We should add --skip-resource-intensive-tests to the current Python upgrade > tests jobs and create new jobs dedicated to the resource-intensive upgrade > tests — the way we do it for the regular Python Dtests(for reference - large > DTests). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18133) In-tree build scripts
[
https://issues.apache.org/jira/browse/CASSANDRA-18133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729742#comment-17729742
]
Josh McKenzie commented on CASSANDRA-18133:
---
{quote}I'm going to split the Jenkinsfile update out to a separate ticket. It
is proving difficult to test, too many moving pieces.
{quote}
+1 to that change from me.
> In-tree build scripts
> -
>
> Key: CASSANDRA-18133
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18133
> Project: Cassandra
> Issue Type: Task
> Components: CI
>Reporter: Michael Semb Wever
>Assignee: Michael Semb Wever
>Priority: Normal
> Fix For: 5.x
>
>
> Bring the artifact/deb/rpm build scripts (and associated docker images) from
> cassandra-builds repo to the .build directory.
> The declarative Jenkinsfile can then directly declare the artifacts jobs in
> its pipeline. And the packaging jobs can be separated and run in parallel.
> This addresses the epic's stated existing problems:
> - difficult to pre-commit test jenkins and cassandra-build changes,
> - CI development efforts is split between ci-cassandra and circleci, despite
> ci-cassandra being our canonical and non-commercial CI,
> - lacking parity of what is tested between ci-cassandra and circleci
> - cassandra-builds as a separate repo (without release branches matching
> in-tree) adds complexity to changing matrix values (jdks, pythons, dist)
> - mixture of jenkins dsl groovy, declarative and scripting pipeline.
> - different pre-commit and post-commit jenkins pipelines are used.
> In addition it addresses:
> - stage jobs don't always running on the same SHA as the pipeline's run,
> - infra issues around networking, specifically git cloning additional
> cassandra-builds repository,
> - a more readable Jenkinsfile
> - more UX friendly build and test scripts
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18119) Handle sstable metadata stats file getting a new mtime after compaction has finished
[
https://issues.apache.org/jira/browse/CASSANDRA-18119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729728#comment-17729728
]
Josh McKenzie commented on CASSANDRA-18119:
---
Just noticed this in my queue [~marcuse]. w/a rebase is this ready to merge?
> Handle sstable metadata stats file getting a new mtime after compaction has
> finished
>
>
> Key: CASSANDRA-18119
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18119
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction, Local/Startup and Shutdown
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 3.11.x, 4.0.x, 4.1.x, 5.x
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Due to a race between compaction finishing and compaction strategies getting
> reloaded there is a chance that we try to add both the new sstable and the
> old compacted sstable to the compaction strategy, and in the LCS case this
> can cause the old sstable to get sent to L0 to avoid overlap. This changes
> the mtime of the stats metadata file and if the node is shut down before the
> sstable is actually deleted from disk, we fail starting with the following
> exception:
> {code}
> .../mockcf1-392b3ff07c5a11ed8c662f5760cb10b3/nb_txn_compaction_3983c030-7c5a-11ed-8c66-2f5760cb10b3.log
> REMOVE:[.../data/TransactionLogsTest/mockcf1-392b3ff07c5a11ed8c662f5760cb10b3/nb-0-big-,1671096247000,5][4003386800]
> ***Unexpected files detected for sstable [nb-0-big-]: last update
> time [Thu Dec 15 10:24:09 CET 2022] (1671096249000) should have been [Thu Dec
> 15 10:24:07 CET 2022] (1671096247000)
> ADD:[.../data/TransactionLogsTest/mockcf1-392b3ff07c5a11ed8c662f5760cb10b3/nb-2-big-,0,5][319189529]
> {code}
> A workaround for this (until we properly fix the way compaction strategies
> get notified about sstable changes) is to ignore the timestamp of the STATS
> component when cleaning up compaction leftovers on startup.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16951) Dtest cluster reusage
[
https://issues.apache.org/jira/browse/CASSANDRA-16951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie updated CASSANDRA-16951:
--
Epic Link: CASSANDRA-17276
> Dtest cluster reusage
> -
>
> Key: CASSANDRA-16951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16951
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/dtest/python
>Reporter: Berenguer Blasi
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0.x
>
>
> Dtests are very heavy but in some instances most of the time is spent
> restarting nodes in between test methods. Not all of them, but many seem
> could benefit form reusing a common cluster sparing the restarts. Obviously
> that is not the case for tests that manipulate the nodes itself during the
> test. The ones that follow a setup node/do test seem to benefit greatly in
> terms of time execution.
> Some classes run time can be cut form 10m to 1,5m. Others only from 30m to
> 25m. But taking a 5m shave and considering it will probably get ran *
> with/without vnodes * j8/j11/j8j11 * 4.0/trunk turns the 5m cut into a 60m
> cut. That should be a nice reduction in CI usage. Unfortunately run time will
> mostly remain the same until we have a majority of tests reusing nodes as the
> 'longest pole' will be the determining factor.
> How it works? It's an opt-in. Annotate the first test with
> {{@reuse_cluster(new_cluster=True)}} and the following ones with
> {{@reuse_cluster}}. Best effort to reuse the cluster will be made. Stop using
> the annotation at any test method and it will start a new one.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15196) Unify CommitLogSegment allocation to one hierarchy and remove test-cdc testing target
[
https://issues.apache.org/jira/browse/CASSANDRA-15196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie updated CASSANDRA-15196:
--
Epic Link: CASSANDRA-17276
> Unify CommitLogSegment allocation to one hierarchy and remove test-cdc
> testing target
> -
>
> Key: CASSANDRA-15196
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15196
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Commit Log
>Reporter: Joshua McKenzie
>Assignee: Josh McKenzie
>Priority: Normal
> Fix For: 5.x
>
>
> {{April 2023 update}}
> With the push towards stabilizing ASF CI, one of the things we can do is
> remove the total runtime of our tests. CDC's segment allocation has been in
> the ecosystem for a long while now and has proven stable in the "CDC enabled
> on the node but not the table" case, so unifying the hierarchy to a single
> allocator and removing the test-cdc target in ant should both help us clean
> up some debt in the codebase and also streamline our CI processes. Along with
> that, we can take this chance to tidy up some of the internals of the
> implementation as previously mentioned on this ticket.
>
> {{ORIGINAL DESCRIPTION}}
> Back when I wrote CASSANDRA-8844 and CASSANDRA-12148, a few things about the
> CommitLog stood out to me that I wasn't in love with. At the time I was much
> more of the mind of "When in Rome..." regarding some of our structures, our
> naming, and specifically regarding our penchant for inheritance vs.
> composition. Turns out I have an itch here I need to scratch.
> This patch refactors the CommitLog in a few key ways:
> * Removes inheritence for CommitLogSegmentManagerX, instead has a single
> CommitLogSegmentManager and a CommitLogSegmentAllocator interface
> ** This interface is implemented by CommitLogSegmentAllocatorStandard and
> CommitLogSegmentAllocatorCDC
> * Renames a few variables and methods within the segment manager to make
> their purpose and role more clear from names alone (hopefully):
> ** allocatingFrom --> activeSegment
> ** allocatingFrom() --> getActiveSegment()
> ** advanceAllocatingFrom() – switchToNewSegment()
> ** activeSegments --> unflushedSegments
> ** getActiveSegments --> getUnflushedSegments()
> ** awaitAvailableSegment(...) --> awaitSegmentAllocation(...)
> ** isStillAllocating() --> hasRoom()
> * Reorders some of the "allocation-and-compare-in-while-loop-decl) to more
> idiomatic style
> * Adds some comments to various under-documented methods, mostly related to
> CDC and timing
> -As far as we are in the 4.0 testing, I don't expect this to hit pre 4.0.
> Given it's a relatively minor refactor in well-tested code, doesn't change
> functionality, and doesn't impact any of the publicly exposed APIs in the
> CommitLog ecosystem, I'd be comfortable with it going in a minor release
> after 4.0 is out. But I'm open to alternative viewpoints.-
> -Have run unit tests locally w/out issue on both regular and cdc-targets;
> need to get CI against the branch internally for full dtest run and I'll post
> when it's done. Just wanted to get visibility to this out there in case
> anyone was curious or thinking of working on the same thing.-
> [ Link to branch on
> github|https://github.com/apache/cassandra/compare/trunk...josh-mckenzie:clean_cl_comp]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-17276) Cassandra CI Improvements
[ https://issues.apache.org/jira/browse/CASSANDRA-17276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-17276: -- Summary: Cassandra CI Improvements (was: Cassandra CI Process Improvements) > Cassandra CI Improvements > - > > Key: CASSANDRA-17276 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17276 > Project: Cassandra > Issue Type: Epic >Reporter: Josh McKenzie >Assignee: Josh McKenzie >Priority: Normal > > Reference dev [ML > Thread|https://lists.apache.org/thread/bq470ml17g106pwxpvwgws2stxc6d7b9] > Reference cwiki ratified final process: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=199530280 > There are some outstanding infrastructural, documentation, and automation > pieces from the agreed upon CI state we need to implement. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-17276) Cassandra CI Improvements
[ https://issues.apache.org/jira/browse/CASSANDRA-17276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-17276: -- Epic Name: CIImprovements (was: CIProcessImprovements) > Cassandra CI Improvements > - > > Key: CASSANDRA-17276 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17276 > Project: Cassandra > Issue Type: Epic >Reporter: Josh McKenzie >Assignee: Josh McKenzie >Priority: Normal > > Reference dev [ML > Thread|https://lists.apache.org/thread/bq470ml17g106pwxpvwgws2stxc6d7b9] > Reference cwiki ratified final process: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=199530280 > There are some outstanding infrastructural, documentation, and automation > pieces from the agreed upon CI state we need to implement. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-11745) Add bytes limit to queries and paging
[ https://issues.apache.org/jira/browse/CASSANDRA-11745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-11745: -- Reviewers: Josh McKenzie > Add bytes limit to queries and paging > - > > Key: CASSANDRA-11745 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11745 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Core >Reporter: Richard Low >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 5.0 > > > For some data models, values may be of very different sizes. When querying > data, limit by count doesn’t work well and leads to timeouts. It would be > much better to limit by size of the response, probably by stopping at the > first row that goes above the limit. This applies to paging too so you can > safely page through such data without timeout worries. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11745) Add bytes limit to queries and paging
[ https://issues.apache.org/jira/browse/CASSANDRA-11745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17726771#comment-17726771 ] Josh McKenzie commented on CASSANDRA-11745: --- [~jlewandowski] - you need a reviewer on this one once it's stabilized a bit? I've been swimming in these waters for the last couple of months so my context is very fresh. > Add bytes limit to queries and paging > - > > Key: CASSANDRA-11745 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11745 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Core >Reporter: Richard Low >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 5.0 > > > For some data models, values may be of very different sizes. When querying > data, limit by count doesn’t work well and leads to timeouts. It would be > much better to limit by size of the response, probably by stopping at the > first row that goes above the limit. This applies to paging too so you can > safely page through such data without timeout worries. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18424) Implement graceful paging across tombstones with short-circuit on paging rather than throwing TombstoneOverwhelmingExceptions
[
https://issues.apache.org/jira/browse/CASSANDRA-18424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17726350#comment-17726350
]
Josh McKenzie commented on CASSANDRA-18424:
---
Yeah - I took a look at the patch on the DS fork and it looks like in the "by
bytes" case you had the luxury of relying on the CQLLimits and the machinery
already expecting early termination of paging based on some kind of successful
"I got as much as I asked for" logic in the {{BaseRowIterator}} / Row context.
The implementation I'm currently validating now for paging across tombstones
has the _rather painful_ property of needing to figure out a way to gracefully
signal up from a stopped transformation on an {{Unfiltered}} up to the
{{AbstractQueryPager}} (or be polled by it which is the route I went) to check
and see if any of the {{ReadCommand}}'s it owned stopped due to tombstone
limits in order to determine whether to adjust the clustering key and
exhaustion state we're sending back to the client or not. The fact that we've
spent a decade relying on "if something bad related to tombstones happens in
the bowels of your unfiltered iteration, you will explode" made this... tricky.
Intersection with {{RTBoundCloser}} is also a bit tricky; think I have a
solution around that as well.
I've chatted with [~jlewandowski] a bit about the topic; will circle back once
I have a patch to attach to the ticket here.
> Implement graceful paging across tombstones with short-circuit on paging
> rather than throwing TombstoneOverwhelmingExceptions
> -
>
> Key: CASSANDRA-18424
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18424
> Project: Cassandra
> Issue Type: New Feature
> Components: Messaging/Client, Messaging/Internode
>Reporter: Josh McKenzie
>Assignee: Josh McKenzie
>Priority: Normal
>
> We implemented the hard stop with a {{TombstoneOverwhelmingException}} almost
> a decade ago since paging across many tombstones was the most common way for
> nodes to OOM as they iterated across all this data during queries and paging.
> With our current implementations and architecture / codebase, we should be
> able to combine the {{StoppingTransformation}} and existing {{clustering}}
> blob we pass back to clients to allow clients to optionally page across
> tombstones when using the async api via the driver and short-circuit a page
> when they hit the tombstone failure threshold rather than throwing a
> {{{}TombstoneOverwhelmingException{}}}. This would allow for more flexible
> data modeling on users' side as well as removing one of the fairly rough
> edges of our API's we're currently constrained by.
> Making sure this is correct will require extensive fuzz-testing of
> pagination; this should likely happen in the Harry project but we could also
> have a bespoke model / implementation in the C* codebase we rely on in the
> interim.
> Client warnings at the current default levels would remain; the gap between
> warn and "short-circuit pages" (100x ratio currently, 1000 vs. 10) should
> be sufficient for clients to take action on their data models well before
> they hit this limit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18084) Introduce tags to snitch for better decision making for replica placement in topology strategies
[
https://issues.apache.org/jira/browse/CASSANDRA-18084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724458#comment-17724458
]
Josh McKenzie commented on CASSANDRA-18084:
---
Missed the ping earlier Stefan; I'm up to my eyeballs in various things right
now. Any chance you have cycles [~paulo] ?
> Introduce tags to snitch for better decision making for replica placement in
> topology strategies
>
>
> Key: CASSANDRA-18084
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18084
> Project: Cassandra
> Issue Type: New Feature
> Components: Cluster/Gossip, Cluster/Schema, Legacy/Distributed
> Metadata
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We would like to have extra meta-information in cassandra-rackdc.properties
> which would further differentiate nodes in dc / racks.
> The main motivation behind this is that we have special requirements around
> node's characteristics based on which we want to make further decisions when
> it comes to replica placement in topology strategies. (New topology strategy
> would be mostly derived from NTS / would be extended)
> The most reasonable way to do that is to introduce a new property into
> cassandra-rackdc.properties called "tags"
> {code:java}
> # Arbitrary tag to assign to this node, they serve as additional
> identificator of a node based on which operators might act.
> # Value of this property is meant to be a comma-separated list of strings.
> #tags=tag1,tag2
> {code}
> We also want to introduce new application state called TAGS. On startup of a
> node, this node would advertise its tags to cluster and vice versa, all nodes
> would tell to that respective node what tags they have so everybody would see
> the same state of tags across the cluster based on which topology strategies
> would do same decisions.
> These tags are not meant to be changed during whole runtime of a node,
> similarly as datacenter and rack is not.
> For performance reasons, we might limit the maximum size of tags (sum of
> their lenght), to be, for example, 64 characters and anything bigger would be
> either shortened or the start would fail.
> Once we have tags for all nodes, we can have access to them, cluster-wide,
> from TokenMetadata which is used quite heavily in topology strategies and it
> exposes other relevant topology information (dc's, racks ...). We would just
> add a new way to look at nodes.
> Tags would be a set.
> This would be persisted to the system.local to see what tags a local node has
> and it would be persisted to system.peers_v2 to see what tags all other nodes
> have. Column would be called "tags".
> {code:java}
> admin@cqlsh> select * from system.local ;
> @ Row 1
> key | local
> bootstrapped| COMPLETED
> broadcast_address | 172.19.0.5
> broadcast_port | 7000
> cluster_name| Test Cluster
> cql_version | 3.4.6
> data_center | dc1
> gossip_generation | 1669739177
> host_id | 54f8c6ea-a6ba-40c5-8fa5-484b2b4184c9
> listen_address | 172.19.0.5
> listen_port | 7000
> native_protocol_version | 5
> partitioner | org.apache.cassandra.dht.Murmur3Partitioner
> rack| rack1
> release_version | 4.2-SNAPSHOT
> rpc_address | 172.19.0.5
> rpc_port| 9042
> schema_version | ef865449-2491-33b8-95b0-47c09cb14ea9
> tags| {'tag1', 'tag2'}
> tokens | {'6504358681601109713'}
> {code}
> for system.peers_v2:
> {code:java}
> admin@cqlsh> select peer,tags from system.peers_v2 ;
> @ Row 1
> --+-
> peer | 172.19.0.15
> tags | {'tag2', 'tag3'}
> @ Row 2
> --+-
> peer | 172.19.0.11
> tags | null
> {code}
> the POC implementation doing exactly that is here:
> We do not want to provide our custom topology strategies which are using this
> feature as that will be the most probably a proprietary solution. This might
> indeed change in the future. For now, we just want to implement hooks we can
> base our in-house implementation on. All other people can benefit from this
> as well if they choose so as this feature enables them to do that.
> Adding tags is not only about custom topology strategies. Operators could tag
> their nodes if they wish to make further distinctions on topology level for
> their operational needs.
> [https://github.com/instaclustr/cassandra/commit/eddd4681d8678515454dabb926d5f56b0c225eea]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (CASSANDRA-18464) Enable Direct I/O For CommitLog Files
[ https://issues.apache.org/jira/browse/CASSANDRA-18464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie reassigned CASSANDRA-18464: - Assignee: Amit Pawar > Enable Direct I/O For CommitLog Files > - > > Key: CASSANDRA-18464 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18464 > Project: Cassandra > Issue Type: New Feature > Components: Local/Commit Log >Reporter: Josh McKenzie >Assignee: Amit Pawar >Priority: Normal > Fix For: 6.x > > Attachments: UseDirectIOFeatureForCommitLogFiles.patch > > > Relocating from [dev@ email > thread.|https://lists.apache.org/thread/j6ny17q2rhkp7jxvwxm69dd6v1dozjrg] > > I shared my investigation about Commitlog I/O issue on large core count > system in my previous email dated July-22 and link to the thread is given > below. > [https://lists.apache.org/thread/xc5ocog2qz2v2gnj4xlw5hbthfqytx2n] > Basically, two solutions looked possible to improve the CommitLog I/O. > # Multi-threaded syncing > # Using Direct-IO through JNA > I worked on 2nd option considering the following benefit compared to the > first one > # Direct I/O read/write throughput is very high compared to non-Direct I/O. > Learnt through FIO benchmarking. > # Reduces kernel file cache uses which in-turn reduces kernel I/O activity > for Commitlog files only. > # Overall CPU usage reduced for flush activity. JVisualvm shows CPU usage < > 30% for Commitlog syncer thread with Direct I/O feature > # Direct I/O implementation is easier compared to multi-threaded > As per the community suggestion, less in code complex is good to have. Direct > I/O enablement looked promising but there was one issue. > Java version 8 does not have native support to enable Direct I/O. So, JNA > library usage is must. The same implementation should also work across other > versions of Java (like 11 and beyond). > I have completed Direct I/O implementation and summary of the attached patch > changes are given below. > # This implementation is not using Java file channels and file is opened > through JNA to use Direct I/O feature. > # New Segment are defined named “DirectIOSegment” for Direct I/O and > “NonDirectIOSegment” for non-direct I/O (NonDirectIOSegment is test purpose > only). > # JNA write call is used to flush the changes. > # New helper functions are defined in NativeLibrary.java and platform > specific file. Currently tested on Linux only. > # Patch allows user to configure optimum block size and alignment if > default values are not OK for CommitLog disk. > # Following configuration options are provided in Cassandra.yaml file > a. use_jna_for_commitlog_io : to use jna feature > b. use_direct_io_for_commitlog : to use Direct I/O feature. > c. direct_io_minimum_block_alignment: 512 (default) > d. nvme_disk_block_size: 32MiB (default and can be changed as per the > required size) > Test matrix is complex so CommitLog related testcases and TPCx-IOT benchmark > was tested. It works with both Java 8 and 11 versions. Compressed and > Encrypted based segments are not supported yet and it can be enabled later > based on the Community feedback. > Following improvement are seen with Direct I/O enablement. > # 32 cores >= ~15% > # 64 cores >= ~80% > Also, another observation would like to share here. Reading Commitlog files > with Direct I/O might help in reducing node bring-up time after the node > crash. > Tested with commit ID: 91f6a9aca8d3c22a03e68aa901a0b154d960ab07 > The attached patch enables Direct I/O feature for Commitlog files. Please > check and share your feedback. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-18464) Enable Direct I/O For CommitLog Files
Josh McKenzie created CASSANDRA-18464: - Summary: Enable Direct I/O For CommitLog Files Key: CASSANDRA-18464 URL: https://issues.apache.org/jira/browse/CASSANDRA-18464 Project: Cassandra Issue Type: New Feature Components: Local/Commit Log Reporter: Josh McKenzie Assignee: Amit Pawar Attachments: UseDirectIOFeatureForCommitLogFiles.patch Relocating from [dev@ email thread.|https://lists.apache.org/thread/j6ny17q2rhkp7jxvwxm69dd6v1dozjrg] I shared my investigation about Commitlog I/O issue on large core count system in my previous email dated July-22 and link to the thread is given below. [https://lists.apache.org/thread/xc5ocog2qz2v2gnj4xlw5hbthfqytx2n] Basically, two solutions looked possible to improve the CommitLog I/O. # Multi-threaded syncing # Using Direct-IO through JNA I worked on 2nd option considering the following benefit compared to the first one # Direct I/O read/write throughput is very high compared to non-Direct I/O. Learnt through FIO benchmarking. # Reduces kernel file cache uses which in-turn reduces kernel I/O activity for Commitlog files only. # Overall CPU usage reduced for flush activity. JVisualvm shows CPU usage < 30% for Commitlog syncer thread with Direct I/O feature # Direct I/O implementation is easier compared to multi-threaded As per the community suggestion, less in code complex is good to have. Direct I/O enablement looked promising but there was one issue. Java version 8 does not have native support to enable Direct I/O. So, JNA library usage is must. The same implementation should also work across other versions of Java (like 11 and beyond). I have completed Direct I/O implementation and summary of the attached patch changes are given below. # This implementation is not using Java file channels and file is opened through JNA to use Direct I/O feature. # New Segment are defined named “DirectIOSegment” for Direct I/O and “NonDirectIOSegment” for non-direct I/O (NonDirectIOSegment is test purpose only). # JNA write call is used to flush the changes. # New helper functions are defined in NativeLibrary.java and platform specific file. Currently tested on Linux only. # Patch allows user to configure optimum block size and alignment if default values are not OK for CommitLog disk. # Following configuration options are provided in Cassandra.yaml file a. use_jna_for_commitlog_io : to use jna feature b. use_direct_io_for_commitlog : to use Direct I/O feature. c. direct_io_minimum_block_alignment: 512 (default) d. nvme_disk_block_size: 32MiB (default and can be changed as per the required size) Test matrix is complex so CommitLog related testcases and TPCx-IOT benchmark was tested. It works with both Java 8 and 11 versions. Compressed and Encrypted based segments are not supported yet and it can be enabled later based on the Community feedback. Following improvement are seen with Direct I/O enablement. # 32 cores >= ~15% # 64 cores >= ~80% Also, another observation would like to share here. Reading Commitlog files with Direct I/O might help in reducing node bring-up time after the node crash. Tested with commit ID: 91f6a9aca8d3c22a03e68aa901a0b154d960ab07 The attached patch enables Direct I/O feature for Commitlog files. Please check and share your feedback. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18464) Enable Direct I/O For CommitLog Files
[ https://issues.apache.org/jira/browse/CASSANDRA-18464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-18464: -- Attachment: UseDirectIOFeatureForCommitLogFiles.patch > Enable Direct I/O For CommitLog Files > - > > Key: CASSANDRA-18464 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18464 > Project: Cassandra > Issue Type: New Feature > Components: Local/Commit Log >Reporter: Josh McKenzie >Assignee: Amit Pawar >Priority: Normal > Attachments: UseDirectIOFeatureForCommitLogFiles.patch > > > Relocating from [dev@ email > thread.|https://lists.apache.org/thread/j6ny17q2rhkp7jxvwxm69dd6v1dozjrg] > > I shared my investigation about Commitlog I/O issue on large core count > system in my previous email dated July-22 and link to the thread is given > below. > [https://lists.apache.org/thread/xc5ocog2qz2v2gnj4xlw5hbthfqytx2n] > Basically, two solutions looked possible to improve the CommitLog I/O. > # Multi-threaded syncing > # Using Direct-IO through JNA > I worked on 2nd option considering the following benefit compared to the > first one > # Direct I/O read/write throughput is very high compared to non-Direct I/O. > Learnt through FIO benchmarking. > # Reduces kernel file cache uses which in-turn reduces kernel I/O activity > for Commitlog files only. > # Overall CPU usage reduced for flush activity. JVisualvm shows CPU usage < > 30% for Commitlog syncer thread with Direct I/O feature > # Direct I/O implementation is easier compared to multi-threaded > As per the community suggestion, less in code complex is good to have. Direct > I/O enablement looked promising but there was one issue. > Java version 8 does not have native support to enable Direct I/O. So, JNA > library usage is must. The same implementation should also work across other > versions of Java (like 11 and beyond). > I have completed Direct I/O implementation and summary of the attached patch > changes are given below. > # This implementation is not using Java file channels and file is opened > through JNA to use Direct I/O feature. > # New Segment are defined named “DirectIOSegment” for Direct I/O and > “NonDirectIOSegment” for non-direct I/O (NonDirectIOSegment is test purpose > only). > # JNA write call is used to flush the changes. > # New helper functions are defined in NativeLibrary.java and platform > specific file. Currently tested on Linux only. > # Patch allows user to configure optimum block size and alignment if > default values are not OK for CommitLog disk. > # Following configuration options are provided in Cassandra.yaml file > a. use_jna_for_commitlog_io : to use jna feature > b. use_direct_io_for_commitlog : to use Direct I/O feature. > c. direct_io_minimum_block_alignment: 512 (default) > d. nvme_disk_block_size: 32MiB (default and can be changed as per the > required size) > Test matrix is complex so CommitLog related testcases and TPCx-IOT benchmark > was tested. It works with both Java 8 and 11 versions. Compressed and > Encrypted based segments are not supported yet and it can be enabled later > based on the Community feedback. > Following improvement are seen with Direct I/O enablement. > # 32 cores >= ~15% > # 64 cores >= ~80% > Also, another observation would like to share here. Reading Commitlog files > with Direct I/O might help in reducing node bring-up time after the node > crash. > Tested with commit ID: 91f6a9aca8d3c22a03e68aa901a0b154d960ab07 > The attached patch enables Direct I/O feature for Commitlog files. Please > check and share your feedback. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18262) Switch checkstyle running only with JDK8 to be run with JDK11
[ https://issues.apache.org/jira/browse/CASSANDRA-18262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709688#comment-17709688 ] Josh McKenzie commented on CASSANDRA-18262: --- Created CASSANDRA-18435 to track the things that came up. > Switch checkstyle running only with JDK8 to be run with JDK11 > - > > Key: CASSANDRA-18262 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18262 > Project: Cassandra > Issue Type: Task > Components: Build >Reporter: Ekaterina Dimitrova >Assignee: Maxim Muzafarov >Priority: Normal > Fix For: 5.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently we run checkstyle, eclipse warnings (and probably anything else?) > only with JDK8. > In preparation to remove JDK8 we need to switch those to run with JDK11. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-18435) Provide checkstyle in ant scripts on all branches; improve documentation on recommended JDK env for dev
Josh McKenzie created CASSANDRA-18435:
-
Summary: Provide checkstyle in ant scripts on all branches;
improve documentation on recommended JDK env for dev
Key: CASSANDRA-18435
URL: https://issues.apache.org/jira/browse/CASSANDRA-18435
Project: Cassandra
Issue Type: Task
Components: Build
Reporter: Josh McKenzie
See CASSANDRA-18262 comment thread for discussion leading up to this
# We should checkstyle only on the lowest supported JDK for our builds in CI
# Having checkstyle be part of the local {{ant jar}} target for all supported
JDKs would be nice ergonomics for folks who are working with another supported
JDK version so they don't get surprised in CI
# We should tighten up our docs in terms of recommendations on which JDK to
build/run with locally and help smooth out that process for folks
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18262) Switch checkstyle running only with JDK8 to be run with JDK11
[ https://issues.apache.org/jira/browse/CASSANDRA-18262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709527#comment-17709527 ] Josh McKenzie commented on CASSANDRA-18262: --- bq. People have just Java 11 installed globally and that's it. They are not going to switch back and forth every time. I've been doing this for a long while without issue; even have a script to switch worktrees and a local git ignored setenv.sh that gets sourced to set up the correct environment for whatever I'm working on + python env. So I guess my n=1 anecdote here is: not _everyone_ has just Java 11 installed globally and that's it. If we're the odd project out w/requirements or recommendations for running an older JDK, we can do things to make it easier for folks (automation, scripting, documentation) to minimize the disruption / lift required from them to get working on our project. > Switch checkstyle running only with JDK8 to be run with JDK11 > - > > Key: CASSANDRA-18262 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18262 > Project: Cassandra > Issue Type: Task > Components: Build >Reporter: Ekaterina Dimitrova >Assignee: Maxim Muzafarov >Priority: Normal > Fix For: 5.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently we run checkstyle, eclipse warnings (and probably anything else?) > only with JDK8. > In preparation to remove JDK8 we need to switch those to run with JDK11. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18262) Switch checkstyle running only with JDK8 to be run with JDK11
[
https://issues.apache.org/jira/browse/CASSANDRA-18262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708941#comment-17708941
]
Josh McKenzie commented on CASSANDRA-18262:
---
bq. Why would you be developing and checking against 11 if you ship with 8 to
get to the point that you broke 8 only when you get to CI?
This is a really good point. This has kind of been nagging at me for years
without bubbling up to the surface.
Is there any reason to develop locally on any JDK other than the oldest one
supported, since we can't use new language features anyway? Is the thought that
the newer JDKs would potentially build faster or run local tests faster to
tighten the feedback loops working on code/
Currently our documentation
[link|https://cassandra.apache.org/_/development/index.html] kind of hand-waves
at the situation as being up to the individual developer:
{quote}
Prerequisites
Building Cassandra from source is the first important step in contributing to
the Apache Cassandra project. You will need to install Java (JDK 8 or 11,
depending on which version you want to build Cassandra against, best is to
install both and then switch between them when needed), Git, and Ant (including
ant-optional).
{quote}
It seems to me that we could have stronger verbiage along the lines of "We
recommend you build and run locally on the oldest supported JDK version X as
newer language features cannot be supported due to compilation requirements."
> Switch checkstyle running only with JDK8 to be run with JDK11
> -
>
> Key: CASSANDRA-18262
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18262
> Project: Cassandra
> Issue Type: Task
> Components: Build
>Reporter: Ekaterina Dimitrova
>Assignee: Maxim Muzafarov
>Priority: Normal
> Fix For: 5.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently we run checkstyle, eclipse warnings (and probably anything else?)
> only with JDK8.
> In preparation to remove JDK8 we need to switch those to run with JDK11.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18427) Test failure: pdtest: dtest-novnode.ttl_test.TestDistributedTTL.test_ttl_is_respected_on_delayed_replication
[
https://issues.apache.org/jira/browse/CASSANDRA-18427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie updated CASSANDRA-18427:
--
Status: Triage Needed (was: Awaiting Feedback)
> Test failure: pdtest:
> dtest-novnode.ttl_test.TestDistributedTTL.test_ttl_is_respected_on_delayed_replication
>
>
> Key: CASSANDRA-18427
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18427
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest/python
>Reporter: Josh McKenzie
>Priority: Normal
>
> {{Failed 1 times in the last 30 runs. Flakiness: 3%, Stability: 96%}}
> {code}
> Error Message
> AssertionError: Expected [[1, 1, None, None], [2, 2, None, None]] from SELECT
> * FROM ttl_table;, but got [[2, 2, None, None]]
> Stacktrace
> self =
> def test_ttl_is_respected_on_delayed_replication(self):
> """ Test that ttl is respected on delayed replication """
> self.prepare()
> self.node2.stop()
> self.session1.execute("""
> INSERT INTO ttl_table (key, col1) VALUES (1, 1) USING TTL 5;
> """)
> self.session1.execute("""
> INSERT INTO ttl_table (key, col1) VALUES (2, 2) USING TTL 1000;
> """)
> > assert_all(
> self.session1,
> "SELECT * FROM ttl_table;",
> [[1, 1, None, None], [2, 2, None, None]]
> )
> ttl_test.py:480:
> {code}
> At face value this didn't look like a classic "oops I timed out", but now
> that I read that error message I'm not as sure. So I'd check that first and
> just close out if it's a timeout by another name until we get the infra
> stabilized.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18262) Switch checkstyle running only with JDK8 to be run with JDK11
[
https://issues.apache.org/jira/browse/CASSANDRA-18262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708820#comment-17708820
]
Josh McKenzie commented on CASSANDRA-18262:
---
{quote}it was decided(not by me) that we run it with the lowest version that we
release our builds with and we skip with the later ones.
{quote}
... when was this? I can see the argument for it (shouldn't be different
between any of the branches and _was_ somewhat slow), but Maxim's point
resonates with me:
{quote}Instead of tying the code-checking tool to a specific version, we should
make it work for every jdk we support. This has its advantages: you can run
checkstyle locally and check the sources without switching jdks
{quote}
Working on another JDK locally and not realizing it's not linting is a pretty
subtle and annoying thing IMO. I'm +1 to it running on all branches if it's
what, the 30 seconds per branch on formal builds to validate?
Maybe worth bringing to the dev list to get a quick consensus check and/or ping
[~mck] and [~brandon.williams] since they've both been maintaining build stuff
recently.
> Switch checkstyle running only with JDK8 to be run with JDK11
> -
>
> Key: CASSANDRA-18262
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18262
> Project: Cassandra
> Issue Type: Task
> Components: Build
>Reporter: Ekaterina Dimitrova
>Assignee: Maxim Muzafarov
>Priority: Normal
> Fix For: 5.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently we run checkstyle, eclipse warnings (and probably anything else?)
> only with JDK8.
> In preparation to remove JDK8 we need to switch those to run with JDK11.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18425) Test failure: utest: org.apache.cassandra.db.RepairedDataTombstonesTest.readTestPartitionTombstones-.jdk11
[
https://issues.apache.org/jira/browse/CASSANDRA-18425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708818#comment-17708818
]
Josh McKenzie commented on CASSANDRA-18425:
---
bq. if that's where this happened.
Ugh; have been neglecting to include those details. I'll update the tickets
I've created recently with that info.
This was [trunk
JDK11|https://ci-cassandra.apache.org/job/Cassandra-trunk/1517/testReport/org.apache.cassandra.db/RepairedDataTombstonesTest/readTestPartitionTombstones__jdk11/]
> Test failure: utest:
> org.apache.cassandra.db.RepairedDataTombstonesTest.readTestPartitionTombstones-.jdk11
> --
>
> Key: CASSANDRA-18425
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18425
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Josh McKenzie
>Priority: Normal
>
> {{Failed 1 times in the last 5 runs. Flakiness: 25%, Stability: 80%}}
> {code}
> Error Message
> val=5
> Stacktrace
> junit.framework.AssertionFailedError: val=5
> at
> org.apache.cassandra.db.RepairedDataTombstonesTest.readTestPartitionTombstones(RepairedDataTombstonesTest.java:189)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18256) Backport CASSANDRA-17205 (remove strong self-ref in tidier) to all supported lines
[ https://issues.apache.org/jira/browse/CASSANDRA-18256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-18256: -- Resolution: Fixed Status: Resolved (was: Triage Needed) > Backport CASSANDRA-17205 (remove strong self-ref in tidier) to all supported > lines > -- > > Key: CASSANDRA-18256 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18256 > Project: Cassandra > Issue Type: Bug > Components: Local/SSTable >Reporter: Josh McKenzie >Assignee: Josh McKenzie >Priority: Normal > > CASSANDRA-17205 should probably have been backported to all supported > branches. It's potentially masking some issues seen w/refs in other locations > so we should pull it back to all supported lines. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15196) Unify CommitLogSegment allocation to one hierarchy and remove test-cdc testing target
[
https://issues.apache.org/jira/browse/CASSANDRA-15196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie updated CASSANDRA-15196:
--
Summary: Unify CommitLogSegment allocation to one hierarchy and remove
test-cdc testing target (was: Unify CommitLogSegment allocaiton to one
hierarchy and remove test-cdc testing target)
> Unify CommitLogSegment allocation to one hierarchy and remove test-cdc
> testing target
> -
>
> Key: CASSANDRA-15196
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15196
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Commit Log
>Reporter: Joshua McKenzie
>Assignee: Josh McKenzie
>Priority: Normal
> Fix For: 5.x
>
>
> {{April 2023 update}}
> With the push towards stabilizing ASF CI, one of the things we can do is
> remove the total runtime of our tests. CDC's segment allocation has been in
> the ecosystem for a long while now and has proven stable in the "CDC enabled
> on the node but not the table" case, so unifying the hierarchy to a single
> allocator and removing the test-cdc target in ant should both help us clean
> up some debt in the codebase and also streamline our CI processes. Along with
> that, we can take this chance to tidy up some of the internals of the
> implementation as previously mentioned on this ticket.
>
> {{ORIGINAL DESCRIPTION}}
> Back when I wrote CASSANDRA-8844 and CASSANDRA-12148, a few things about the
> CommitLog stood out to me that I wasn't in love with. At the time I was much
> more of the mind of "When in Rome..." regarding some of our structures, our
> naming, and specifically regarding our penchant for inheritance vs.
> composition. Turns out I have an itch here I need to scratch.
> This patch refactors the CommitLog in a few key ways:
> * Removes inheritence for CommitLogSegmentManagerX, instead has a single
> CommitLogSegmentManager and a CommitLogSegmentAllocator interface
> ** This interface is implemented by CommitLogSegmentAllocatorStandard and
> CommitLogSegmentAllocatorCDC
> * Renames a few variables and methods within the segment manager to make
> their purpose and role more clear from names alone (hopefully):
> ** allocatingFrom --> activeSegment
> ** allocatingFrom() --> getActiveSegment()
> ** advanceAllocatingFrom() – switchToNewSegment()
> ** activeSegments --> unflushedSegments
> ** getActiveSegments --> getUnflushedSegments()
> ** awaitAvailableSegment(...) --> awaitSegmentAllocation(...)
> ** isStillAllocating() --> hasRoom()
> * Reorders some of the "allocation-and-compare-in-while-loop-decl) to more
> idiomatic style
> * Adds some comments to various under-documented methods, mostly related to
> CDC and timing
> -As far as we are in the 4.0 testing, I don't expect this to hit pre 4.0.
> Given it's a relatively minor refactor in well-tested code, doesn't change
> functionality, and doesn't impact any of the publicly exposed APIs in the
> CommitLog ecosystem, I'd be comfortable with it going in a minor release
> after 4.0 is out. But I'm open to alternative viewpoints.-
> -Have run unit tests locally w/out issue on both regular and cdc-targets;
> need to get CI against the branch internally for full dtest run and I'll post
> when it's done. Just wanted to get visibility to this out there in case
> anyone was curious or thinking of working on the same thing.-
> [ Link to branch on
> github|https://github.com/apache/cassandra/compare/trunk...josh-mckenzie:clean_cl_comp]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-18427) Test failure: pdtest: dtest-novnode.ttl_test.TestDistributedTTL.test_ttl_is_respected_on_delayed_replication
Josh McKenzie created CASSANDRA-18427:
-
Summary: Test failure: pdtest:
dtest-novnode.ttl_test.TestDistributedTTL.test_ttl_is_respected_on_delayed_replication
Key: CASSANDRA-18427
URL: https://issues.apache.org/jira/browse/CASSANDRA-18427
Project: Cassandra
Issue Type: Bug
Components: Test/dtest/python
Reporter: Josh McKenzie
{{Failed 1 times in the last 30 runs. Flakiness: 3%, Stability: 96%}}
{code}
Error Message
AssertionError: Expected [[1, 1, None, None], [2, 2, None, None]] from SELECT *
FROM ttl_table;, but got [[2, 2, None, None]]
Stacktrace
self =
def test_ttl_is_respected_on_delayed_replication(self):
""" Test that ttl is respected on delayed replication """
self.prepare()
self.node2.stop()
self.session1.execute("""
INSERT INTO ttl_table (key, col1) VALUES (1, 1) USING TTL 5;
""")
self.session1.execute("""
INSERT INTO ttl_table (key, col1) VALUES (2, 2) USING TTL 1000;
""")
> assert_all(
self.session1,
"SELECT * FROM ttl_table;",
[[1, 1, None, None], [2, 2, None, None]]
)
ttl_test.py:480:
{code}
At face value this didn't look like a classic "oops I timed out", but now that
I read that error message I'm not as sure. So I'd check that first and just
close out if it's a timeout by another name until we get the infra stabilized.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-18426) Test failure: pdtest: dtest-upgrade.upgrade_tests.cql_tests.TestCQLNodes2RF1_Upgrade_current_2_1_x_To_indev_3_11_x.test_clustering_order_and_functions
Josh McKenzie created CASSANDRA-18426:
-
Summary: Test failure: pdtest:
dtest-upgrade.upgrade_tests.cql_tests.TestCQLNodes2RF1_Upgrade_current_2_1_x_To_indev_3_11_x.test_clustering_order_and_functions
Key: CASSANDRA-18426
URL: https://issues.apache.org/jira/browse/CASSANDRA-18426
Project: Cassandra
Issue Type: Bug
Components: Test/dtest/python
Reporter: Josh McKenzie
Same failure on:
{{dtest-upgrade.upgrade_tests.cql_tests.TestCQLNodes2RF1_Upgrade_current_2_1_x_To_indev_3_11_x.test_timeuuid}},
also on 3.0 branch
{{Failed 3 times in the last 30 runs. Flakiness: 3%, Stability: 90%}}
{code}
Error Message
cassandra.InvalidRequest: Error from server: code=2200 [Invalid query]
message="Unknown function 'totimestamp'"
{code}
Maybe related to merge of CASSANDRA-18328?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-18425) Test failure: utest: org.apache.cassandra.db.RepairedDataTombstonesTest.readTestPartitionTombstones-.jdk11
Josh McKenzie created CASSANDRA-18425:
-
Summary: Test failure: utest:
org.apache.cassandra.db.RepairedDataTombstonesTest.readTestPartitionTombstones-.jdk11
Key: CASSANDRA-18425
URL: https://issues.apache.org/jira/browse/CASSANDRA-18425
Project: Cassandra
Issue Type: Bug
Components: Test/unit
Reporter: Josh McKenzie
{{Failed 1 times in the last 5 runs. Flakiness: 25%, Stability: 80%}}
{code}
Error Message
val=5
Stacktrace
junit.framework.AssertionFailedError: val=5
at
org.apache.cassandra.db.RepairedDataTombstonesTest.readTestPartitionTombstones(RepairedDataTombstonesTest.java:189)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16895) Build with Java 17
[ https://issues.apache.org/jira/browse/CASSANDRA-16895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708543#comment-17708543 ] Josh McKenzie commented on CASSANDRA-16895: --- bq. Once Accord is working as part of the C* code base I do not understand how it could use a different JDK. For sure; this merge wouldn't make that viable. I forgot about the subproject nesting build-time coupling change. For the simulator if it's in another project, one could theoretically use more advanced JDK features in it if C* itself were able to _run_ on those newer JDK's. I think, in theory, you wouldn't really risk violating any of your ordering and correctness contracts the simulator is stepping through just by virtue of a newer JDK running something built to an older language model. In theory. Way out of my area of deep knowledge here so I'll defer to [~benedict] to provide further context or detail if he so chooses. > Build with Java 17 > -- > > Key: CASSANDRA-16895 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16895 > Project: Cassandra > Issue Type: Epic > Components: Build >Reporter: Ekaterina Dimitrova >Assignee: Ekaterina Dimitrova >Priority: Normal > Fix For: 5.x > > > This ticket is intended to group all issues found to support Java 17 in the > future. > Upgrade steps: > * [Dependencies > |https://mvnrepository.com/artifact/org.apache.cassandra/cassandra-all/4.0.1]to > be updated (not all but at least those that require an update in order to > work with Java 17) > * More encapsulated JDK internal APIs. Some of the issues might be solved > with the dependencies updates > * Currently trunk compiles if we remove the Nashorn dependency (ant script > tag, used for the test environment; UDFs) . The oracle recommendation to use > Nashorn-core won't work for the project as it is under GPL 2.0. Most probably > we will opt in for graal-sdk licensed under UPL > * All tests to be cleaned > * CI environment to be setup > *NOTE:* GC tuning, performance testing were never agreed to be part of this > ticket. > Below is a snapshot of current CI failures with JDK17, it will be updated on > a regular basis with a date of update > *March 19th 2023* > || ||Failing Test Classes||Ticket Numbers|| > | |_Python DTests_| | > |1|test_sjk|CASSANDRA-18343| > | |_Java Ditributed Tests_| | > |1-6|org.apache.cassandra.distributed.test.ReprepareOldBehaviourTest - all > tests, > org.apache.cassandra.distributed.test.PrepareBatchStatementsTest - all tests, > org.apache.cassandra.distributed.test.IPMembershipTest - both tests, > org.apache.cassandra.distributed.test.MixedModeFuzzTest, > org.apache.cassandra.distributed.test.ReprepareFuzzTest, > org.apache.cassandra.distributed.test.ReprepareNewBehaviourTest|CASSANDRA-16304| > |7,8|org.apache.cassandra.distributed.test.NativeTransportEncryptionOptionsTest > - all tests > org.apache.cassandra.distributed.test.InternodeEncryptionOptionsTest - all > tests|Could be related to CASSANDRA-18180, to be checked | > |9|org.apache.cassandra.distributed.test.SSTableLoaderEncryptionOptionsTest - > 2 tests|CASSANDRA-18180| > | |_Unit Tests_| | > |1|org.apache.cassandra.repair.RepairJobTest - 1 test|CASSANDRA-17884| > |2|org.apache.cassandra.security.SSLFactoryTest - all tests|CASSANDRA-17992| > |3,4|org.apache.cassandra.db.memtable.MemtableSizeOffheapBuffersTest, > org.apache.cassandra.utils.concurrent.RefCountedTest|CASSANDRA-18329| > |5,6|org.apache.cassandra.cql3.validation.entities.UFJavaTest, > org.apache.cassandra.cql3.validation.entities.UFSecurityTest|CASSANDRA-18190| > | | | | > | | | | > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-18424) Implement graceful paging across tombstones with short-circuit on paging rather than throwing TombstoneOverwhelmingExceptions
Josh McKenzie created CASSANDRA-18424:
-
Summary: Implement graceful paging across tombstones with
short-circuit on paging rather than throwing TombstoneOverwhelmingExceptions
Key: CASSANDRA-18424
URL: https://issues.apache.org/jira/browse/CASSANDRA-18424
Project: Cassandra
Issue Type: New Feature
Components: Messaging/Client, Messaging/Internode
Reporter: Josh McKenzie
Assignee: Josh McKenzie
We implemented the hard stop with a {{TombstoneOverwhelmingException}} almost a
decade ago since paging across many tombstones was the most common way for
nodes to OOM as they iterated across all this data during queries and paging.
With our current implementations and architecture / codebase, we should be able
to combine the {{StoppingTransformation}} and existing {{clustering}} blob we
pass back to clients to allow clients to optionally page across tombstones when
using the async api via the driver and short-circuit a page when they hit the
tombstone failure threshold rather than throwing a
{{{}TombstoneOverwhelmingException{}}}. This would allow for more flexible data
modeling on users' side as well as removing one of the fairly rough edges of
our API's we're currently constrained by.
Making sure this is correct will require extensive fuzz-testing of pagination;
this should likely happen in the Harry project but we could also have a bespoke
model / implementation in the C* codebase we rely on in the interim.
Client warnings at the current default levels would remain; the gap between
warn and "short-circuit pages" (100x ratio currently, 1000 vs. 10) should
be sufficient for clients to take action on their data models well before they
hit this limit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12367) Add API / vtable support to request the uncompressed, compressed, and estimated tombstoned size of a CQL partition
[ https://issues.apache.org/jira/browse/CASSANDRA-12367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708531#comment-17708531 ] Josh McKenzie commented on CASSANDRA-12367: --- This one's been quiet for a long while; I'm working on something for this [~geoffxy] and I snagged the ticket; let me know if there's any issues there. > Add API / vtable support to request the uncompressed, compressed, and > estimated tombstoned size of a CQL partition > -- > > Key: CASSANDRA-12367 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12367 > Project: Cassandra > Issue Type: Improvement > Components: CQL/Semantics >Reporter: Geoffrey Yu >Assignee: Josh McKenzie >Priority: Normal > Fix For: 5.x > > Attachments: 12367-trunk-v2.txt, 12367-trunk.txt > > > It would be useful to have an API that we could use to get the total > serialized size of a CQL partition, scoped by keyspace and table, on disk. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-12367) Add API / vtable support to request the uncompressed, compressed, and estimated tombstoned size of a CQL partition
[ https://issues.apache.org/jira/browse/CASSANDRA-12367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-12367: -- Description: It would be useful to have an endpoint in our vtables where we could use to get the uncompressed and compressed size of a partition on disk quickly, as well as the ability to scan across a partition and calculate how much of said partition was live vs. dead data. (was: It would be useful to have an API that we could use to get the total serialized size of a CQL partition, scoped by keyspace and table, on disk.) > Add API / vtable support to request the uncompressed, compressed, and > estimated tombstoned size of a CQL partition > -- > > Key: CASSANDRA-12367 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12367 > Project: Cassandra > Issue Type: Improvement > Components: CQL/Semantics >Reporter: Geoffrey Yu >Assignee: Josh McKenzie >Priority: Normal > Fix For: 5.x > > Attachments: 12367-trunk-v2.txt, 12367-trunk.txt > > > It would be useful to have an endpoint in our vtables where we could use to > get the uncompressed and compressed size of a partition on disk quickly, as > well as the ability to scan across a partition and calculate how much of said > partition was live vs. dead data. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-12367) Add API / vtable support to request the uncompressed, compressed, and estimated tombstoned size of a CQL partition
[ https://issues.apache.org/jira/browse/CASSANDRA-12367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-12367: -- Status: Triage Needed (was: Awaiting Feedback) > Add API / vtable support to request the uncompressed, compressed, and > estimated tombstoned size of a CQL partition > -- > > Key: CASSANDRA-12367 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12367 > Project: Cassandra > Issue Type: Improvement >Reporter: Geoffrey Yu >Assignee: Josh McKenzie >Priority: Low > Fix For: 3.11.x > > Attachments: 12367-trunk-v2.txt, 12367-trunk.txt > > > It would be useful to have an API that we could use to get the total > serialized size of a CQL partition, scoped by keyspace and table, on disk. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-12367) Add API / vtable support to request the uncompressed, compressed, and estimated tombstoned size of a CQL partition
[ https://issues.apache.org/jira/browse/CASSANDRA-12367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-12367: -- Change Category: Operability Complexity: Normal Component/s: CQL/Semantics Fix Version/s: 5.x (was: 3.11.x) Priority: Normal (was: Low) Status: Open (was: Triage Needed) > Add API / vtable support to request the uncompressed, compressed, and > estimated tombstoned size of a CQL partition > -- > > Key: CASSANDRA-12367 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12367 > Project: Cassandra > Issue Type: Improvement > Components: CQL/Semantics >Reporter: Geoffrey Yu >Assignee: Josh McKenzie >Priority: Normal > Fix For: 5.x > > Attachments: 12367-trunk-v2.txt, 12367-trunk.txt > > > It would be useful to have an API that we could use to get the total > serialized size of a CQL partition, scoped by keyspace and table, on disk. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-12367) Add API / vtable support to request the uncompressed, compressed, and estimated tombstoned size of a CQL partition
[ https://issues.apache.org/jira/browse/CASSANDRA-12367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie reassigned CASSANDRA-12367: - Assignee: Josh McKenzie (was: Geoffrey Yu) > Add API / vtable support to request the uncompressed, compressed, and > estimated tombstoned size of a CQL partition > -- > > Key: CASSANDRA-12367 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12367 > Project: Cassandra > Issue Type: Improvement >Reporter: Geoffrey Yu >Assignee: Josh McKenzie >Priority: Low > Fix For: 3.11.x > > Attachments: 12367-trunk-v2.txt, 12367-trunk.txt > > > It would be useful to have an API that we could use to get the total > serialized size of a CQL partition, scoped by keyspace and table, on disk. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-12367) Add API / vtable support to request the uncompressed, compressed, and estimated tombstoned size of a CQL partition
[ https://issues.apache.org/jira/browse/CASSANDRA-12367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-12367: -- Summary: Add API / vtable support to request the uncompressed, compressed, and estimated tombstoned size of a CQL partition (was: Add an API to request the size of a CQL partition) > Add API / vtable support to request the uncompressed, compressed, and > estimated tombstoned size of a CQL partition > -- > > Key: CASSANDRA-12367 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12367 > Project: Cassandra > Issue Type: Improvement >Reporter: Geoffrey Yu >Assignee: Geoffrey Yu >Priority: Low > Fix For: 3.11.x > > Attachments: 12367-trunk-v2.txt, 12367-trunk.txt > > > It would be useful to have an API that we could use to get the total > serialized size of a CQL partition, scoped by keyspace and table, on disk. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15196) Unify CommitLogSegment allocaiton to one hierarchy and remove test-cdc testing target
[
https://issues.apache.org/jira/browse/CASSANDRA-15196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh McKenzie updated CASSANDRA-15196:
--
Description:
{{April 2023 update}}
With the push towards stabilizing ASF CI, one of the things we can do is remove
the total runtime of our tests. CDC's segment allocation has been in the
ecosystem for a long while now and has proven stable in the "CDC enabled on the
node but not the table" case, so unifying the hierarchy to a single allocator
and removing the test-cdc target in ant should both help us clean up some debt
in the codebase and also streamline our CI processes. Along with that, we can
take this chance to tidy up some of the internals of the implementation as
previously mentioned on this ticket.
{{ORIGINAL DESCRIPTION}}
Back when I wrote CASSANDRA-8844 and CASSANDRA-12148, a few things about the
CommitLog stood out to me that I wasn't in love with. At the time I was much
more of the mind of "When in Rome..." regarding some of our structures, our
naming, and specifically regarding our penchant for inheritance vs.
composition. Turns out I have an itch here I need to scratch.
This patch refactors the CommitLog in a few key ways:
* Removes inheritence for CommitLogSegmentManagerX, instead has a single
CommitLogSegmentManager and a CommitLogSegmentAllocator interface
** This interface is implemented by CommitLogSegmentAllocatorStandard and
CommitLogSegmentAllocatorCDC
* Renames a few variables and methods within the segment manager to make their
purpose and role more clear from names alone (hopefully):
** allocatingFrom --> activeSegment
** allocatingFrom() --> getActiveSegment()
** advanceAllocatingFrom() – switchToNewSegment()
** activeSegments --> unflushedSegments
** getActiveSegments --> getUnflushedSegments()
** awaitAvailableSegment(...) --> awaitSegmentAllocation(...)
** isStillAllocating() --> hasRoom()
* Reorders some of the "allocation-and-compare-in-while-loop-decl) to more
idiomatic style
* Adds some comments to various under-documented methods, mostly related to
CDC and timing
-As far as we are in the 4.0 testing, I don't expect this to hit pre 4.0. Given
it's a relatively minor refactor in well-tested code, doesn't change
functionality, and doesn't impact any of the publicly exposed APIs in the
CommitLog ecosystem, I'd be comfortable with it going in a minor release after
4.0 is out. But I'm open to alternative viewpoints.-
-Have run unit tests locally w/out issue on both regular and cdc-targets; need
to get CI against the branch internally for full dtest run and I'll post when
it's done. Just wanted to get visibility to this out there in case anyone was
curious or thinking of working on the same thing.-
[ Link to branch on
github|https://github.com/apache/cassandra/compare/trunk...josh-mckenzie:clean_cl_comp]
was:
Back when I wrote CASSANDRA-8844 and CASSANDRA-12148, a few things about the
CommitLog stood out to me that I wasn't in love with. At the time I was much
more of the mind of "When in Rome..." regarding some of our structures, our
naming, and specifically regarding our penchant for inheritance vs.
composition. Turns out I have an itch here I need to scratch.
This patch refactors the CommitLog in a few key ways:
* Removes inheritence for CommitLogSegmentManagerX, instead has a single
CommitLogSegmentManager and a CommitLogSegmentAllocator interface
** This interface is implemented by CommitLogSegmentAllocatorStandard and
CommitLogSegmentAllocatorCDC
* Renames a few variables and methods within the segment manager to make their
purpose and role more clear from names alone (hopefully):
** allocatingFrom --> activeSegment
** allocatingFrom() --> getActiveSegment()
** advanceAllocatingFrom() – switchToNewSegment()
** activeSegments --> unflushedSegments
** getActiveSegments --> getUnflushedSegments()
** awaitAvailableSegment(...) --> awaitSegmentAllocation(...)
** isStillAllocating() --> hasRoom()
* Reorders some of the "allocation-and-compare-in-while-loop-decl) to more
idiomatic style
* Adds some comments to various under-documented methods, mostly related to
CDC and timing
As far as we are in the 4.0 testing, I don't expect this to hit pre 4.0. Given
it's a relatively minor refactor in well-tested code, doesn't change
functionality, and doesn't impact any of the publicly exposed APIs in the
CommitLog ecosystem, I'd be comfortable with it going in a minor release after
4.0 is out. But I'm open to alternative viewpoints.
Have run unit tests locally w/out issue on both regular and cdc-targets; need
to get CI against the branch internally for full dtest run and I'll post when
it's done. Just wanted to get visibility to this out there in case anyone was
curious or thinking of working on the same thing.
[ Link to branch on
[jira] [Updated] (CASSANDRA-15196) Refactor inheritance around CommitLogSegment management and allocation
[ https://issues.apache.org/jira/browse/CASSANDRA-15196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-15196: -- Resolution: (was: Not A Problem) Status: Open (was: Resolved) > Refactor inheritance around CommitLogSegment management and allocation > -- > > Key: CASSANDRA-15196 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15196 > Project: Cassandra > Issue Type: Improvement > Components: Local/Commit Log >Reporter: Joshua McKenzie >Assignee: Josh McKenzie >Priority: Normal > > Back when I wrote CASSANDRA-8844 and CASSANDRA-12148, a few things about the > CommitLog stood out to me that I wasn't in love with. At the time I was much > more of the mind of "When in Rome..." regarding some of our structures, our > naming, and specifically regarding our penchant for inheritance vs. > composition. Turns out I have an itch here I need to scratch. > This patch refactors the CommitLog in a few key ways: > * Removes inheritence for CommitLogSegmentManagerX, instead has a single > CommitLogSegmentManager and a CommitLogSegmentAllocator interface > ** This interface is implemented by CommitLogSegmentAllocatorStandard and > CommitLogSegmentAllocatorCDC > * Renames a few variables and methods within the segment manager to make > their purpose and role more clear from names alone (hopefully): > ** allocatingFrom --> activeSegment > ** allocatingFrom() --> getActiveSegment() > ** advanceAllocatingFrom() – switchToNewSegment() > ** activeSegments --> unflushedSegments > ** getActiveSegments --> getUnflushedSegments() > ** awaitAvailableSegment(...) --> awaitSegmentAllocation(...) > ** isStillAllocating() --> hasRoom() > * Reorders some of the "allocation-and-compare-in-while-loop-decl) to more > idiomatic style > * Adds some comments to various under-documented methods, mostly related to > CDC and timing > As far as we are in the 4.0 testing, I don't expect this to hit pre 4.0. > Given it's a relatively minor refactor in well-tested code, doesn't change > functionality, and doesn't impact any of the publicly exposed APIs in the > CommitLog ecosystem, I'd be comfortable with it going in a minor release > after 4.0 is out. But I'm open to alternative viewpoints. > Have run unit tests locally w/out issue on both regular and cdc-targets; need > to get CI against the branch internally for full dtest run and I'll post when > it's done. Just wanted to get visibility to this out there in case anyone was > curious or thinking of working on the same thing. > [ Link to branch on > github|https://github.com/apache/cassandra/compare/trunk...josh-mckenzie:clean_cl_comp] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15196) Unify CommitLogSegment allocaiton to one hierarchy and remove test-cdc testing target
[ https://issues.apache.org/jira/browse/CASSANDRA-15196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-15196: -- Summary: Unify CommitLogSegment allocaiton to one hierarchy and remove test-cdc testing target (was: Refactor inheritance around CommitLogSegment management and allocation) > Unify CommitLogSegment allocaiton to one hierarchy and remove test-cdc > testing target > - > > Key: CASSANDRA-15196 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15196 > Project: Cassandra > Issue Type: Improvement > Components: Local/Commit Log >Reporter: Joshua McKenzie >Assignee: Josh McKenzie >Priority: Normal > > Back when I wrote CASSANDRA-8844 and CASSANDRA-12148, a few things about the > CommitLog stood out to me that I wasn't in love with. At the time I was much > more of the mind of "When in Rome..." regarding some of our structures, our > naming, and specifically regarding our penchant for inheritance vs. > composition. Turns out I have an itch here I need to scratch. > This patch refactors the CommitLog in a few key ways: > * Removes inheritence for CommitLogSegmentManagerX, instead has a single > CommitLogSegmentManager and a CommitLogSegmentAllocator interface > ** This interface is implemented by CommitLogSegmentAllocatorStandard and > CommitLogSegmentAllocatorCDC > * Renames a few variables and methods within the segment manager to make > their purpose and role more clear from names alone (hopefully): > ** allocatingFrom --> activeSegment > ** allocatingFrom() --> getActiveSegment() > ** advanceAllocatingFrom() – switchToNewSegment() > ** activeSegments --> unflushedSegments > ** getActiveSegments --> getUnflushedSegments() > ** awaitAvailableSegment(...) --> awaitSegmentAllocation(...) > ** isStillAllocating() --> hasRoom() > * Reorders some of the "allocation-and-compare-in-while-loop-decl) to more > idiomatic style > * Adds some comments to various under-documented methods, mostly related to > CDC and timing > As far as we are in the 4.0 testing, I don't expect this to hit pre 4.0. > Given it's a relatively minor refactor in well-tested code, doesn't change > functionality, and doesn't impact any of the publicly exposed APIs in the > CommitLog ecosystem, I'd be comfortable with it going in a minor release > after 4.0 is out. But I'm open to alternative viewpoints. > Have run unit tests locally w/out issue on both regular and cdc-targets; need > to get CI against the branch internally for full dtest run and I'll post when > it's done. Just wanted to get visibility to this out there in case anyone was > curious or thinking of working on the same thing. > [ Link to branch on > github|https://github.com/apache/cassandra/compare/trunk...josh-mckenzie:clean_cl_comp] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15196) Unify CommitLogSegment allocaiton to one hierarchy and remove test-cdc testing target
[ https://issues.apache.org/jira/browse/CASSANDRA-15196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-15196: -- Fix Version/s: 5.x > Unify CommitLogSegment allocaiton to one hierarchy and remove test-cdc > testing target > - > > Key: CASSANDRA-15196 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15196 > Project: Cassandra > Issue Type: Improvement > Components: Local/Commit Log >Reporter: Joshua McKenzie >Assignee: Josh McKenzie >Priority: Normal > Fix For: 5.x > > > Back when I wrote CASSANDRA-8844 and CASSANDRA-12148, a few things about the > CommitLog stood out to me that I wasn't in love with. At the time I was much > more of the mind of "When in Rome..." regarding some of our structures, our > naming, and specifically regarding our penchant for inheritance vs. > composition. Turns out I have an itch here I need to scratch. > This patch refactors the CommitLog in a few key ways: > * Removes inheritence for CommitLogSegmentManagerX, instead has a single > CommitLogSegmentManager and a CommitLogSegmentAllocator interface > ** This interface is implemented by CommitLogSegmentAllocatorStandard and > CommitLogSegmentAllocatorCDC > * Renames a few variables and methods within the segment manager to make > their purpose and role more clear from names alone (hopefully): > ** allocatingFrom --> activeSegment > ** allocatingFrom() --> getActiveSegment() > ** advanceAllocatingFrom() – switchToNewSegment() > ** activeSegments --> unflushedSegments > ** getActiveSegments --> getUnflushedSegments() > ** awaitAvailableSegment(...) --> awaitSegmentAllocation(...) > ** isStillAllocating() --> hasRoom() > * Reorders some of the "allocation-and-compare-in-while-loop-decl) to more > idiomatic style > * Adds some comments to various under-documented methods, mostly related to > CDC and timing > As far as we are in the 4.0 testing, I don't expect this to hit pre 4.0. > Given it's a relatively minor refactor in well-tested code, doesn't change > functionality, and doesn't impact any of the publicly exposed APIs in the > CommitLog ecosystem, I'd be comfortable with it going in a minor release > after 4.0 is out. But I'm open to alternative viewpoints. > Have run unit tests locally w/out issue on both regular and cdc-targets; need > to get CI against the branch internally for full dtest run and I'll post when > it's done. Just wanted to get visibility to this out there in case anyone was > curious or thinking of working on the same thing. > [ Link to branch on > github|https://github.com/apache/cassandra/compare/trunk...josh-mckenzie:clean_cl_comp] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18332) Backport CASSANDRA-17205 to 4.0 branch (strong ref leak)
[ https://issues.apache.org/jira/browse/CASSANDRA-18332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708105#comment-17708105 ] Josh McKenzie commented on CASSANDRA-18332: --- Committed. Thanks for the review and assist on CI [~smiklosovic] [cassandra-4.0 commit|https://gitbox.apache.org/repos/asf?p=cassandra.git;a=commit;h=f6fce7ab51e41dfd8d9584ce4c8dd35e0ed61598] [cassandra-4.1 merge|https://gitbox.apache.org/repos/asf?p=cassandra.git;a=commit;h=085765a07c9eb87e05a700ba4395c9df672a60c1] [trunk merge|https://gitbox.apache.org/repos/asf?p=cassandra.git;a=commit;h=e96c20e8150fd81716d203aa40301c09db0bc04a] > Backport CASSANDRA-17205 to 4.0 branch (strong ref leak) > > > Key: CASSANDRA-18332 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18332 > Project: Cassandra > Issue Type: Bug > Components: Local/SSTable >Reporter: Josh McKenzie >Assignee: Josh McKenzie >Priority: Normal > Fix For: 4.0.9, 5.0, 4.1.2 > > > See description in CASSANDRA-17205; this should have been applied on 4.0 and > merged up but was overlooked. > > Also double-check that strong leaks are logged at ERROR instead of WARN on > both 4.0, 4.1, and trunk (see > [comment|https://issues.apache.org/jira/browse/CASSANDRA-18176?focusedCommentId=17687184=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17687184]) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18332) Backport CASSANDRA-17205 to 4.0 branch (strong ref leak)
[ https://issues.apache.org/jira/browse/CASSANDRA-18332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-18332: -- Fix Version/s: 4.0.9 5.0 4.1.2 (was: 4.0.x) Since Version: 4.0 Source Control Link: https://gitbox.apache.org/repos/asf?p=cassandra.git;a=commit;h=f6fce7ab51e41dfd8d9584ce4c8dd35e0ed61598 Resolution: Fixed Status: Resolved (was: Ready to Commit) > Backport CASSANDRA-17205 to 4.0 branch (strong ref leak) > > > Key: CASSANDRA-18332 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18332 > Project: Cassandra > Issue Type: Bug > Components: Local/SSTable >Reporter: Josh McKenzie >Assignee: Josh McKenzie >Priority: Normal > Fix For: 4.0.9, 5.0, 4.1.2 > > > See description in CASSANDRA-17205; this should have been applied on 4.0 and > merged up but was overlooked. > > Also double-check that strong leaks are logged at ERROR instead of WARN on > both 4.0, 4.1, and trunk (see > [comment|https://issues.apache.org/jira/browse/CASSANDRA-18176?focusedCommentId=17687184=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17687184]) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18332) Backport CASSANDRA-17205 to 4.0 branch (strong ref leak)
[ https://issues.apache.org/jira/browse/CASSANDRA-18332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-18332: -- Reviewers: Stefan Miklosovic, Josh McKenzie Stefan Miklosovic, Josh McKenzie (was: Josh McKenzie, Stefan Miklosovic) Status: Review In Progress (was: Patch Available) > Backport CASSANDRA-17205 to 4.0 branch (strong ref leak) > > > Key: CASSANDRA-18332 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18332 > Project: Cassandra > Issue Type: Bug > Components: Local/SSTable >Reporter: Josh McKenzie >Assignee: Josh McKenzie >Priority: Normal > Fix For: 4.0.x > > > See description in CASSANDRA-17205; this should have been applied on 4.0 and > merged up but was overlooked. > > Also double-check that strong leaks are logged at ERROR instead of WARN on > both 4.0, 4.1, and trunk (see > [comment|https://issues.apache.org/jira/browse/CASSANDRA-18176?focusedCommentId=17687184=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17687184]) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18332) Backport CASSANDRA-17205 to 4.0 branch (strong ref leak)
[ https://issues.apache.org/jira/browse/CASSANDRA-18332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-18332: -- Status: Ready to Commit (was: Review In Progress) > Backport CASSANDRA-17205 to 4.0 branch (strong ref leak) > > > Key: CASSANDRA-18332 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18332 > Project: Cassandra > Issue Type: Bug > Components: Local/SSTable >Reporter: Josh McKenzie >Assignee: Josh McKenzie >Priority: Normal > Fix For: 4.0.x > > > See description in CASSANDRA-17205; this should have been applied on 4.0 and > merged up but was overlooked. > > Also double-check that strong leaks are logged at ERROR instead of WARN on > both 4.0, 4.1, and trunk (see > [comment|https://issues.apache.org/jira/browse/CASSANDRA-18176?focusedCommentId=17687184=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17687184]) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16895) Build with Java 17
[ https://issues.apache.org/jira/browse/CASSANDRA-16895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708009#comment-17708009 ] Josh McKenzie commented on CASSANDRA-16895: --- bq. you need to be compatible with the lowest JDK supported by C* to provide a patch On the language features you use, yes, but in terms of the ecosystem you're working with, to Benedict's point we could have a version of the simulator / accord / etc that run on newer JDK's leveraging these newer features and just have the core C* repo constrained to the lower language level. Definitely agree that it's not for the faint of heart / new to the ecosystem, but I think there is value for it given those projects are independent from the core C* DB. > Build with Java 17 > -- > > Key: CASSANDRA-16895 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16895 > Project: Cassandra > Issue Type: Epic > Components: Build >Reporter: Ekaterina Dimitrova >Assignee: Ekaterina Dimitrova >Priority: Normal > Fix For: 5.x > > > This ticket is intended to group all issues found to support Java 17 in the > future. > Upgrade steps: > * [Dependencies > |https://mvnrepository.com/artifact/org.apache.cassandra/cassandra-all/4.0.1]to > be updated (not all but at least those that require an update in order to > work with Java 17) > * More encapsulated JDK internal APIs. Some of the issues might be solved > with the dependencies updates > * Currently trunk compiles if we remove the Nashorn dependency (ant script > tag, used for the test environment; UDFs) . The oracle recommendation to use > Nashorn-core won't work for the project as it is under GPL 2.0. Most probably > we will opt in for graal-sdk licensed under UPL > * All tests to be cleaned > * CI environment to be setup > *NOTE:* GC tuning, performance testing were never agreed to be part of this > ticket. > Below is a snapshot of current CI failures with JDK17, it will be updated on > a regular basis with a date of update > *March 19th 2023* > || ||Failing Test Classes||Ticket Numbers|| > | |_Python DTests_| | > |1|test_sjk|CASSANDRA-18343| > | |_Java Ditributed Tests_| | > |1-6|org.apache.cassandra.distributed.test.ReprepareOldBehaviourTest - all > tests, > org.apache.cassandra.distributed.test.PrepareBatchStatementsTest - all tests, > org.apache.cassandra.distributed.test.IPMembershipTest - both tests, > org.apache.cassandra.distributed.test.MixedModeFuzzTest, > org.apache.cassandra.distributed.test.ReprepareFuzzTest, > org.apache.cassandra.distributed.test.ReprepareNewBehaviourTest|CASSANDRA-16304| > |7,8|org.apache.cassandra.distributed.test.NativeTransportEncryptionOptionsTest > - all tests > org.apache.cassandra.distributed.test.InternodeEncryptionOptionsTest - all > tests|Could be related to CASSANDRA-18180, to be checked | > |9|org.apache.cassandra.distributed.test.SSTableLoaderEncryptionOptionsTest - > 2 tests|CASSANDRA-18180| > | |_Unit Tests_| | > |1|org.apache.cassandra.repair.RepairJobTest - 1 test|CASSANDRA-17884| > |2|org.apache.cassandra.security.SSLFactoryTest - all tests|CASSANDRA-17992| > |3,4|org.apache.cassandra.db.memtable.MemtableSizeOffheapBuffersTest, > org.apache.cassandra.utils.concurrent.RefCountedTest|CASSANDRA-18329| > |5,6|org.apache.cassandra.cql3.validation.entities.UFJavaTest, > org.apache.cassandra.cql3.validation.entities.UFSecurityTest|CASSANDRA-18190| > | | | | > | | | | > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-18332) Backport CASSANDRA-17205 to 4.0 branch (strong ref leak)
[ https://issues.apache.org/jira/browse/CASSANDRA-18332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17707593#comment-17707593 ] Josh McKenzie edited comment on CASSANDRA-18332 at 4/1/23 10:58 PM: bq. If I understand it correctly, this has to go to 4.0 and then the changes in constructor on that parentRef order as well as logging from warn to error level should be applied to 4.1 and trunk, right? Correct. The logging level change is material and the other is more or less just trivial hygiene. Edit: looks like just 1 flake on both [JDK8|https://app.circleci.com/pipelines/github/instaclustr/cassandra/2044/workflows/80d2d0e8-6253-43af-b90f-d75c8cb05726/jobs/18359/tests] and [JDK11|https://app.circleci.com/pipelines/github/instaclustr/cassandra/2044/workflows/0b2d215a-8de2-4f69-8b67-314a0eb04bc4/jobs/18335/tests], same test. I'll circle back to that test on Monday and see how it holds up under clean multiplexing on 4.0 on circle; if it's got issues I'll go ahead and create a JIRA for it. You good w/me going through merge up process Monday otherwise [~smiklosovic]? was (Author: jmckenzie): bq. If I understand it correctly, this has to go to 4.0 and then the changes in constructor on that parentRef order as well as logging from warn to error level should be applied to 4.1 and trunk, right? Correct. The logging level change is material and the other is more or less just trivial hygiene. > Backport CASSANDRA-17205 to 4.0 branch (strong ref leak) > > > Key: CASSANDRA-18332 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18332 > Project: Cassandra > Issue Type: Bug > Components: Local/SSTable >Reporter: Josh McKenzie >Assignee: Josh McKenzie >Priority: Normal > Fix For: 4.0.x > > > See description in CASSANDRA-17205; this should have been applied on 4.0 and > merged up but was overlooked. > > Also double-check that strong leaks are logged at ERROR instead of WARN on > both 4.0, 4.1, and trunk (see > [comment|https://issues.apache.org/jira/browse/CASSANDRA-18176?focusedCommentId=17687184=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17687184]) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18332) Backport CASSANDRA-17205 to 4.0 branch (strong ref leak)
[ https://issues.apache.org/jira/browse/CASSANDRA-18332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17707593#comment-17707593 ] Josh McKenzie commented on CASSANDRA-18332: --- bq. If I understand it correctly, this has to go to 4.0 and then the changes in constructor on that parentRef order as well as logging from warn to error level should be applied to 4.1 and trunk, right? Correct. The logging level change is material and the other is more or less just trivial hygiene. > Backport CASSANDRA-17205 to 4.0 branch (strong ref leak) > > > Key: CASSANDRA-18332 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18332 > Project: Cassandra > Issue Type: Bug > Components: Local/SSTable >Reporter: Josh McKenzie >Assignee: Josh McKenzie >Priority: Normal > Fix For: 4.0.x > > > See description in CASSANDRA-17205; this should have been applied on 4.0 and > merged up but was overlooked. > > Also double-check that strong leaks are logged at ERROR instead of WARN on > both 4.0, 4.1, and trunk (see > [comment|https://issues.apache.org/jira/browse/CASSANDRA-18176?focusedCommentId=17687184=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17687184]) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18399) Add simple helper script for commiting a change to multiple branches
[
https://issues.apache.org/jira/browse/CASSANDRA-18399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17707592#comment-17707592
]
Josh McKenzie commented on CASSANDRA-18399:
---
ping [~dcapwell]
> Add simple helper script for commiting a change to multiple branches
>
>
> Key: CASSANDRA-18399
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18399
> Project: Cassandra
> Issue Type: Task
> Components: Build
>Reporter: Jacek Lewandowski
>Assignee: Jacek Lewandowski
>Priority: Low
>
> This is just a simple additional script which committers can use to help them
> merge commits, especially if the change applies to multiple Cassandra
> versions.
> For example, for CASSANDRA-18153, which applies to 3.0..trunk, it generates
> the following commands, which then can be run manually:
> {noformat}
> git fetch origin
> git fetch jacek-lewandowski
> # jacek-lewandowski/CASSANDRA-18153-3.0 -> cassandra-3.0
> #
>
> git switch cassandra-3.0
> git reset --hard origin/cassandra-3.0
> git cherry-pick e7e9b42559 # e7e9b42559 Save host id to system.local and
> flush immediately after startup
> git cherry-pick -n 99e96a4f95 && git commit -a --amend --no-edit # 99e96a4f95
> fixes
> git cherry-pick -n c63e3f29f1 && git commit -a --amend --no-edit # c63e3f29f1
> DO NOT MERGE - CircleCI config
> grep 'CASSANDRA-18153' CHANGES.txt || sed -E -i '/^[0-9]+\.[0-9]+/{s/.*/&\n\
> * Save host id to system.local and flush immediately after startup
> (CASSANDRA-18153)/;:a;n;ba}' CHANGES.txt
> git diff CHANGES.txt
> git add CHANGES.txt
> git commit --amend --no-edit
> (git diff origin/cassandra-3.0..HEAD -- .circleci/ | git apply -R --index) &&
> git commit -a --amend --no-edit # Remove all changes in .circleci directory
> if you need to
> git diff --name-only origin/cassandra-3.0..HEAD # print a list of all changes
> files
> # jacek-lewandowski/CASSANDRA-18153-3.11 -> cassandra-3.11
> #
>
> git switch cassandra-3.11
> git reset --hard origin/cassandra-3.11
> git merge -s ours --log --no-edit cassandra-3.0
> git cherry-pick -n 79e7b90c8f && git commit -a --amend --no-edit # 79e7b90c8f
> Save host id to system.local and flush immediately after startup
> git cherry-pick -n bdca4c384b && git commit -a --amend --no-edit # bdca4c384b
> fixes
> git cherry-pick -n 45078e6793 && git commit -a --amend --no-edit # 45078e6793
> DO NOT MERGE - CircleCI config
> grep 'CASSANDRA-18153' CHANGES.txt || sed -E -i '/^Merged from 3.0/{s/.*/&\n\
> * Save host id to system.local and flush immediately after startup
> (CASSANDRA-18153)/;:a;n;ba}' CHANGES.txt
> git diff CHANGES.txt
> git add CHANGES.txt
> git commit --amend --no-edit
> (git diff origin/cassandra-3.11..HEAD -- .circleci/ | git apply -R --index)
> && git commit -a --amend --no-edit # Remove all changes in .circleci
> directory if you need to
> git diff --name-only origin/cassandra-3.11..HEAD # print a list of all
> changes files
> # jacek-lewandowski/CASSANDRA-18153-4.0 -> cassandra-4.0
> #
>
> git switch cassandra-4.0
> git reset --hard origin/cassandra-4.0
> git merge -s ours --log --no-edit cassandra-3.11
> git cherry-pick -n 2227c5c7af && git commit -a --amend --no-edit # 2227c5c7af
> Save host id to system.local and flush immediately after startup
> git cherry-pick -n a71d4e3408 && git commit -a --amend --no-edit # a71d4e3408
> DO NOT MERGE - CircleCI config
> git cherry-pick -n 6dc53f4e97 && git commit -a --amend --no-edit # 6dc53f4e97
> fixes
> grep 'CASSANDRA-18153' CHANGES.txt || sed -E -i '/^Merged from 3.0/{s/.*/&\n\
> * Save host id to system.local and flush immediately after startup
> (CASSANDRA-18153)/;:a;n;ba}' CHANGES.txt
> git diff CHANGES.txt
> git add CHANGES.txt
> git commit --amend --no-edit
> (git diff origin/cassandra-4.0..HEAD -- .circleci/ | git apply -R --index) &&
> git commit -a --amend --no-edit # Remove all changes in .circleci directory
> if you need to
> git diff --name-only origin/cassandra-4.0..HEAD # print a list of all changes
> files
> # jacek-lewandowski/CASSANDRA-18153-4.1 -> cassandra-4.1
> #
>
> git switch cassandra-4.1
> git reset --hard origin/cassandra-4.1
> git merge -s ours --log --no-edit cassandra-4.0
> git cherry-pick -n 3584d17b36 && git commit -a --amend --no-edit # 3584d17b36
> Save host id to system.local and flush immediately after startup
> git cherry-pick -n
[jira] [Commented] (CASSANDRA-16895) Build with Java 17
[ https://issues.apache.org/jira/browse/CASSANDRA-16895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17707590#comment-17707590 ] Josh McKenzie commented on CASSANDRA-16895: --- bq. it is not a tiny work to officially support another release and build + green CI is probably not enough in my opinion I'm thinking of "support" in terms of 2 separate use-cases: 1) Supported for development: you can build and run it locally 2) Supported for production deployment: you can run it in prod If someone comes along and wants to bring the project up to supporting JDK19 *for development* for instance without destabilizing the other supported branches, my intuition is that that won't be too painful of a lift for us as a community to maintain if they contributed and disappeared. And that'd probably incrementally get us a bit of the way towards #2. I'm not personally viewing this through the lens of adding CI or any of the rest of it, beyond maybe adjusting the build jobs to make sure the config still builds. > Build with Java 17 > -- > > Key: CASSANDRA-16895 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16895 > Project: Cassandra > Issue Type: Epic > Components: Build >Reporter: Ekaterina Dimitrova >Assignee: Ekaterina Dimitrova >Priority: Normal > Fix For: 5.x > > > This ticket is intended to group all issues found to support Java 17 in the > future. > Upgrade steps: > * [Dependencies > |https://mvnrepository.com/artifact/org.apache.cassandra/cassandra-all/4.0.1]to > be updated (not all but at least those that require an update in order to > work with Java 17) > * More encapsulated JDK internal APIs. Some of the issues might be solved > with the dependencies updates > * Currently trunk compiles if we remove the Nashorn dependency (ant script > tag, used for the test environment; UDFs) . The oracle recommendation to use > Nashorn-core won't work for the project as it is under GPL 2.0. Most probably > we will opt in for graal-sdk licensed under UPL > * All tests to be cleaned > * CI environment to be setup > *NOTE:* GC tuning, performance testing were never agreed to be part of this > ticket. > Below is a snapshot of current CI failures with JDK17, it will be updated on > a regular basis with a date of update > *March 19th 2023* > || ||Failing Test Classes||Ticket Numbers|| > | |_Python DTests_| | > |1|test_sjk|CASSANDRA-18343| > | |_Java Ditributed Tests_| | > |1-6|org.apache.cassandra.distributed.test.ReprepareOldBehaviourTest - all > tests, > org.apache.cassandra.distributed.test.PrepareBatchStatementsTest - all tests, > org.apache.cassandra.distributed.test.IPMembershipTest - both tests, > org.apache.cassandra.distributed.test.MixedModeFuzzTest, > org.apache.cassandra.distributed.test.ReprepareFuzzTest, > org.apache.cassandra.distributed.test.ReprepareNewBehaviourTest|CASSANDRA-16304| > |7,8|org.apache.cassandra.distributed.test.NativeTransportEncryptionOptionsTest > - all tests > org.apache.cassandra.distributed.test.InternodeEncryptionOptionsTest - all > tests|Could be related to CASSANDRA-18180, to be checked | > |9|org.apache.cassandra.distributed.test.SSTableLoaderEncryptionOptionsTest - > 2 tests|CASSANDRA-18180| > | |_Unit Tests_| | > |1|org.apache.cassandra.repair.RepairJobTest - 1 test|CASSANDRA-17884| > |2|org.apache.cassandra.security.SSLFactoryTest - all tests|CASSANDRA-17992| > |3,4|org.apache.cassandra.db.memtable.MemtableSizeOffheapBuffersTest, > org.apache.cassandra.utils.concurrent.RefCountedTest|CASSANDRA-18329| > |5,6|org.apache.cassandra.cql3.validation.entities.UFJavaTest, > org.apache.cassandra.cql3.validation.entities.UFSecurityTest|CASSANDRA-18190| > | | | | > | | | | > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16895) Build with Java 17
[ https://issues.apache.org/jira/browse/CASSANDRA-16895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17707024#comment-17707024 ] Josh McKenzie commented on CASSANDRA-16895: --- Strong +1 to making it easy to work with non-LTS versions; also get the benefit of some of this work happening incrementally instead of the lift being as big as this 11-17 one has been. I'd track that in a different ticket thought. :) > Build with Java 17 > -- > > Key: CASSANDRA-16895 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16895 > Project: Cassandra > Issue Type: Epic > Components: Build >Reporter: Ekaterina Dimitrova >Assignee: Ekaterina Dimitrova >Priority: Normal > Fix For: 5.x > > > This ticket is intended to group all issues found to support Java 17 in the > future. > Upgrade steps: > * [Dependencies > |https://mvnrepository.com/artifact/org.apache.cassandra/cassandra-all/4.0.1]to > be updated (not all but at least those that require an update in order to > work with Java 17) > * More encapsulated JDK internal APIs. Some of the issues might be solved > with the dependencies updates > * Currently trunk compiles if we remove the Nashorn dependency (ant script > tag, used for the test environment; UDFs) . The oracle recommendation to use > Nashorn-core won't work for the project as it is under GPL 2.0. Most probably > we will opt in for graal-sdk licensed under UPL > * All tests to be cleaned > * CI environment to be setup > *NOTE:* GC tuning, performance testing were never agreed to be part of this > ticket. > Below is a snapshot of current CI failures with JDK17, it will be updated on > a regular basis with a date of update > *March 19th 2023* > || ||Failing Test Classes||Ticket Numbers|| > | |_Python DTests_| | > |1|test_sjk|CASSANDRA-18343| > | |_Java Ditributed Tests_| | > |1-6|org.apache.cassandra.distributed.test.ReprepareOldBehaviourTest - all > tests, > org.apache.cassandra.distributed.test.PrepareBatchStatementsTest - all tests, > org.apache.cassandra.distributed.test.IPMembershipTest - both tests, > org.apache.cassandra.distributed.test.MixedModeFuzzTest, > org.apache.cassandra.distributed.test.ReprepareFuzzTest, > org.apache.cassandra.distributed.test.ReprepareNewBehaviourTest|CASSANDRA-16304| > |7,8|org.apache.cassandra.distributed.test.NativeTransportEncryptionOptionsTest > - all tests > org.apache.cassandra.distributed.test.InternodeEncryptionOptionsTest - all > tests|Could be related to CASSANDRA-18180, to be checked | > |9|org.apache.cassandra.distributed.test.SSTableLoaderEncryptionOptionsTest - > 2 tests|CASSANDRA-18180| > | |_Unit Tests_| | > |1|org.apache.cassandra.repair.RepairJobTest - 1 test|CASSANDRA-17884| > |2|org.apache.cassandra.security.SSLFactoryTest - all tests|CASSANDRA-17992| > |3,4|org.apache.cassandra.db.memtable.MemtableSizeOffheapBuffersTest, > org.apache.cassandra.utils.concurrent.RefCountedTest|CASSANDRA-18329| > |5,6|org.apache.cassandra.cql3.validation.entities.UFJavaTest, > org.apache.cassandra.cql3.validation.entities.UFSecurityTest|CASSANDRA-18190| > | | | | > | | | | > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18390) Run Sonar analyzer over the Cassandra project
[
https://issues.apache.org/jira/browse/CASSANDRA-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17707020#comment-17707020
]
Josh McKenzie commented on CASSANDRA-18390:
---
Right. Hence the documentation angle; the sonarlint plugin should integrate
w/the sonarqube instance and pick up the rules there. Was a passing thought on
the "auto-magically do this as part of generating idea files", but likely
wouldn't work since it requires the plugin + likely auth.
> Run Sonar analyzer over the Cassandra project
> -
>
> Key: CASSANDRA-18390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18390
> Project: Cassandra
> Issue Type: Task
> Components: Build
>Reporter: Maxim Muzafarov
>Assignee: Maxim Muzafarov
>Priority: Normal
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As we already have Cassandra's project configured for the sonarcloud.io
> INFRA-24196, I wonder if we will be able to release branches, trunk, and pull
> requests to get analyzed by the SonarAnalyzer tool.
> Sonar is a code quality and security tool that is free to open-source
> projects and recommended by the INFRA team:
> https://cwiki.apache.org/confluence/display/INFRA/SonarCloud+for+ASF+projects
> It can have the following benefits without introducing any drawbacks (except
> for a few lines of source code)
> - visualise the LFH problems to work on;
> - see the trends in the source code;
> - add an extra layer of static code analysis;
> Changes below I have tested it locally with my SonarQube deployed on
> http://localhost:9000 and run the `act` for the GA part of the PR. It seems
> to work and parse classes correctly, but there are a few steps that need to
> be done by Cassandra's Committer or PMC (I do not have sufficient privileges):
> - Get the {{sonar.projectKey}} from the INFRA team;
> - make sure that the {{SONARCLOUD_TOKEN}} is available for GA and enabled for
> the project;
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-18393) Test failure: org.apache.cassandra.cql3.validation.operations.InsertUpdateIfConditionTest.testConditionalUpdate[1: clusterMinVersion=3.11]-compression.jdk1.8 on trun
Josh McKenzie created CASSANDRA-18393:
-
Summary: Test failure:
org.apache.cassandra.cql3.validation.operations.InsertUpdateIfConditionTest.testConditionalUpdate[1:
clusterMinVersion=3.11]-compression.jdk1.8 on trunk
Key: CASSANDRA-18393
URL: https://issues.apache.org/jira/browse/CASSANDRA-18393
Project: Cassandra
Issue Type: Bug
Components: Test/unit
Reporter: Josh McKenzie
Failed 1 times in the last 1 runs. Flakiness: 0%, Stability: 0%
{code}
Error Message
5.0.0-SNAPSHOT boolean:false
Stacktrace
junit.framework.AssertionFailedError: 5.0.0-SNAPSHOT boolean:false
at
org.apache.cassandra.cql3.validation.operations.InsertUpdateIfConditionTest.lambda$data$1(InsertUpdateIfConditionTest.java:70)
at
org.apache.cassandra.cql3.validation.operations.InsertUpdateIfConditionTest.beforeSetup(InsertUpdateIfConditionTest.java:95)
at
org.apache.cassandra.cql3.validation.operations.InsertUpdateIfConditionTest.before(InsertUpdateIfConditionTest.java:89)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
