[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837214#comment-17837214
]
Claude Warren commented on CASSANDRA-12937:
---
I think a few questions are in order:
# Is compression selection really part of the schema? My impression is that
the schema defines what goes into the tables, not how the tables are written
(e.g. different SSTable formats).
# is compaction selection part of the schema?
# do we allow nodes to have different compaction?
# If we allow different compaction on different nodes why would we not allow
different compression?
# If we have settings that must be the same across all the nodes do we have a
way to identify them in the configuration?
# if we can identify them, how do we guarantee that they are not different
across the nodes?
# If we can identify them, how do we set them?
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Stefan Miklosovic
>Priority: Low
> Labels: AdventCalendar2021
> Fix For: 5.x
>
> Time Spent: 8h
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-19502) PropertyFileSnitchTest overwrites test/conf/cassandra-topology.properties
[ https://issues.apache.org/jira/browse/CASSANDRA-19502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Claude Warren reassigned CASSANDRA-19502: - Assignee: Claude Warren > PropertyFileSnitchTest overwrites test/conf/cassandra-topology.properties > - > > Key: CASSANDRA-19502 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19502 > Project: Cassandra > Issue Type: Bug >Reporter: Claude Warren >Assignee: Claude Warren >Priority: Normal > Time Spent: 10m > Remaining Estimate: 0h > > While this is not an issue for CI test or other items that replace the file > on a development system the file is left in its modified state and under risk > of being accidentally checked into the repository. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-19502) PropertyFileSnitchTest overwrites test/conf/cassandra-topology.properties
Claude Warren created CASSANDRA-19502: - Summary: PropertyFileSnitchTest overwrites test/conf/cassandra-topology.properties Key: CASSANDRA-19502 URL: https://issues.apache.org/jira/browse/CASSANDRA-19502 Project: Cassandra Issue Type: Bug Reporter: Claude Warren While this is not an issue for CI test or other items that replace the file on a development system the file is left in its modified state and under risk of being accidentally checked into the repository. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18661) Update cassandra-stress to use Apache Commons CLI
[
https://issues.apache.org/jira/browse/CASSANDRA-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824635#comment-17824635
]
Claude Warren commented on CASSANDRA-18661:
---
The help for the new version looks like this:
{{{}{}}}{{{}Usage: cassandra-stress HELP {}}}{{[options] }}
{{---Commands---}}
{{ COUNTER_READ Multiple concurrent reads of counters. The cluster must
first be populated by a counterwrite test.}}
{{ COUNTER_WRITE Multiple concurrent updates of counters.}}
{{ HELP Prints complete help.}}
{{ MIXED Interleaving of any basic commands, with configurable
ratio and distribution - the cluster must first be}}
{{ populated by a write test}}
{{ PRINT Inspect the output of a distribution definition}}
{{ READ Multiple concurrent reads - the cluster must first be
populated by a write test}}
{{ USER Interleaving of user provided queries, with configurable
ratio and distribution}}
{{ VERSION Print the version of cassandra stress}}
{{ WRITE Multiple concurrent writes against the cluster}}
{{---Options---}}
{{ -?,help Prints help for the
current command.}}
{{ -auth-provider Fully qualified name of
an implementation of}}
{{
com.datastax.driver.core.AuthProvider}}
{{ -cl Consistency level to
use. Valid options are ANY, ONE, TWO, THREE, QUORUM,}}
{{ ALL, LOCAL_QUORUM,
EACH_QUORUM, SERIAL, LOCAL_SERIAL, LOCAL_ONE, NODE_LOCAL.}}
{{ (Default LOCAL_ONE)}}
{{ -col-comparator Column Comparator to
use. Only applicable if -col-names is specified. Valid}}
{{ values are:
TimeUUIDType, AsciiType, UTF8Type (Default AsciiType)}}
{{ -col-count Cell count distribution,
per operation (Default FIXED(5)). May not be used}}
{{ with -col-names.}}
{{ -col-names A comma separated list
of column names. May not be used with -column-count.}}
{{ -col-size Cell size distribution.
(Default FIXED(34))}}
{{ -col-slice If set, range slices
will be used for reads, otherwise a names query is used.}}
{{ -col-timestamp If set, all columns will
be written with the given timestamp.}}
{{ -command-add Distribution of value of
counter increments. (Default FIXED(1))}}
{{ -command-clustering Distribution clustering
runs of operations of the same kind. (Default}}
{{ GAUSSIAN(1..10))}}
{{ -command-keysize Key size in bytes.
(Default 10)}}
{{ -command-profile Specify the path to a
yaml cql3 profile. Multiple files can be added.}}
{{ -command-ratio Specify the ratios for
operations to perform. (see argument type notes below)}}
{{ -connections-per-host Number of connections
per host. Onluy valid with -simple-native. (Default}}
{{ value=8)}}
{{ -cql-style CQL connections style.
Valid options are: CQL, CQL_PREPARED}}
{{ -credential-file File is supposed to be a
standard property file with 'cql.username',}}
{{ 'cql.password',
'jmx.username', 'jmx.password',}}
{{
'transport.keystore.password', and 'transport.truststore.password' as keys.}}
{{ The values for these
keys will be overriden by their command-line}}
{{ counterparts when
specified.}}
{{ -datacenter Datacenter used for
DCAwareRoundRobinLoadPolicy}}
{{ -duration Time to run. Not valid
with -uncert-err or -n options.}}
{{ -error-ignore Do not fail on errors.}}
{{ -graph-file HTML file to create or
append to.}}
{{ -graph-name Alternative name for
current operation (Default: stress op name)}}
{{ -graph-revision Unique name to assign to
the current configuration being stressed.}}
{{ -graph-title Title for chart.
(Default: current date)}}
{{
[jira] [Commented] (CASSANDRA-18661) Update cassandra-stress to use Apache Commons CLI
[
https://issues.apache.org/jira/browse/CASSANDRA-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823670#comment-17823670
]
Claude Warren commented on CASSANDRA-18661:
---
[~bschoeni]
I have managed to get Stress to use the commons-cli code (after adding some
more functionality to commons-cli). So we will have to wait for commons-cli
1.7.0 to be released.
However, there is a requirement in the code for the StressSettings to be
serializable. Is this an old Thrift requirement and can it be removed? As I
recall serialization is fraught with security issues, though this is only a
test tool. I just don't see how it would be used in the tool. Any suggestions?
> Update cassandra-stress to use Apache Commons CLI
> -
>
> Key: CASSANDRA-18661
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18661
> Project: Cassandra
> Issue Type: Improvement
> Components: Tool/stress
>Reporter: Brad Schoening
>Assignee: Claude Warren
>Priority: Normal
> Labels: lhf
>
> The Apache Commons CLI library provides an API for parsing command line
> options with the package org.apache.commons.cli and this is already used by a
> dozen of existing Cassandra utilities including:
> {quote}SSTableMetadataViewer, StandaloneScrubber, StandaloneSplitter,
> SSTableExport, BulkLoader, and others.
> {quote}
> However, cassandra-stress is an outlier which uses its own custom classes to
> parse command line options with classes such as OptionsSimple. In addition,
> the options syntax for username, password, and others are not aligned with
> the format used by CQLSH.
> Currently, there are > 5K lines of code in 'settings' which appears to just
> process command line args.
> This suggestion is to:
>
> a) Upgrade cassandra-stress to use Apache Commons CLI (no new dependencies
> are required as this library is already used by the project)
>
> b) Align the cassandra-stress CLI options with those in CQLSH,
>
> {quote}For example, using the new syntax like CQLSH:
> {quote}
>
> cassandra-stress -username foo -password bar
> {quote}and replacing the old syntax:
> {quote}
> cassandra-stress -mode username=foo and password=bar
>
> This will simplify and unify the code base, eliminate code and reduce the
> confusion between similar named classes such as
> org.apache.cassandra.stress.settings.\{Option, OptionsMulti, OptionsSimple}
> and org.apache.commons.cli.{Option, OptionGroup, Options)
>
> Note: documentation will need to be updated as well
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18934) Downgrade to 4.1 fails due to schema changes
[
https://issues.apache.org/jira/browse/CASSANDRA-18934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823476#comment-17823476
]
Claude Warren commented on CASSANDRA-18934:
---
[~maxwellguo] I have been thinking about this ticket on and off for awhile now
and have some ideas that I would like to talk with you about. Would you reach
out to me directly [cla...@apache.org.|mailto:cla...@apache.org.] thank you.
> Downgrade to 4.1 fails due to schema changes
>
>
> Key: CASSANDRA-18934
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18934
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
>Reporter: David Capwell
>Assignee: Maxwell Guo
>Priority: Normal
> Fix For: 5.x
>
>
> We are required to support 5.0 downgrading to 4.1 as a migration step, but we
> don’t have tests to show this is working… I wrote a quick test to make sure a
> change we needed in Accord wouldn’t block the downgrade and see that we fail
> right now.
> {code}
> ERROR 20:56:39 Exiting due to error while processing commit log during
> initialization.
> org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException:
> Unexpected error deserializing mutation; saved to
> /var/folders/h1/s_3p1x3s3hl0hltbpck67m0hgn/T/mutation418421767150092dat.
> This may be caused by replaying a mutation against a table with the same
> name but incompatible schema. Exception follows: java.lang.RuntimeException:
> Unknown column compaction_properties during deserialization
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readMutation(CommitLogReader.java:464)
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:397)
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:244)
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147)
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:191)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:223)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:204)
> {code}
> This was caused by a schema change in CASSANDRA-18061
> {code}
> /*
> * Licensed to the Apache Software Foundation (ASF) under one
> * or more contributor license agreements. See the NOTICE file
> * distributed with this work for additional information
> * regarding copyright ownership. The ASF licenses this file
> * to you under the Apache License, Version 2.0 (the
> * "License"); you may not use this file except in compliance
> * with the License. You may obtain a copy of the License at
> *
> * http://www.apache.org/licenses/LICENSE-2.0
> *
> * Unless required by applicable law or agreed to in writing, software
> * distributed under the License is distributed on an "AS IS" BASIS,
> * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> * See the License for the specific language governing permissions and
> * limitations under the License.
> */
> package org.apache.cassandra.distributed.upgrade;
> import java.io.IOException;
> import java.io.File;
> import java.util.concurrent.atomic.AtomicBoolean;
> import org.junit.Test;
> import org.apache.cassandra.distributed.api.IUpgradeableInstance;
> public class DowngradeTest extends UpgradeTestBase
> {
> @Test
> public void test() throws Throwable
> {
> AtomicBoolean first = new AtomicBoolean(true);
> new TestCase()
> .nodes(1)
> .withConfig(c -> {
> if (first.compareAndSet(true, false))
> c.set("storage_compatibility_mode", "CASSANDRA_4");
> })
> .downgradeTo(v41)
> .setup(cluster -> {})
> // Uncomment if you want to test what happens after reading the commit log,
> which fails right now
> //.runBeforeNodeRestart((cluster, nodeId) -> {
> //IUpgradeableInstance inst = cluster.get(nodeId);
> //File f = new File((String)
> inst.config().get("commitlog_directory"));
> //deleteRecursive(f);
> //})
> .runAfterClusterUpgrade(cluster -> {})
> .run();
> }
> private void deleteRecursive(File f)
> {
> if (f.isDirectory())
> {
> File[] children = f.listFiles();
> if (children != null)
> {
> for (File c : children)
> deleteRecursive(c);
> }
> }
> f.delete();
> }
> }
> {code}
> {code}
> diff --git
> a/test/distributed/org/apache/cassandra/distributed/upgrade/UpgradeTestBase.java
>
>
[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823463#comment-17823463
]
Claude Warren commented on CASSANDRA-12937:
---
[~smiklosovic] , we have finished 16565, Can we look at moving this forward?
The pull request I placed against your branch changes the exclusion code so
that it is all in one place. calling for the default compression will properly
exclude system keyspaces, and gives us a single point in which to fix any other
issues that arise.
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Stefan Miklosovic
>Priority: Low
> Labels: AdventCalendar2021
> Fix For: 5.x
>
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18934) Downgrade to 4.1 fails due to schema changes
[
https://issues.apache.org/jira/browse/CASSANDRA-18934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811265#comment-17811265
]
Claude Warren commented on CASSANDRA-18934:
---
I don't think we can modify 4.x code for this purpose. I encountered the same
issue in the 4->3 downgrade. My solution was to rewrite the system tables that
need modification. System tables will need modification when columns have been
added or removed, or when column data types change. Non-system tables are
not affected because we can specify column addition/deletion/modification in
the schema table(s) to address the issues. But as Guo discovered the system
tables are hard coded in each version.
In the solution I arrived at I noted the column changes and then rewrote the
necessary tables during the downgrade process. This means that we have to
track, by hand, all the system table changes between version. To simplify
this, my original solution wrote the earliest form of the V3 table and let the
"we can read earlier versions" processes handle changing it to the proper
state. I cleaner solution might be to have the jars from the earlier version
available so that the prior classes can be loaded and interrogated to see if
there are changes and stop if there are any changes that the system can not
deal with.
> Downgrade to 4.1 fails due to schema changes
>
>
> Key: CASSANDRA-18934
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18934
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
>Reporter: David Capwell
>Assignee: Maxwell Guo
>Priority: Normal
> Fix For: 5.x
>
>
> We are required to support 5.0 downgrading to 4.1 as a migration step, but we
> don’t have tests to show this is working… I wrote a quick test to make sure a
> change we needed in Accord wouldn’t block the downgrade and see that we fail
> right now.
> {code}
> ERROR 20:56:39 Exiting due to error while processing commit log during
> initialization.
> org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException:
> Unexpected error deserializing mutation; saved to
> /var/folders/h1/s_3p1x3s3hl0hltbpck67m0hgn/T/mutation418421767150092dat.
> This may be caused by replaying a mutation against a table with the same
> name but incompatible schema. Exception follows: java.lang.RuntimeException:
> Unknown column compaction_properties during deserialization
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readMutation(CommitLogReader.java:464)
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:397)
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:244)
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147)
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:191)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:223)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:204)
> {code}
> This was caused by a schema change in CASSANDRA-18061
> {code}
> /*
> * Licensed to the Apache Software Foundation (ASF) under one
> * or more contributor license agreements. See the NOTICE file
> * distributed with this work for additional information
> * regarding copyright ownership. The ASF licenses this file
> * to you under the Apache License, Version 2.0 (the
> * "License"); you may not use this file except in compliance
> * with the License. You may obtain a copy of the License at
> *
> * http://www.apache.org/licenses/LICENSE-2.0
> *
> * Unless required by applicable law or agreed to in writing, software
> * distributed under the License is distributed on an "AS IS" BASIS,
> * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> * See the License for the specific language governing permissions and
> * limitations under the License.
> */
> package org.apache.cassandra.distributed.upgrade;
> import java.io.IOException;
> import java.io.File;
> import java.util.concurrent.atomic.AtomicBoolean;
> import org.junit.Test;
> import org.apache.cassandra.distributed.api.IUpgradeableInstance;
> public class DowngradeTest extends UpgradeTestBase
> {
> @Test
> public void test() throws Throwable
> {
> AtomicBoolean first = new AtomicBoolean(true);
> new TestCase()
> .nodes(1)
> .withConfig(c -> {
> if (first.compareAndSet(true, false))
> c.set("storage_compatibility_mode", "CASSANDRA_4");
> })
> .downgradeTo(v41)
> .setup(cluster -> {})
> // Uncomment
[jira] [Updated] (CASSANDRA-19264) Please delete -- opened in wrong project
[ https://issues.apache.org/jira/browse/CASSANDRA-19264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Claude Warren updated CASSANDRA-19264: -- Priority: Low (was: Normal) > Please delete -- opened in wrong project > > > Key: CASSANDRA-19264 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19264 > Project: Cassandra > Issue Type: New Feature >Reporter: Claude Warren >Assignee: Claude Warren >Priority: Low > > Issue opened in wrong project. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-19264) Please delete -- opened in wrong project
[ https://issues.apache.org/jira/browse/CASSANDRA-19264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Claude Warren reassigned CASSANDRA-19264: - Assignee: (was: Claude Warren) > Please delete -- opened in wrong project > > > Key: CASSANDRA-19264 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19264 > Project: Cassandra > Issue Type: New Feature >Reporter: Claude Warren >Priority: Low > > Issue opened in wrong project. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19264) Please delete -- opened in wrong project
[ https://issues.apache.org/jira/browse/CASSANDRA-19264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Claude Warren updated CASSANDRA-19264: -- Description: Issue opened in wrong project. (was: [~claude], Could you please raise WARN in case user defined License family has the same name as internal one? It seems user defined one is `just ignored` in such case :(() > Please delete -- opened in wrong project > > > Key: CASSANDRA-19264 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19264 > Project: Cassandra > Issue Type: New Feature >Reporter: Claude Warren >Assignee: Claude Warren >Priority: Normal > > Issue opened in wrong project. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-19264) Please delete -- opened in wrong project
[ https://issues.apache.org/jira/browse/CASSANDRA-19264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Claude Warren updated CASSANDRA-19264: -- Summary: Please delete -- opened in wrong project (was: WARN in case user defined License family has the same id as internal one) > Please delete -- opened in wrong project > > > Key: CASSANDRA-19264 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19264 > Project: Cassandra > Issue Type: New Feature >Reporter: Claude Warren >Assignee: Claude Warren >Priority: Normal > > [~claude], Could you please raise WARN in case user defined License family > has the same name as internal one? > It seems user defined one is `just ignored` in such case :(( -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-19264) WARN in case user defined License family has the same id as internal one
Claude Warren created CASSANDRA-19264: - Summary: WARN in case user defined License family has the same id as internal one Key: CASSANDRA-19264 URL: https://issues.apache.org/jira/browse/CASSANDRA-19264 Project: Cassandra Issue Type: New Feature Reporter: Claude Warren Assignee: Claude Warren [~claude], Could you please raise WARN in case user defined License family has the same name as internal one? It seems user defined one is `just ignored` in such case :(( -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17804659#comment-17804659
]
Claude Warren commented on CASSANDRA-12937:
---
Where are we with this change? I would like to see it move forward.
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Stefan Miklosovic
>Priority: Low
> Labels: AdventCalendar2021
> Fix For: 5.x
>
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16565) Remove dependency on sigar
[ https://issues.apache.org/jira/browse/CASSANDRA-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17800737#comment-17800737 ] Claude Warren commented on CASSANDRA-16565: --- [https://github.com/apache/cassandra/pull/2842] contains the latest code and should be the basis for any reviews at this point. Thanks to [~smiklosovic] for helping to drive this forward. > Remove dependency on sigar > -- > > Key: CASSANDRA-16565 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16565 > Project: Cassandra > Issue Type: Improvement > Components: Build >Reporter: David Capwell >Assignee: Claude Warren >Priority: Normal > Fix For: 5.x > > > sigar is used to check if the environment has good settings for running C*, > but requires we bundle a lot of native libraries to perform this check (which > can also be done elsewhere). This project also appears to be dead as the > last commit was around 6 years ago. > With the move to resolve artifacts rather than commit them, removing this > dependency would remove majority of the artifacts fetched from GitHub. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18661) Update cassandra-stress to use Apache Commons CLI
[
https://issues.apache.org/jira/browse/CASSANDRA-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799433#comment-17799433
]
Claude Warren commented on CASSANDRA-18661:
---
We could change the sub-option arguments by prefixing them with what is now a
"sub-option"
For example, this would change the example in [1] from:
{quote}{{#Load one row with default schema}}
{{$ cassandra-stress write n=1 cl=one -mode native cql3 -log
file=create_schema.log}}{quote}
{quote}{{#Modify schema in CQL}}
{{$ cqlsh }}{quote}
{quote}{{#Run a real write workload}}
{{$ cassandra-stress write n=100 cl=one -mode native cql3 -schema
keyspace="keyspace1" -log file=load_1M_rows.log}}{quote}
To:
{quote}{{#Load one row with default schema}}
{{$ cassandra-stress write n=1 cl=one -mode-native -mode-cql3 -log-file
create_schema.log}}{quote}
{quote}{{#Modify schema in CQL}}
{{$ cqlsh}}{quote}
{quote}{{#Run a real write workload}}
{{$ cassandra-stress write n=100 cl=one -mode-native -mode-cql3
-schema-keyspace "keyspace1" -log-file load_1M_rows.log}}{quote}
We could also change the "additional stress parameters to options which would
then look like:
{quote}{{#Load one row with default schema}}
{{$ cassandra-stress write -n 1 -cl one -mode-native -mode-cql3 -log-file
create_schema.log}}{quote}
{quote}{{#Modify schema in CQL}}
{{$ cqlsh }}{quote}
{quote}{{#Run a real write workload}}
{{$ cassandra-stress write -n 100 -cl one -mode-native -mode-cql3
-schema-keyspace "keyspace1" -log-file load_1M_rows.log}}{quote}
This is a significant breaking change, though it might be possible to build an
upgrade tool to convert old command lines to new ones.
[1]
https://docs.datastax.com/en/dse/5.1/docs/tooling/cassandra-stress-tool.html
> Update cassandra-stress to use Apache Commons CLI
> -
>
> Key: CASSANDRA-18661
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18661
> Project: Cassandra
> Issue Type: Improvement
> Components: Tool/stress
>Reporter: Brad Schoening
>Assignee: Claude Warren
>Priority: Normal
> Labels: lhf
>
> The Apache Commons CLI library provides an API for parsing command line
> options with the package org.apache.commons.cli and this is already used by a
> dozen of existing Cassandra utilities including:
> {quote}SSTableMetadataViewer, StandaloneScrubber, StandaloneSplitter,
> SSTableExport, BulkLoader, and others.
> {quote}
> However, cassandra-stress is an outlier which uses its own custom classes to
> parse command line options with classes such as OptionsSimple. In addition,
> the options syntax for username, password, and others are not aligned with
> the format used by CQLSH.
> Currently, there are > 5K lines of code in 'settings' which appears to just
> process command line args.
> This suggestion is to:
>
> a) Upgrade cassandra-stress to use Apache Commons CLI (no new dependencies
> are required as this library is already used by the project)
>
> b) Align the cassandra-stress CLI options with those in CQLSH,
>
> {quote}For example, using the new syntax like CQLSH:
> {quote}
>
> cassandra-stress -username foo -password bar
> {quote}and replacing the old syntax:
> {quote}
> cassandra-stress -mode username=foo and password=bar
>
> This will simplify and unify the code base, eliminate code and reduce the
> confusion between similar named classes such as
> org.apache.cassandra.stress.settings.\{Option, OptionsMulti, OptionsSimple}
> and org.apache.commons.cli.{Option, OptionGroup, Options)
>
> Note: documentation will need to be updated as well
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18661) Update cassandra-stress to use Apache Commons CLI
[
https://issues.apache.org/jira/browse/CASSANDRA-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798930#comment-17798930
]
Claude Warren commented on CASSANDRA-18661:
---
I spent some time this morning looking at this. I can not find a way to
process options without a leading dash. There are comments in the mailing list
about how options require such prefixes to distinguish them from arguments to
the options.
I looked at adding dashes to the existing sub-options but this all gets very
complicated very quickly. For example there is not a way to support 'n<',
'n>', and 'n=' in the same set of options. if we preprocess the line and add
'-D' (or something similar) to the front so we have something like '-Dn<5' that
will probably work. However we are then replacing the custom code with new
custom code to make it fit the CLI code.
The other option is to take the arguments from the sub-options and process them
as we do now. In this case we are only using the CLI to process the
sub-options.
We could build a system that takes the arguments for the sub-options, splits
the sub options so that items like "n<5" become \{"n", "<5"} and then process
the result with an iterator where we lookup the argument in a table and pass
the iterator to a method that then handles the configuration for that argument
before returning and allowing the next argument to be pulled from the iterator.
In this way we can process arguments that take arguments (e.g. err<0.5 or
n>6). But again we are back to lots of custom processing. However we can
switch to using Predicate> and perhaps simplify some of the
system while making it easier to extend with new options later.
I am on the fence with this. I am willing to take a look to see if we can
simplify and/or update, but I don't know if there is enough support to change
this when we will still have a fair amount of custom code. I'll continue to
explore for the rest of today and perhaps tomorrow. But I would appreciate any
guidance with respect to whether or not this would be accepted if the change to
CLI provides minimal change to the processing.
> Update cassandra-stress to use Apache Commons CLI
> -
>
> Key: CASSANDRA-18661
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18661
> Project: Cassandra
> Issue Type: Improvement
> Components: Tool/stress
>Reporter: Brad Schoening
>Assignee: Claude Warren
>Priority: Normal
> Labels: lhf
>
> The Apache Commons CLI library provides an API for parsing command line
> options with the package org.apache.commons.cli and this is already used by a
> dozen of existing Cassandra utilities including:
> {quote}SSTableMetadataViewer, StandaloneScrubber, StandaloneSplitter,
> SSTableExport, BulkLoader, and others.
> {quote}
> However, cassandra-stress is an outlier which uses its own custom classes to
> parse command line options with classes such as OptionsSimple. In addition,
> the options syntax for username, password, and others are not aligned with
> the format used by CQLSH.
> Currently, there are > 5K lines of code in 'settings' which appears to just
> process command line args.
> This suggestion is to:
>
> a) Upgrade cassandra-stress to use Apache Commons CLI (no new dependencies
> are required as this library is already used by the project)
>
> b) Align the cassandra-stress CLI options with those in CQLSH,
>
> {quote}For example, using the new syntax like CQLSH:
> {quote}
>
> cassandra-stress -username foo -password bar
> {quote}and replacing the old syntax:
> {quote}
> cassandra-stress -mode username=foo and password=bar
>
> This will simplify and unify the code base, eliminate code and reduce the
> confusion between similar named classes such as
> org.apache.cassandra.stress.settings.\{Option, OptionsMulti, OptionsSimple}
> and org.apache.commons.cli.{Option, OptionGroup, Options)
>
> Note: documentation will need to be updated as well
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18661) Update cassandra-stress to use Apache Commons CLI
[
https://issues.apache.org/jira/browse/CASSANDRA-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798518#comment-17798518
]
Claude Warren commented on CASSANDRA-18661:
---
Using Apache Commons CLI is not going to remove much code.
Apache Commons requires that options be prefixed with a '-' or 2 dashes. So it
will only parse the sub-options. It will create a list of the arguments to
each sub-option but those will have to be parsed using a different tool as they
are really positional arguments and the Apache CLI does not seem to handle
those. This means that all the code currently in place for parsing the
sub-options will need to remain or be rewritten or we will need to find a
positional command line parser to read them. And then the presence of the '?'
in some of the common options (i.e. n Update cassandra-stress to use Apache Commons CLI
> -
>
> Key: CASSANDRA-18661
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18661
> Project: Cassandra
> Issue Type: Improvement
> Components: Tool/stress
>Reporter: Brad Schoening
>Assignee: Claude Warren
>Priority: Normal
> Labels: lhf
>
> The Apache Commons CLI library provides an API for parsing command line
> options with the package org.apache.commons.cli and this is already used by a
> dozen of existing Cassandra utilities including:
> {quote}SSTableMetadataViewer, StandaloneScrubber, StandaloneSplitter,
> SSTableExport, BulkLoader, and others.
> {quote}
> However, cassandra-stress is an outlier which uses its own custom classes to
> parse command line options with classes such as OptionsSimple. In addition,
> the options syntax for username, password, and others are not aligned with
> the format used by CQLSH.
> Currently, there are > 5K lines of code in 'settings' which appears to just
> process command line args.
> This suggestion is to:
>
> a) Upgrade cassandra-stress to use Apache Commons CLI (no new dependencies
> are required as this library is already used by the project)
>
> b) Align the cassandra-stress CLI options with those in CQLSH,
>
> {quote}For example, using the new syntax like CQLSH:
> {quote}
>
> cassandra-stress -username foo -password bar
> {quote}and replacing the old syntax:
> {quote}
> cassandra-stress -mode username=foo and password=bar
>
> This will simplify and unify the code base, eliminate code and reduce the
> confusion between similar named classes such as
> org.apache.cassandra.stress.settings.\{Option, OptionsMulti, OptionsSimple}
> and org.apache.commons.cli.{Option, OptionGroup, Options)
>
> Note: documentation will need to be updated as well
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-18661) Update cassandra-stress to use Apache Commons CLI
[
https://issues.apache.org/jira/browse/CASSANDRA-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claude Warren reassigned CASSANDRA-18661:
-
Assignee: Claude Warren
> Update cassandra-stress to use Apache Commons CLI
> -
>
> Key: CASSANDRA-18661
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18661
> Project: Cassandra
> Issue Type: Improvement
> Components: Tool/stress
>Reporter: Brad Schoening
>Assignee: Claude Warren
>Priority: Normal
> Labels: lhf
>
> The Apache Commons CLI library provides an API for parsing command line
> options with the package org.apache.commons.cli and this is already used by a
> dozen of existing Cassandra utilities including:
> {quote}SSTableMetadataViewer, StandaloneScrubber, StandaloneSplitter,
> SSTableExport, BulkLoader, and others.
> {quote}
> However, cassandra-stress is an outlier which uses its own custom classes to
> parse command line options with classes such as OptionsSimple. In addition,
> the options syntax for username, password, and others are not aligned with
> the format used by CQLSH.
> Currently, there are > 5K lines of code in 'settings' which appears to just
> process command line args.
> This suggestion is to:
>
> a) Upgrade cassandra-stress to use Apache Commons CLI (no new dependencies
> are required as this library is already used by the project)
>
> b) Align the cassandra-stress CLI options with those in CQLSH,
>
> {quote}For example, using the new syntax like CQLSH:
> {quote}
>
> cassandra-stress -username foo -password bar
> {quote}and replacing the old syntax:
> {quote}
> cassandra-stress -mode username=foo and password=bar
>
> This will simplify and unify the code base, eliminate code and reduce the
> confusion between similar named classes such as
> org.apache.cassandra.stress.settings.\{Option, OptionsMulti, OptionsSimple}
> and org.apache.commons.cli.{Option, OptionGroup, Options)
>
> Note: documentation will need to be updated as well
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13855) URL Seed provider
[ https://issues.apache.org/jira/browse/CASSANDRA-13855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17796580#comment-17796580 ] Claude Warren commented on CASSANDRA-13855: --- The original code simply created a URL, opened it, and read lines from it. This implementation does the same. The changes are those to have it return an InetAddressAndPort. It should work for any URL that has a protocol handler. There is no effort made here to limit the use of the protocols. While some of the earlier comments reference HTTP the original code is not limited to HTTP. I don't think there is a reason to restrict the URL to an HTTP protocol. This soluiton is a dead simple seed provider that can fetch from a URL, just as stated in the description. > URL Seed provider > - > > Key: CASSANDRA-13855 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13855 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Coordination, Legacy/Core >Reporter: Jon Haddad >Assignee: Claude Warren >Priority: Low > Labels: lhf > Fix For: 5.x > > Attachments: 0001-Add-URL-Seed-Provider-trunk.txt, signature.asc, > signature.asc, signature.asc > > Time Spent: 10m > Remaining Estimate: 0h > > Seems like including a dead simple seed provider that can fetch from a URL, 1 > line per seed, would be useful. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13855) URL Seed provider
[ https://issues.apache.org/jira/browse/CASSANDRA-13855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17796290#comment-17796290 ] Claude Warren commented on CASSANDRA-13855: --- What were you asking for in your comment from yesterday? I wouldnt' say that using a File URL is "faking"it. It is an actual URL and it actually works. > URL Seed provider > - > > Key: CASSANDRA-13855 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13855 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Coordination, Legacy/Core >Reporter: Jon Haddad >Assignee: Claude Warren >Priority: Low > Labels: lhf > Fix For: 5.x > > Attachments: 0001-Add-URL-Seed-Provider-trunk.txt, signature.asc > > Time Spent: 10m > Remaining Estimate: 0h > > Seems like including a dead simple seed provider that can fetch from a URL, 1 > line per seed, would be useful. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13855) URL Seed provider
[ https://issues.apache.org/jira/browse/CASSANDRA-13855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17796048#comment-17796048 ] Claude Warren commented on CASSANDRA-13855: --- Those examples only deal with HTTP/HTTPS there is no support for other protocols. The requirement here was a URL (hence the test code uses a File URL to a temp file for testing). I see that there is a requirement to add headers to HTTP urls in some environments. But I am unsure where the information for those headers would even come from for this solution. I assume that one does not put the values in the yaml file. Do you know where to find the information for the headers? > URL Seed provider > - > > Key: CASSANDRA-13855 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13855 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Coordination, Legacy/Core >Reporter: Jon Haddad >Assignee: Claude Warren >Priority: Low > Labels: lhf > Fix For: 5.x > > Attachments: 0001-Add-URL-Seed-Provider-trunk.txt > > Time Spent: 10m > Remaining Estimate: 0h > > Seems like including a dead simple seed provider that can fetch from a URL, 1 > line per seed, would be useful. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13855) URL Seed provider
[ https://issues.apache.org/jira/browse/CASSANDRA-13855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795786#comment-17795786 ] Claude Warren commented on CASSANDRA-13855: --- Opened [https://github.com/apache/cassandra/pull/2988] as a fix. > URL Seed provider > - > > Key: CASSANDRA-13855 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13855 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Coordination, Legacy/Core >Reporter: Jon Haddad >Assignee: Claude Warren >Priority: Low > Labels: lhf > Fix For: 5.x > > Attachments: 0001-Add-URL-Seed-Provider-trunk.txt > > Time Spent: 10m > Remaining Estimate: 0h > > Seems like including a dead simple seed provider that can fetch from a URL, 1 > line per seed, would be useful. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-13855) URL Seed provider
[ https://issues.apache.org/jira/browse/CASSANDRA-13855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Claude Warren reassigned CASSANDRA-13855: - Assignee: Claude Warren > URL Seed provider > - > > Key: CASSANDRA-13855 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13855 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Coordination, Legacy/Core >Reporter: Jon Haddad >Assignee: Claude Warren >Priority: Low > Labels: lhf > Fix For: 5.x > > Attachments: 0001-Add-URL-Seed-Provider-trunk.txt > > Time Spent: 10m > Remaining Estimate: 0h > > Seems like including a dead simple seed provider that can fetch from a URL, 1 > line per seed, would be useful. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13855) URL Seed provider
[ https://issues.apache.org/jira/browse/CASSANDRA-13855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795381#comment-17795381 ] Claude Warren commented on CASSANDRA-13855: --- [~paulo] [~rustyrazorblade] [~denniskline] is there still a desire to see the URL Seed provider? If so, and if no one objects, I'll pick this up and make the suggested changes. > URL Seed provider > - > > Key: CASSANDRA-13855 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13855 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Coordination, Legacy/Core >Reporter: Jon Haddad >Priority: Low > Labels: lhf > Fix For: 5.x > > Attachments: 0001-Add-URL-Seed-Provider-trunk.txt > > > Seems like including a dead simple seed provider that can fetch from a URL, 1 > line per seed, would be useful. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18969) Pre-release required legal changes for cassandra-java-driver
[ https://issues.apache.org/jira/browse/CASSANDRA-18969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17783196#comment-17783196 ] Claude Warren commented on CASSANDRA-18969: --- New question: ci/install-jdk.sh:# (C) 2018 Christian Stein Since the install-jdk.sh is in the CI code do we list the copyright in the Notice file or not? > Pre-release required legal changes for cassandra-java-driver > > > Key: CASSANDRA-18969 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18969 > Project: Cassandra > Issue Type: Sub-task > Components: Build >Reporter: Michael Semb Wever >Priority: Urgent > Time Spent: 3.5h > Remaining Estimate: 0h > > TODO > * there is still code copyrighted to others that isn't mentioned in > NOTICE.txt (e.g. ci/install-jdk.sh ) > * "© DataStax" still appears in the README > * -spotbugs is LGPL, in 4.x, and not correctly made provided scope- > * add any BSD/MIT deps to LICENCE > * ensure copyright headers are on all files, use rat plugin > These things must be done before a release. > Cat X deps we included as build tool by using scope provided we should put a > test in to ensure they work at runtime without the dep doesn't regress in the > future. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18969) Pre-release required legal changes for cassandra-java-driver
[ https://issues.apache.org/jira/browse/CASSANDRA-18969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17782553#comment-17782553 ] Claude Warren commented on CASSANDRA-18969: --- Question 4.1.0_fixes NOTICE file lists a Wayne Meissner copyright -- but there is no code in the codebase that has his copyright in it. The text in the Notice file reads JNR project Copyright (C) 2008-2010 Wayne Meissner This product includes software developed as part of the JNR project ([https://github.com/jnr/jnr-ffi] ). see core/src/main/java/com/datastax/oss/driver/internal/core/os/CpuInfo.java The CpuInfo.java file does not exist in the project. However there is adependency on the jrn-ffi jars. Should the Wayne Messiner entry be removed from the NOTICE? I think so. > Pre-release required legal changes for cassandra-java-driver > > > Key: CASSANDRA-18969 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18969 > Project: Cassandra > Issue Type: Sub-task > Components: Build >Reporter: Michael Semb Wever >Priority: Urgent > Time Spent: 3h 10m > Remaining Estimate: 0h > > TODO > * there is still code copyrighted to others that isn't mentioned in > NOTICE.txt (e.g. ci/install-jdk.sh ) > * "© DataStax" still appears in the README > * spotbugs is LGPL, in 4.x, and not correctly made provided scope > * add any BSD/MIT deps to LICENCE > * ensure copyright headers are on all files, use rat plugin > These things must be done before a release. > Cat X deps we included as build tool by using scope provided we should put a > test in to ensure they work at runtime without the dep doesn't regress in the > future. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18339) Cassandra 3.11.2 windows remote share as data files directory
[ https://issues.apache.org/jira/browse/CASSANDRA-18339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781351#comment-17781351 ] Claude Warren commented on CASSANDRA-18339: --- Navya, I think the answer is it probably isn't supported and if it doesn't work there will not be work to support it, unless you want to find a way to make it work. However, I must concur with my cohorts here that if you can get it to work performance will probably be abysmal. May we close this issue? Are you satisfied with the response? > Cassandra 3.11.2 windows remote share as data files directory > -- > > Key: CASSANDRA-18339 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18339 > Project: Cassandra > Issue Type: Bug >Reporter: Navya Krishna Dubbala >Priority: Urgent > > Hi Team, > > We are using windows 2018 server for running Cassandra 3.11.2 version with > remote windows server shared folder as data file directory path and we are > getting below error. > > ERROR [CompactionExecutor:1] 2023-03-10 09:18:44,511 > DefaultFSErrorHandler.java:92 - Exiting forcefully due to file system > exception on startup, disk failure policy "stop" > org.apache.cassandra.io.FSReadError: java.io.IOException: There are no more > files at > org.apache.cassandra.io.util.FileUtils.getCanonicalPath(FileUtils.java:303) > It is working inbetween. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18301) Let the user select the sstable version to write
[ https://issues.apache.org/jira/browse/CASSANDRA-18301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781010#comment-17781010 ] Claude Warren commented on CASSANDRA-18301: --- I think there are 2 separate issues here: the sstables format and the schema format. When I was working on CASSANDRA-8928 (downgrade sstables) the easy part was to write the sstable. My strategy was if downgrading from 4 to 3 write the earliest v3 sstable format that was still supported. Then the v3 servers could apply changes as they would normally for an upgrade. The hard part was the schema format. Between 3 and 4 several columns were added to system tables and an extra field added to compaction info (I think). Now it seems from CASSANDRA-18934 that there is a similar issue between 4 and 5. So I think we need to be very clear if we are talking about sstables downgrades or schema table downgrades. I am not certain I see the benefit of downgrading sstables without accounting for schema table definitions as well. > Let the user select the sstable version to write > > > Key: CASSANDRA-18301 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18301 > Project: Cassandra > Issue Type: New Feature > Components: Local/Config, Local/SSTable >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16565) Remove dependency on sigar
[ https://issues.apache.org/jira/browse/CASSANDRA-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779017#comment-17779017 ] Claude Warren edited comment on CASSANDRA-16565 at 10/25/23 11:49 AM: -- The OSHI library does not have a method to retrieve the maximum processes. The maximum process request is equivalent to "ulimit -u". The OSHI library also returns 64 bit longs rather thant 32 bit values as sigar does. This results in some differences in the tests. I have extracted the sigar code into an evaluation package and coded an OSHI replacement to verify that there are no differences in the results. Evaluation code for the differences between Sigar and OSHI can be found at: https://github.com/Aiven-Labs/compare_oshi_sigar was (Author: claudenw): The OSHI library does not have a method to retrieve the maximum processes. The maximum process request is equivalent to "ulimit -u". The OSHI library also returns 64 bit longs rather thant 32 bit values as sigar does. This results in some differences in the tests. I have extracted the sigar code into an evaluation package and coded an OSHI replacement to verify that there are no differences in the results. > Remove dependency on sigar > -- > > Key: CASSANDRA-16565 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16565 > Project: Cassandra > Issue Type: Improvement > Components: Build >Reporter: David Capwell >Assignee: Claude Warren >Priority: Normal > Fix For: 5.x > > > sigar is used to check if the environment has good settings for running C*, > but requires we bundle a lot of native libraries to perform this check (which > can also be done elsewhere). This project also appears to be dead as the > last commit was around 6 years ago. > With the move to resolve artifacts rather than commit them, removing this > dependency would remove majority of the artifacts fetched from GitHub. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16565) Remove dependency on sigar
[ https://issues.apache.org/jira/browse/CASSANDRA-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779479#comment-17779479 ] Claude Warren commented on CASSANDRA-16565: --- There are several checks. That is the only one not supported by OSHI > Remove dependency on sigar > -- > > Key: CASSANDRA-16565 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16565 > Project: Cassandra > Issue Type: Improvement > Components: Build >Reporter: David Capwell >Assignee: Claude Warren >Priority: Normal > Fix For: 5.x > > > sigar is used to check if the environment has good settings for running C*, > but requires we bundle a lot of native libraries to perform this check (which > can also be done elsewhere). This project also appears to be dead as the > last commit was around 6 years ago. > With the move to resolve artifacts rather than commit them, removing this > dependency would remove majority of the artifacts fetched from GitHub. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16565) Remove dependency on sigar
[ https://issues.apache.org/jira/browse/CASSANDRA-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779464#comment-17779464 ] Claude Warren commented on CASSANDRA-16565: --- I have submitted a pull request https://github.com/apache/cassandra/pull/2842 that achieves the following: Removed Sigar references from the build scripts and library. Removed SigarLibrary.java and replaced it with SystemInfo.java and updated the calls. Switched to [OSHI library|https://github.com/oshi/oshi] Due to the differences between the Sigar library and the OSHI library some changes in the "warnIfRunningInDegradedMode" method were necessary. There is no OSHI support for determining the maximum number of processes allowed. However, all modern Linuxes have a "/proc/PID/limits" file. The SystemInfo.warnIfRunningInDegradedMode method will parse that file on a Linux box and if not on a Linux box will return the default number of PIDs for Linux (1024). We should not return 0 as that is obviously wrong since at least 1 process is running, We can not return -1 because that is the historical value of UNLIMITED, or INFINITY as it was called in the SigarLibrary. Since we only officially support Linux and the number of processes is probably wrong for other cases, the check now declares a degraded mode if the system is anything but Linux (yes, Mac OS will now report running in degraded mode). But since this is a warning and developers know to expect it there should not be a problem. The logging message now indicates what values are expected when the value is not met. > Remove dependency on sigar > -- > > Key: CASSANDRA-16565 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16565 > Project: Cassandra > Issue Type: Improvement > Components: Build >Reporter: David Capwell >Assignee: Claude Warren >Priority: Normal > Fix For: 5.x > > > sigar is used to check if the environment has good settings for running C*, > but requires we bundle a lot of native libraries to perform this check (which > can also be done elsewhere). This project also appears to be dead as the > last commit was around 6 years ago. > With the move to resolve artifacts rather than commit them, removing this > dependency would remove majority of the artifacts fetched from GitHub. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16565) Remove dependency on sigar
[ https://issues.apache.org/jira/browse/CASSANDRA-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779017#comment-17779017 ] Claude Warren commented on CASSANDRA-16565: --- The OSHI library does not have a method to retrieve the maximum processes. The maximum process request is equivalent to "ulimit -u". The OSHI library also returns 64 bit longs rather thant 32 bit values as sigar does. This results in some differences in the tests. I have extracted the sigar code into an evaluation package and coded an OSHI replacement to verify that there are no differences in the results. > Remove dependency on sigar > -- > > Key: CASSANDRA-16565 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16565 > Project: Cassandra > Issue Type: Improvement > Components: Build >Reporter: David Capwell >Assignee: Claude Warren >Priority: Normal > Fix For: 5.x > > > sigar is used to check if the environment has good settings for running C*, > but requires we bundle a lot of native libraries to perform this check (which > can also be done elsewhere). This project also appears to be dead as the > last commit was around 6 years ago. > With the move to resolve artifacts rather than commit them, removing this > dependency would remove majority of the artifacts fetched from GitHub. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-16565) Remove dependency on sigar
[ https://issues.apache.org/jira/browse/CASSANDRA-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Claude Warren reassigned CASSANDRA-16565: - Assignee: Claude Warren > Remove dependency on sigar > -- > > Key: CASSANDRA-16565 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16565 > Project: Cassandra > Issue Type: Improvement > Components: Build >Reporter: David Capwell >Assignee: Claude Warren >Priority: Normal > Fix For: 5.x > > > sigar is used to check if the environment has good settings for running C*, > but requires we bundle a lot of native libraries to perform this check (which > can also be done elsewhere). This project also appears to be dead as the > last commit was around 6 years ago. > With the move to resolve artifacts rather than commit them, removing this > dependency would remove majority of the artifacts fetched from GitHub. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16565) Remove dependency on sigar
[ https://issues.apache.org/jira/browse/CASSANDRA-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778533#comment-17778533 ] Claude Warren commented on CASSANDRA-16565: --- After the discussion on the mailing list [1] CASSANDRA-18775 was closed as "won't do" . I will look into removing lib/sigar from the project. [1]https://lists.apache.org/thread/6gzyh1zhxnkz50lld7hlgq172xc0pg3t > Remove dependency on sigar > -- > > Key: CASSANDRA-16565 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16565 > Project: Cassandra > Issue Type: Improvement > Components: Build >Reporter: David Capwell >Priority: Normal > Fix For: 5.x > > > sigar is used to check if the environment has good settings for running C*, > but requires we bundle a lot of native libraries to perform this check (which > can also be done elsewhere). This project also appears to be dead as the > last commit was around 6 years ago. > With the move to resolve artifacts rather than commit them, removing this > dependency would remove majority of the artifacts fetched from GitHub. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18775) Remove libraries in lib/sigar-bin for unsupported architectures
[
https://issues.apache.org/jira/browse/CASSANDRA-18775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claude Warren updated CASSANDRA-18775:
--
Resolution: Won't Do
Status: Resolved (was: Open)
Discussion on dev mailing list indicated that this approach is not supported.
Mailing list thread:
https://lists.apache.org/thread/6gzyh1zhxnkz50lld7hlgq172xc0pg3t
> Remove libraries in lib/sigar-bin for unsupported architectures
> ---
>
> Key: CASSANDRA-18775
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18775
> Project: Cassandra
> Issue Type: Task
> Components: Dependencies
>Reporter: Stefan Miklosovic
>Assignee: Maxim Muzafarov
>Priority: Normal
> Fix For: 5.x
>
>
> {code}
> ✔ ~/dev/cassandra/cassandra-instaclustr/cassandra [trunk L|⚑ 49]
> 15:41 $ ls -la lib/sigar-bin/
> total 6376
> drwxrwxr-x 2 fermat fermat 4096 aug 17 11:26 .
> drwxrwxr-x 5 fermat fermat 12288 aug 17 11:28 ..
> -rw-rw-r-- 1 fermat fermat 210641 aug 17 11:26 libsigar-amd64-freebsd-6.so
> -rw-rw-r-- 1 fermat fermat 246605 aug 17 11:26 libsigar-amd64-linux.so
> -rw-rw-r-- 1 fermat fermat 251360 aug 17 11:26 libsigar-amd64-solaris.so
> -rw-rw-r-- 1 fermat fermat 577452 aug 17 11:26 libsigar-ia64-hpux-11.sl
> -rw-rw-r-- 1 fermat fermat 494929 aug 17 11:26 libsigar-ia64-linux.so
> -rw-rw-r-- 1 fermat fermat 516096 aug 17 11:26 libsigar-pa-hpux-11.sl
> -rw-rw-r-- 1 fermat fermat 425077 aug 17 11:26 libsigar-ppc64-aix-5.so
> -rw-rw-r-- 1 fermat fermat 310792 aug 17 11:26 libsigar-ppc64le-linux.so
> -rw-rw-r-- 1 fermat fermat 330767 aug 17 11:26 libsigar-ppc64-linux.so
> -rw-rw-r-- 1 fermat fermat 400925 aug 17 11:26 libsigar-ppc-aix-5.so
> -rw-rw-r-- 1 fermat fermat 258547 aug 17 11:26 libsigar-ppc-linux.so
> -rw-rw-r-- 1 fermat fermat 269932 aug 17 11:26 libsigar-s390x-linux.so
> -rw-rw-r-- 1 fermat fermat 261896 aug 17 11:26 libsigar-sparc64-solaris.so
> -rw-rw-r-- 1 fermat fermat 285004 aug 17 11:26 libsigar-sparc-solaris.so
> -rw-rw-r-- 1 fermat fermat 397440 aug 17 11:26
> libsigar-universal64-macosx.dylib
> -rw-rw-r-- 1 fermat fermat 377668 aug 17 11:26 libsigar-universal-macosx.dylib
> -rw-rw-r-- 1 fermat fermat 179751 aug 17 11:26 libsigar-x86-freebsd-5.so
> -rw-rw-r-- 1 fermat fermat 179379 aug 17 11:26 libsigar-x86-freebsd-6.so
> -rw-rw-r-- 1 fermat fermat 233385 aug 17 11:26 libsigar-x86-linux.so
> -rw-rw-r-- 1 fermat fermat 242880 aug 17 11:26 libsigar-x86-solaris.so
> ✔ ~/dev/cassandra/cassandra-instaclustr/cassandra [trunk L|⚑ 49]
> 15:43 $ du -shc lib/sigar-bin/
> 6,3Mlib/sigar-bin/
> 6,3Mtotal
> {code}
> I think we could definitely improve this. We basically need just x86-linux,
> amd64-linux and two libs for macosx.
> We could save like 5M from the final tarball size. From 75M down to 70M is
> not bad!
> Or maybe I am not getting something and we need this?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-18775) Remove libraries in lib/sigar-bin for unsupported architectures
[
https://issues.apache.org/jira/browse/CASSANDRA-18775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1794#comment-1794
]
Claude Warren edited comment on CASSANDRA-18775 at 10/20/23 2:08 PM:
-
I think that preserving any sigar library where the file name contains the word
"linux" or "macosx" should be acceptable. This will preserve:
libsigar-amd64-linux.so
libsigar-ia64-linux.so
libsigar-ppc-linux.so
libsigar-ppc64-linux.so
libsigar-ppc64le-linux.so
libsigar-s390x-linux.so
libsigar-universal-macosx.dylib
libsigar-universal64-macosx.dylib
libsigar-x86-linux.so
and remove:
libsigar-amd64-freebsd-6.so
libsigar-amd64-solaris.so
libsigar-ia64-hpux-11.sl
libsigar-pa-hpux-11.sl
libsigar-ppc-aix-5.so
libsigar-ppc64-aix-5.so
libsigar-sparc-solaris.so
libsigar-sparc64-solaris.so
libsigar-x86-freebsd-5.so
libsigar-x86-freebsd-6.so
libsigar-x86-solaris.so
resulting in a savings of 3530461 bytes out of 6,450,526 from the
/lib/sigar-bin directory, an approximately 50% reduction.
Does anyone see any reason _not_ to do this?
was (Author: claudenw):
I think that preserving any sigar library where the file name contains the word
"linux" or "macosx" should be acceptable. This will preserve:
libsigar-amd64-linux.so
libsigar-ia64-linux.so
libsigar-ppc-linux.so
libsigar-ppc64-linux.so
libsigar-ppc64le-linux.so
libsigar-s390x-linux.so
libsigar-universal-macosx.dylib
libsigar-universal64-macosx.dylib
libsigar-x86-linux.so
and remove:
libsigar-amd64-freebsd-6.solibsigar-amd64-solaris.so
libsigar-ia64-hpux-11.sl
libsigar-pa-hpux-11.sl
libsigar-ppc-aix-5.solibsigar-ppc64-aix-5.so
libsigar-sparc-solaris.so
libsigar-sparc64-solaris.so
libsigar-x86-freebsd-5.so
libsigar-x86-freebsd-6.so
libsigar-x86-solaris.so
resulting in a savings of 3530461 bytes out of 6,450,526 from the
/lib/sigar-bin directory, a 48% reduction.
Does anyone see any reason _not_ to do this?
> Remove libraries in lib/sigar-bin for unsupported architectures
> ---
>
> Key: CASSANDRA-18775
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18775
> Project: Cassandra
> Issue Type: Task
> Components: Dependencies
>Reporter: Stefan Miklosovic
>Assignee: Maxim Muzafarov
>Priority: Normal
> Fix For: 5.x
>
>
> {code}
> ✔ ~/dev/cassandra/cassandra-instaclustr/cassandra [trunk L|⚑ 49]
> 15:41 $ ls -la lib/sigar-bin/
> total 6376
> drwxrwxr-x 2 fermat fermat 4096 aug 17 11:26 .
> drwxrwxr-x 5 fermat fermat 12288 aug 17 11:28 ..
> -rw-rw-r-- 1 fermat fermat 210641 aug 17 11:26 libsigar-amd64-freebsd-6.so
> -rw-rw-r-- 1 fermat fermat 246605 aug 17 11:26 libsigar-amd64-linux.so
> -rw-rw-r-- 1 fermat fermat 251360 aug 17 11:26 libsigar-amd64-solaris.so
> -rw-rw-r-- 1 fermat fermat 577452 aug 17 11:26 libsigar-ia64-hpux-11.sl
> -rw-rw-r-- 1 fermat fermat 494929 aug 17 11:26 libsigar-ia64-linux.so
> -rw-rw-r-- 1 fermat fermat 516096 aug 17 11:26 libsigar-pa-hpux-11.sl
> -rw-rw-r-- 1 fermat fermat 425077 aug 17 11:26 libsigar-ppc64-aix-5.so
> -rw-rw-r-- 1 fermat fermat 310792 aug 17 11:26 libsigar-ppc64le-linux.so
> -rw-rw-r-- 1 fermat fermat 330767 aug 17 11:26 libsigar-ppc64-linux.so
> -rw-rw-r-- 1 fermat fermat 400925 aug 17 11:26 libsigar-ppc-aix-5.so
> -rw-rw-r-- 1 fermat fermat 258547 aug 17 11:26 libsigar-ppc-linux.so
> -rw-rw-r-- 1 fermat fermat 269932 aug 17 11:26 libsigar-s390x-linux.so
> -rw-rw-r-- 1 fermat fermat 261896 aug 17 11:26 libsigar-sparc64-solaris.so
> -rw-rw-r-- 1 fermat fermat 285004 aug 17 11:26 libsigar-sparc-solaris.so
> -rw-rw-r-- 1 fermat fermat 397440 aug 17 11:26
> libsigar-universal64-macosx.dylib
> -rw-rw-r-- 1 fermat fermat 377668 aug 17 11:26 libsigar-universal-macosx.dylib
> -rw-rw-r-- 1 fermat fermat 179751 aug 17 11:26 libsigar-x86-freebsd-5.so
> -rw-rw-r-- 1 fermat fermat 179379 aug 17 11:26 libsigar-x86-freebsd-6.so
> -rw-rw-r-- 1 fermat fermat 233385 aug 17 11:26 libsigar-x86-linux.so
> -rw-rw-r-- 1 fermat fermat 242880 aug 17 11:26 libsigar-x86-solaris.so
> ✔ ~/dev/cassandra/cassandra-instaclustr/cassandra [trunk L|⚑ 49]
> 15:43 $ du -shc lib/sigar-bin/
> 6,3Mlib/sigar-bin/
> 6,3Mtotal
> {code}
> I think we could definitely improve this. We basically need just x86-linux,
> amd64-linux and two libs for macosx.
> We could save like 5M from the final tarball size. From 75M down to 70M is
> not bad!
> Or maybe I am not getting something and we need this?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-18775) Remove libraries in lib/sigar-bin for unsupported architectures
[
https://issues.apache.org/jira/browse/CASSANDRA-18775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1794#comment-1794
]
Claude Warren edited comment on CASSANDRA-18775 at 10/20/23 2:06 PM:
-
I think that preserving any sigar library where the file name contains the word
"linux" or "macosx" should be acceptable. This will preserve:
libsigar-amd64-linux.so
libsigar-ia64-linux.so
libsigar-ppc-linux.so
libsigar-ppc64-linux.so
libsigar-ppc64le-linux.so
libsigar-s390x-linux.so
libsigar-universal-macosx.dylib
libsigar-universal64-macosx.dylib
libsigar-x86-linux.so
and remove:
libsigar-amd64-freebsd-6.solibsigar-amd64-solaris.so
libsigar-ia64-hpux-11.sl
libsigar-pa-hpux-11.sl
libsigar-ppc-aix-5.solibsigar-ppc64-aix-5.so
libsigar-sparc-solaris.so
libsigar-sparc64-solaris.so
libsigar-x86-freebsd-5.so
libsigar-x86-freebsd-6.so
libsigar-x86-solaris.so
resulting in a savings of 3530461 bytes out of 6,450,526 from the
/lib/sigar-bin directory, a 48% reduction.
Does anyone see any reason _not_ to do this?
was (Author: claudenw):
I think that preserving any sigar library where the file name contains the word
"linux" or "macosx" should be acceptable. This will preserve:
libsigar-amd64-linux.so
libsigar-ia64-linux.so
libsigar-ppc-linux.so
libsigar-ppc64-aix-5.so
libsigar-ppc64-linux.so
libsigar-ppc64le-linux.so
libsigar-s390x-linux.so
libsigar-universal-macosx.dylib
libsigar-universal64-macosx.dylib
libsigar-x86-linux.so
and remove:
libsigar-amd64-freebsd-6.solibsigar-amd64-solaris.so
libsigar-ia64-hpux-11.sl
libsigar-pa-hpux-11.sl
libsigar-ppc-aix-5.so
libsigar-sparc-solaris.so
libsigar-sparc64-solaris.so
libsigar-x86-freebsd-5.so
libsigar-x86-freebsd-6.so
libsigar-x86-solaris.so
resulting in a savings of 3,105,384 bytes out of 6,450,526 from the
/lib/sigar-bin directory, a 48% reduction.
Does anyone see any reason _not_ to do this?
> Remove libraries in lib/sigar-bin for unsupported architectures
> ---
>
> Key: CASSANDRA-18775
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18775
> Project: Cassandra
> Issue Type: Task
> Components: Dependencies
>Reporter: Stefan Miklosovic
>Assignee: Maxim Muzafarov
>Priority: Normal
> Fix For: 5.x
>
>
> {code}
> ✔ ~/dev/cassandra/cassandra-instaclustr/cassandra [trunk L|⚑ 49]
> 15:41 $ ls -la lib/sigar-bin/
> total 6376
> drwxrwxr-x 2 fermat fermat 4096 aug 17 11:26 .
> drwxrwxr-x 5 fermat fermat 12288 aug 17 11:28 ..
> -rw-rw-r-- 1 fermat fermat 210641 aug 17 11:26 libsigar-amd64-freebsd-6.so
> -rw-rw-r-- 1 fermat fermat 246605 aug 17 11:26 libsigar-amd64-linux.so
> -rw-rw-r-- 1 fermat fermat 251360 aug 17 11:26 libsigar-amd64-solaris.so
> -rw-rw-r-- 1 fermat fermat 577452 aug 17 11:26 libsigar-ia64-hpux-11.sl
> -rw-rw-r-- 1 fermat fermat 494929 aug 17 11:26 libsigar-ia64-linux.so
> -rw-rw-r-- 1 fermat fermat 516096 aug 17 11:26 libsigar-pa-hpux-11.sl
> -rw-rw-r-- 1 fermat fermat 425077 aug 17 11:26 libsigar-ppc64-aix-5.so
> -rw-rw-r-- 1 fermat fermat 310792 aug 17 11:26 libsigar-ppc64le-linux.so
> -rw-rw-r-- 1 fermat fermat 330767 aug 17 11:26 libsigar-ppc64-linux.so
> -rw-rw-r-- 1 fermat fermat 400925 aug 17 11:26 libsigar-ppc-aix-5.so
> -rw-rw-r-- 1 fermat fermat 258547 aug 17 11:26 libsigar-ppc-linux.so
> -rw-rw-r-- 1 fermat fermat 269932 aug 17 11:26 libsigar-s390x-linux.so
> -rw-rw-r-- 1 fermat fermat 261896 aug 17 11:26 libsigar-sparc64-solaris.so
> -rw-rw-r-- 1 fermat fermat 285004 aug 17 11:26 libsigar-sparc-solaris.so
> -rw-rw-r-- 1 fermat fermat 397440 aug 17 11:26
> libsigar-universal64-macosx.dylib
> -rw-rw-r-- 1 fermat fermat 377668 aug 17 11:26 libsigar-universal-macosx.dylib
> -rw-rw-r-- 1 fermat fermat 179751 aug 17 11:26 libsigar-x86-freebsd-5.so
> -rw-rw-r-- 1 fermat fermat 179379 aug 17 11:26 libsigar-x86-freebsd-6.so
> -rw-rw-r-- 1 fermat fermat 233385 aug 17 11:26 libsigar-x86-linux.so
> -rw-rw-r-- 1 fermat fermat 242880 aug 17 11:26 libsigar-x86-solaris.so
> ✔ ~/dev/cassandra/cassandra-instaclustr/cassandra [trunk L|⚑ 49]
> 15:43 $ du -shc lib/sigar-bin/
> 6,3Mlib/sigar-bin/
> 6,3Mtotal
> {code}
> I think we could definitely improve this. We basically need just x86-linux,
> amd64-linux and two libs for macosx.
> We could save like 5M from the final tarball size. From 75M down to 70M is
> not bad!
> Or maybe I am not getting something and we need this?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18775) Remove libraries in lib/sigar-bin for unsupported architectures
[
https://issues.apache.org/jira/browse/CASSANDRA-18775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1794#comment-1794
]
Claude Warren commented on CASSANDRA-18775:
---
I think that preserving any sigar library where the file name contains the word
"linux" or "macosx" should be acceptable. This will preserve:
libsigar-amd64-linux.so
libsigar-ia64-linux.so
libsigar-ppc-linux.so
libsigar-ppc64-aix-5.so
libsigar-ppc64-linux.so
libsigar-ppc64le-linux.so
libsigar-s390x-linux.so
libsigar-universal-macosx.dylib
libsigar-universal64-macosx.dylib
libsigar-x86-linux.so
and remove:
libsigar-amd64-freebsd-6.solibsigar-amd64-solaris.so
libsigar-ia64-hpux-11.sl
libsigar-pa-hpux-11.sl
libsigar-ppc-aix-5.so
libsigar-sparc-solaris.so
libsigar-sparc64-solaris.so
libsigar-x86-freebsd-5.so
libsigar-x86-freebsd-6.so
libsigar-x86-solaris.so
resulting in a savings of 3,105,384 bytes out of 6,450,526 from the
/lib/sigar-bin directory, a 48% reduction.
Does anyone see any reason _not_ to do this?
> Remove libraries in lib/sigar-bin for unsupported architectures
> ---
>
> Key: CASSANDRA-18775
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18775
> Project: Cassandra
> Issue Type: Task
> Components: Dependencies
>Reporter: Stefan Miklosovic
>Assignee: Maxim Muzafarov
>Priority: Normal
> Fix For: 5.x
>
>
> {code}
> ✔ ~/dev/cassandra/cassandra-instaclustr/cassandra [trunk L|⚑ 49]
> 15:41 $ ls -la lib/sigar-bin/
> total 6376
> drwxrwxr-x 2 fermat fermat 4096 aug 17 11:26 .
> drwxrwxr-x 5 fermat fermat 12288 aug 17 11:28 ..
> -rw-rw-r-- 1 fermat fermat 210641 aug 17 11:26 libsigar-amd64-freebsd-6.so
> -rw-rw-r-- 1 fermat fermat 246605 aug 17 11:26 libsigar-amd64-linux.so
> -rw-rw-r-- 1 fermat fermat 251360 aug 17 11:26 libsigar-amd64-solaris.so
> -rw-rw-r-- 1 fermat fermat 577452 aug 17 11:26 libsigar-ia64-hpux-11.sl
> -rw-rw-r-- 1 fermat fermat 494929 aug 17 11:26 libsigar-ia64-linux.so
> -rw-rw-r-- 1 fermat fermat 516096 aug 17 11:26 libsigar-pa-hpux-11.sl
> -rw-rw-r-- 1 fermat fermat 425077 aug 17 11:26 libsigar-ppc64-aix-5.so
> -rw-rw-r-- 1 fermat fermat 310792 aug 17 11:26 libsigar-ppc64le-linux.so
> -rw-rw-r-- 1 fermat fermat 330767 aug 17 11:26 libsigar-ppc64-linux.so
> -rw-rw-r-- 1 fermat fermat 400925 aug 17 11:26 libsigar-ppc-aix-5.so
> -rw-rw-r-- 1 fermat fermat 258547 aug 17 11:26 libsigar-ppc-linux.so
> -rw-rw-r-- 1 fermat fermat 269932 aug 17 11:26 libsigar-s390x-linux.so
> -rw-rw-r-- 1 fermat fermat 261896 aug 17 11:26 libsigar-sparc64-solaris.so
> -rw-rw-r-- 1 fermat fermat 285004 aug 17 11:26 libsigar-sparc-solaris.so
> -rw-rw-r-- 1 fermat fermat 397440 aug 17 11:26
> libsigar-universal64-macosx.dylib
> -rw-rw-r-- 1 fermat fermat 377668 aug 17 11:26 libsigar-universal-macosx.dylib
> -rw-rw-r-- 1 fermat fermat 179751 aug 17 11:26 libsigar-x86-freebsd-5.so
> -rw-rw-r-- 1 fermat fermat 179379 aug 17 11:26 libsigar-x86-freebsd-6.so
> -rw-rw-r-- 1 fermat fermat 233385 aug 17 11:26 libsigar-x86-linux.so
> -rw-rw-r-- 1 fermat fermat 242880 aug 17 11:26 libsigar-x86-solaris.so
> ✔ ~/dev/cassandra/cassandra-instaclustr/cassandra [trunk L|⚑ 49]
> 15:43 $ du -shc lib/sigar-bin/
> 6,3Mlib/sigar-bin/
> 6,3Mtotal
> {code}
> I think we could definitely improve this. We basically need just x86-linux,
> amd64-linux and two libs for macosx.
> We could save like 5M from the final tarball size. From 75M down to 70M is
> not bad!
> Or maybe I am not getting something and we need this?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18934) Downgrade to 4.1 fails due to schema changes
[
https://issues.apache.org/jira/browse/CASSANDRA-18934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776507#comment-17776507
]
Claude Warren commented on CASSANDRA-18934:
---
I discovered similar issues while attempting to write a downgrader from v 4 to
3.1 as noted in CASSANDRA-8928 and associated pull request 2045.
CASSANDRA-8928 is part of CASSANDRA-18300. I believe that this should also be
part of that epic.
The design for the Downgrader was to have the Downgrader be able to downgrade
to the earliest format of the previous version so 5,0 should downgrade to 4.0.
Then the standard roll forward from 4.0 to 4.x should work correctly.
> Downgrade to 4.1 fails due to schema changes
>
>
> Key: CASSANDRA-18934
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18934
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
>Reporter: David Capwell
>Priority: Normal
> Fix For: 5.0.x, 5.x
>
>
> We are required to support 5.0 downgrading to 4.1 as a migration step, but we
> don’t have tests to show this is working… I wrote a quick test to make sure a
> change we needed in Accord wouldn’t block the downgrade and see that we fail
> right now.
> {code}
> ERROR 20:56:39 Exiting due to error while processing commit log during
> initialization.
> org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException:
> Unexpected error deserializing mutation; saved to
> /var/folders/h1/s_3p1x3s3hl0hltbpck67m0hgn/T/mutation418421767150092dat.
> This may be caused by replaying a mutation against a table with the same
> name but incompatible schema. Exception follows: java.lang.RuntimeException:
> Unknown column compaction_properties during deserialization
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readMutation(CommitLogReader.java:464)
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:397)
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:244)
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147)
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:191)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:223)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:204)
> {code}
> This was caused by a schema change in CASSANDRA-18061
> {code}
> /*
> * Licensed to the Apache Software Foundation (ASF) under one
> * or more contributor license agreements. See the NOTICE file
> * distributed with this work for additional information
> * regarding copyright ownership. The ASF licenses this file
> * to you under the Apache License, Version 2.0 (the
> * "License"); you may not use this file except in compliance
> * with the License. You may obtain a copy of the License at
> *
> * http://www.apache.org/licenses/LICENSE-2.0
> *
> * Unless required by applicable law or agreed to in writing, software
> * distributed under the License is distributed on an "AS IS" BASIS,
> * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> * See the License for the specific language governing permissions and
> * limitations under the License.
> */
> package org.apache.cassandra.distributed.upgrade;
> import java.io.IOException;
> import java.io.File;
> import java.util.concurrent.atomic.AtomicBoolean;
> import org.junit.Test;
> import org.apache.cassandra.distributed.api.IUpgradeableInstance;
> public class DowngradeTest extends UpgradeTestBase
> {
> @Test
> public void test() throws Throwable
> {
> AtomicBoolean first = new AtomicBoolean(true);
> new TestCase()
> .nodes(1)
> .withConfig(c -> {
> if (first.compareAndSet(true, false))
> c.set("storage_compatibility_mode", "CASSANDRA_4");
> })
> .downgradeTo(v41)
> .setup(cluster -> {})
> // Uncomment if you want to test what happens after reading the commit log,
> which fails right now
> //.runBeforeNodeRestart((cluster, nodeId) -> {
> //IUpgradeableInstance inst = cluster.get(nodeId);
> //File f = new File((String)
> inst.config().get("commitlog_directory"));
> //deleteRecursive(f);
> //})
> .runAfterClusterUpgrade(cluster -> {})
> .run();
> }
> private void deleteRecursive(File f)
> {
> if (f.isDirectory())
> {
> File[] children = f.listFiles();
> if (children != null)
> {
> for (File c : children)
>
[jira] [Commented] (CASSANDRA-18817) RPM install does not work with jdk17
[ https://issues.apache.org/jira/browse/CASSANDRA-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761849#comment-17761849 ] Claude Warren commented on CASSANDRA-18817: --- My change is for CASSANDRA-17773 where I propose extracting the common "boilerplate" code to simplify writing files that start/stop Cassandra and its stand-alone tools. I don't think that it would impact this issue in any way. The changes just centralize the code to discover where the cassandra.in.sh file is, it does not change the contents of the file. > RPM install does not work with jdk17 > > > Key: CASSANDRA-18817 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18817 > Project: Cassandra > Issue Type: Bug > Components: Packaging >Reporter: Michael Semb Wever >Priority: Normal > Fix For: 5.0-alpha > > > The cassandra.in.sh file was updated as part of CASSANDRA-18255 but the > corresponding redhat/cassandra.in.sh was not. > This means when cassandra starts on a rpm install using jdk17 the > jvm11-server.options file is still used. And so it fails. > Initial patch that works: > https://github.com/apache/cassandra/compare/cassandra-5.0...thelastpickle:cassandra:mck/redhat-jdk17-fix/5.0 > > Maybe it is possible to find a superior solution, like wrapping this file > instead of duplicating it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18739) UDF functions fail to load on rolling restart
[
https://issues.apache.org/jira/browse/CASSANDRA-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755520#comment-17755520
]
Claude Warren commented on CASSANDRA-18739:
---
Further analysis shows the issue is in non-system keyspaces, where there is a
user defined type (UDT) and a UDF that may create the UDT using the
udfContxt.newUTDValue() method. Alternatively calling as UDF calling
udfContext().newTupleValue() method will also trigger the NPE.
The issue is that the keyspace variable in udfContextImpl is set to null
because the keyspace is being created when the value is set.
There is no requirement for more than one server.
> UDF functions fail to load on rolling restart
> -
>
> Key: CASSANDRA-18739
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18739
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/UDF
>Reporter: Claude Warren
>Assignee: Claude Warren
>Priority: High
> Fix For: 4.0.x, 4.1.x, 5.x
>
> Attachments: udf_error.cql, udf_error_data.cql
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> UDFs fail to reload properly after a rolling restart.
> h3. *Symptom:*
> NPE thrown when used after restart.
> h3. *Steps to recreate:*
> # Create a cluster as per cql file
> # Populate the cluster with data.cql.
> # Execute SELECT city_measurements(city, measurement, 16.5) AS m FROM current
> # expect min and max values for cities.
> # Performing a rolling restart on one server.
> # When the server is back up
> # Execute SELECT city_measurements(city, measurement, 16.5) AS m FROM current
> # expect: error result with NPE message.
> {*}Analysis{*}:
> During system restart the SchemaKeyspace.fetchNonSystemKeyspaces() is called,
> when a keyspace with a UDF is loaded the SchemaKeyspace method
> createUDFFromRow() is called, this in turn calls UDFunction.create() which
> eventually calls back to UDFunction constructor where the
> Schema.instance.getKeyspaceMetadata() is called with the keyspace for the UDF
> name as the argument. However, the keyspace for the UDF name is being
> constructed and is not yet in the instance so the method returns null for the
> KeyspaceMetadata. That null KeyspaceMetadata is then used in the udfContext.
> Later when the UDF method is called, if there is a need to call a method on
> the keyspaceMetadata, such as udfContext.newUDTValue() where the
> implementation uses keyspaceMetadata.types, a null pointer is thrown.
> I have verified this affects version 4.0, 4.1 and trunk. I have not verified
> 3.x but I suspect it is the same there.
> I modified UDFunction constructor to assert that the metadata was not null
> and received the following stack trace
> ERROR [main] 2023-08-09 11:44:46,408 CassandraDaemon.java:911 - Exception
> encountered during startup
> java.lang.AssertionError: No metadata for temperatures.city_measurements_sfunc
> at
> org.apache.cassandra.cql3.functions.UDFunction.(UDFunction.java:240)
> at
> org.apache.cassandra.cql3.functions.JavaBasedUDFunction.(JavaBasedUDFunction.java:195)
> at
> org.apache.cassandra.cql3.functions.UDFunction.create(UDFunction.java:276)
> at
> org.apache.cassandra.schema.SchemaKeyspace.createUDFFromRow(SchemaKeyspace.java:1182)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchUDFs(SchemaKeyspace.java:1131)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchFunctions(SchemaKeyspace.java:1119)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:859)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:848)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:836)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:132)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:121)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:287)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
>
> {{*Possible solution:*}}
> *Version 4.x*
> Create a KeyspaceMetadata.Builder class that uses accepts the types, tables
> and views but uses a builder for the functions.
> Add a KeyspaceMetadata constructor to accept the KeyspaceMetadata.Builder so
> that the function builder keyspaceMetadata value can be set correctly during
> construction of the KeyspaceMetadata.
> Modify SchemaKeyspace.fetchKeyspace(string) so that it uses the
> KeyspaceMetadata.Builder.
>
> *Version 5.x*
> Similar to 4.x except that the KeyspaceMetadata.Builder will have to have
> builders for Views and Tables because the functions
[jira] [Commented] (CASSANDRA-18739) UDF functions fail to load on rolling restart
[
https://issues.apache.org/jira/browse/CASSANDRA-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753229#comment-17753229
]
Claude Warren commented on CASSANDRA-18739:
---
Fixes and test code for v4.0 in
[https://github.com/claudenw/cassandra/tree/CASSANDRA-18739_v4.0]
Fixes and test code for v4.1 in
[https://github.com/claudenw/cassandra/tree/CASSANDRA-18739_v4.1]
I will open pull requests next week.
> UDF functions fail to load on rolling restart
> -
>
> Key: CASSANDRA-18739
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18739
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/UDF
>Reporter: Claude Warren
>Assignee: Claude Warren
>Priority: Normal
> Attachments: udf_error.cql, udf_error_data.cql
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> UDFs fail to reload properly after a rolling restart.
> h3. *Symptom:*
> NPE thrown when used after restart.
> h3. *Steps to recreate:*
> # Create a cluster as per cql file
> # Populate the cluster with data.cql.
> # Execute SELECT city_measurements(city, measurement, 16.5) AS m FROM current
> # expect min and max values for cities.
> # Performing a rolling restart on one server.
> # When the server is back up
> # Execute SELECT city_measurements(city, measurement, 16.5) AS m FROM current
> # expect: error result with NPE message.
> {*}Analysis{*}:
> During system restart the SchemaKeyspace.fetchNonSystemKeyspaces() is called,
> when a keyspace with a UDF is loaded the SchemaKeyspace method
> createUDFFromRow() is called, this in turn calls UDFunction.create() which
> eventually calls back to UDFunction constructor where the
> Schema.instance.getKeyspaceMetadata() is called with the keyspace for the UDF
> name as the argument. However, the keyspace for the UDF name is being
> constructed and is not yet in the instance so the method returns null for the
> KeyspaceMetadata. That null KeyspaceMetadata is then used in the udfContext.
> Later when the UDF method is called, if there is a need to call a method on
> the keyspaceMetadata, such as udfContext.newUDTValue() where the
> implementation uses keyspaceMetadata.types, a null pointer is thrown.
> I have verified this affects version 4.0, 4.1 and trunk. I have not verified
> 3.x but I suspect it is the same there.
> I modified UDFunction constructor to assert that the metadata was not null
> and received the following stack trace
> ERROR [main] 2023-08-09 11:44:46,408 CassandraDaemon.java:911 - Exception
> encountered during startup
> java.lang.AssertionError: No metadata for temperatures.city_measurements_sfunc
> at
> org.apache.cassandra.cql3.functions.UDFunction.(UDFunction.java:240)
> at
> org.apache.cassandra.cql3.functions.JavaBasedUDFunction.(JavaBasedUDFunction.java:195)
> at
> org.apache.cassandra.cql3.functions.UDFunction.create(UDFunction.java:276)
> at
> org.apache.cassandra.schema.SchemaKeyspace.createUDFFromRow(SchemaKeyspace.java:1182)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchUDFs(SchemaKeyspace.java:1131)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchFunctions(SchemaKeyspace.java:1119)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:859)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:848)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:836)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:132)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:121)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:287)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
>
> {{*Possible solution:*}}
> *Version 4.x*
> Create a KeyspaceMetadata.Builder class that uses accepts the types, tables
> and views but uses a builder for the functions.
> Add a KeyspaceMetadata constructor to accept the KeyspaceMetadata.Builder so
> that the function builder keyspaceMetadata value can be set correctly during
> construction of the KeyspaceMetadata.
> Modify SchemaKeyspace.fetchKeyspace(string) so that it uses the
> KeyspaceMetadata.Builder.
>
> *Version 5.x*
> Similar to 4.x except that the KeyspaceMetadata.Builder will have to have
> builders for Views and Tables because the functions necessary to construct
> those objects will not be available until the KeyspaceMetadata.Builder
> constructs it.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe,
[jira] [Commented] (CASSANDRA-18739) UDF functions fail to load on rolling restart
[
https://issues.apache.org/jira/browse/CASSANDRA-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752679#comment-17752679
]
Claude Warren commented on CASSANDRA-18739:
---
Dang. I accidentally deleted my earlier question.
A quick visual review indicates that it probably will.
> UDF functions fail to load on rolling restart
> -
>
> Key: CASSANDRA-18739
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18739
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/UDF
>Reporter: Claude Warren
>Assignee: Claude Warren
>Priority: Normal
> Attachments: udf_error.cql, udf_error_data.cql
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> UDFs fail to reload properly after a rolling restart.
> h3. *Symptom:*
> NPE thrown when used after restart.
> h3. *Steps to recreate:*
> # Create a cluster as per cql file
> # Populate the cluster with data.cql.
> # Execute SELECT city_measurements(city, measurement, 16.5) AS m FROM current
> # expect min and max values for cities.
> # Performing a rolling restart on one server.
> # When the server is back up
> # Execute SELECT city_measurements(city, measurement, 16.5) AS m FROM current
> # expect: error result with NPE message.
> {*}Analysis{*}:
> During system restart the SchemaKeyspace.fetchNonSystemKeyspaces() is called,
> when a keyspace with a UDF is loaded the SchemaKeyspace method
> createUDFFromRow() is called, this in turn calls UDFunction.create() which
> eventually calls back to UDFunction constructor where the
> Schema.instance.getKeyspaceMetadata() is called with the keyspace for the UDF
> name as the argument. However, the keyspace for the UDF name is being
> constructed and is not yet in the instance so the method returns null for the
> KeyspaceMetadata. That null KeyspaceMetadata is then used in the udfContext.
> Later when the UDF method is called, if there is a need to call a method on
> the keyspaceMetadata, such as udfContext.newUDTValue() where the
> implementation uses keyspaceMetadata.types, a null pointer is thrown.
> I have verified this affects version 4.0, 4.1 and trunk. I have not verified
> 3.x but I suspect it is the same there.
> I modified UDFunction constructor to assert that the metadata was not null
> and received the following stack trace
> ERROR [main] 2023-08-09 11:44:46,408 CassandraDaemon.java:911 - Exception
> encountered during startup
> java.lang.AssertionError: No metadata for temperatures.city_measurements_sfunc
> at
> org.apache.cassandra.cql3.functions.UDFunction.(UDFunction.java:240)
> at
> org.apache.cassandra.cql3.functions.JavaBasedUDFunction.(JavaBasedUDFunction.java:195)
> at
> org.apache.cassandra.cql3.functions.UDFunction.create(UDFunction.java:276)
> at
> org.apache.cassandra.schema.SchemaKeyspace.createUDFFromRow(SchemaKeyspace.java:1182)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchUDFs(SchemaKeyspace.java:1131)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchFunctions(SchemaKeyspace.java:1119)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:859)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:848)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:836)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:132)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:121)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:287)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
>
> {{*Possible solution:*}}
> *Version 4.x*
> Create a KeyspaceMetadata.Builder class that uses accepts the types, tables
> and views but uses a builder for the functions.
> Add a KeyspaceMetadata constructor to accept the KeyspaceMetadata.Builder so
> that the function builder keyspaceMetadata value can be set correctly during
> construction of the KeyspaceMetadata.
> Modify SchemaKeyspace.fetchKeyspace(string) so that it uses the
> KeyspaceMetadata.Builder.
>
> *Version 5.x*
> Similar to 4.x except that the KeyspaceMetadata.Builder will have to have
> builders for Views and Tables because the functions necessary to construct
> those objects will not be available until the KeyspaceMetadata.Builder
> constructs it.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] (CASSANDRA-18739) UDF functions fail to load on rolling restart
[ https://issues.apache.org/jira/browse/CASSANDRA-18739 ]
Claude Warren deleted comment on CASSANDRA-18739:
---
was (Author: claudenw):
I did not look in depth into the trunk issues but I saw that Views and Tables
now require userfunctions in their constructors. Do you know if your change
work in that case as well?
> UDF functions fail to load on rolling restart
> -
>
> Key: CASSANDRA-18739
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18739
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/UDF
>Reporter: Claude Warren
>Assignee: Claude Warren
>Priority: Normal
> Attachments: udf_error.cql, udf_error_data.cql
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> UDFs fail to reload properly after a rolling restart.
> h3. *Symptom:*
> NPE thrown when used after restart.
> h3. *Steps to recreate:*
> # Create a cluster as per cql file
> # Populate the cluster with data.cql.
> # Execute SELECT city_measurements(city, measurement, 16.5) AS m FROM current
> # expect min and max values for cities.
> # Performing a rolling restart on one server.
> # When the server is back up
> # Execute SELECT city_measurements(city, measurement, 16.5) AS m FROM current
> # expect: error result with NPE message.
> {*}Analysis{*}:
> During system restart the SchemaKeyspace.fetchNonSystemKeyspaces() is called,
> when a keyspace with a UDF is loaded the SchemaKeyspace method
> createUDFFromRow() is called, this in turn calls UDFunction.create() which
> eventually calls back to UDFunction constructor where the
> Schema.instance.getKeyspaceMetadata() is called with the keyspace for the UDF
> name as the argument. However, the keyspace for the UDF name is being
> constructed and is not yet in the instance so the method returns null for the
> KeyspaceMetadata. That null KeyspaceMetadata is then used in the udfContext.
> Later when the UDF method is called, if there is a need to call a method on
> the keyspaceMetadata, such as udfContext.newUDTValue() where the
> implementation uses keyspaceMetadata.types, a null pointer is thrown.
> I have verified this affects version 4.0, 4.1 and trunk. I have not verified
> 3.x but I suspect it is the same there.
> I modified UDFunction constructor to assert that the metadata was not null
> and received the following stack trace
> ERROR [main] 2023-08-09 11:44:46,408 CassandraDaemon.java:911 - Exception
> encountered during startup
> java.lang.AssertionError: No metadata for temperatures.city_measurements_sfunc
> at
> org.apache.cassandra.cql3.functions.UDFunction.(UDFunction.java:240)
> at
> org.apache.cassandra.cql3.functions.JavaBasedUDFunction.(JavaBasedUDFunction.java:195)
> at
> org.apache.cassandra.cql3.functions.UDFunction.create(UDFunction.java:276)
> at
> org.apache.cassandra.schema.SchemaKeyspace.createUDFFromRow(SchemaKeyspace.java:1182)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchUDFs(SchemaKeyspace.java:1131)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchFunctions(SchemaKeyspace.java:1119)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:859)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:848)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:836)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:132)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:121)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:287)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
>
> {{*Possible solution:*}}
> *Version 4.x*
> Create a KeyspaceMetadata.Builder class that uses accepts the types, tables
> and views but uses a builder for the functions.
> Add a KeyspaceMetadata constructor to accept the KeyspaceMetadata.Builder so
> that the function builder keyspaceMetadata value can be set correctly during
> construction of the KeyspaceMetadata.
> Modify SchemaKeyspace.fetchKeyspace(string) so that it uses the
> KeyspaceMetadata.Builder.
>
> *Version 5.x*
> Similar to 4.x except that the KeyspaceMetadata.Builder will have to have
> builders for Views and Tables because the functions necessary to construct
> those objects will not be available until the KeyspaceMetadata.Builder
> constructs it.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18739) UDF functions fail to load on rolling restart
[
https://issues.apache.org/jira/browse/CASSANDRA-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752668#comment-17752668
]
Claude Warren commented on CASSANDRA-18739:
---
I did not look in depth into the trunk issues but I saw that Views and Tables
now require userfunctions in their constructors. Do you know if your change
work in that case as well?
> UDF functions fail to load on rolling restart
> -
>
> Key: CASSANDRA-18739
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18739
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/UDF
>Reporter: Claude Warren
>Assignee: Claude Warren
>Priority: Normal
> Attachments: udf_error.cql, udf_error_data.cql
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> UDFs fail to reload properly after a rolling restart.
> h3. *Symptom:*
> NPE thrown when used after restart.
> h3. *Steps to recreate:*
> # Create a cluster as per cql file
> # Populate the cluster with data.cql.
> # Execute SELECT city_measurements(city, measurement, 16.5) AS m FROM current
> # expect min and max values for cities.
> # Performing a rolling restart on one server.
> # When the server is back up
> # Execute SELECT city_measurements(city, measurement, 16.5) AS m FROM current
> # expect: error result with NPE message.
> {*}Analysis{*}:
> During system restart the SchemaKeyspace.fetchNonSystemKeyspaces() is called,
> when a keyspace with a UDF is loaded the SchemaKeyspace method
> createUDFFromRow() is called, this in turn calls UDFunction.create() which
> eventually calls back to UDFunction constructor where the
> Schema.instance.getKeyspaceMetadata() is called with the keyspace for the UDF
> name as the argument. However, the keyspace for the UDF name is being
> constructed and is not yet in the instance so the method returns null for the
> KeyspaceMetadata. That null KeyspaceMetadata is then used in the udfContext.
> Later when the UDF method is called, if there is a need to call a method on
> the keyspaceMetadata, such as udfContext.newUDTValue() where the
> implementation uses keyspaceMetadata.types, a null pointer is thrown.
> I have verified this affects version 4.0, 4.1 and trunk. I have not verified
> 3.x but I suspect it is the same there.
> I modified UDFunction constructor to assert that the metadata was not null
> and received the following stack trace
> ERROR [main] 2023-08-09 11:44:46,408 CassandraDaemon.java:911 - Exception
> encountered during startup
> java.lang.AssertionError: No metadata for temperatures.city_measurements_sfunc
> at
> org.apache.cassandra.cql3.functions.UDFunction.(UDFunction.java:240)
> at
> org.apache.cassandra.cql3.functions.JavaBasedUDFunction.(JavaBasedUDFunction.java:195)
> at
> org.apache.cassandra.cql3.functions.UDFunction.create(UDFunction.java:276)
> at
> org.apache.cassandra.schema.SchemaKeyspace.createUDFFromRow(SchemaKeyspace.java:1182)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchUDFs(SchemaKeyspace.java:1131)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchFunctions(SchemaKeyspace.java:1119)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:859)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:848)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:836)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:132)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:121)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:287)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
>
> {{*Possible solution:*}}
> *Version 4.x*
> Create a KeyspaceMetadata.Builder class that uses accepts the types, tables
> and views but uses a builder for the functions.
> Add a KeyspaceMetadata constructor to accept the KeyspaceMetadata.Builder so
> that the function builder keyspaceMetadata value can be set correctly during
> construction of the KeyspaceMetadata.
> Modify SchemaKeyspace.fetchKeyspace(string) so that it uses the
> KeyspaceMetadata.Builder.
>
> *Version 5.x*
> Similar to 4.x except that the KeyspaceMetadata.Builder will have to have
> builders for Views and Tables because the functions necessary to construct
> those objects will not be available until the KeyspaceMetadata.Builder
> constructs it.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For
[jira] [Updated] (CASSANDRA-18739) USF functions fail to load on rolling restart
[
https://issues.apache.org/jira/browse/CASSANDRA-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claude Warren updated CASSANDRA-18739:
--
Description:
UDFs fail to reload properly after a rolling restart.
h3. *Symptom:*
NPE thrown when used after restart.
h3. *Steps to recreate:*
# Create a cluster as per cql file
# Populate the cluster with data.cql.
# Execute SELECT city_measurements(city, measurement, 16.5) AS m FROM current
# expect min and max values for cities.
# Performing a rolling restart on one server.
# When the server is back up
# Execute SELECT city_measurements(city, measurement, 16.5) AS m FROM current
# expect: error result with NPE message.
{*}Analysis{*}:
During system restart the SchemaKeyspace.fetchNonSystemKeyspaces() is called,
when a keyspace with a UDF is loaded the ShcemaKeyspace method
createUDFFromRow() is called, this in tern calls UDFunction.create() which
eventually calls back to UDFunction constructor where the
Schema.instance.getKeySpaceMetadata() is called with the keyspace for the UDF
name as the argument. However, the keyspace for the UDF name is being
constructed and is not yet in the instance so the method returns null for the
KeyspaceMetadata. That null KeyspaceMetadata is then used in the udfContext.
Later when the UDF method is called, if there is a need to call a method on the
keyspaceMetadata, such as udfContext.newUDTValue() where the implementation
uses keyspaceMetadata.types, a null pointer is thrown.
I have verified this affects version 4.0, 4.1 and trunk. I have not verified
3.x but I suspect it is the same there.
I modified UDFunction constructor to assert that the metadata was not null and
received the following stack trace
ERROR [main] 2023-08-09 11:44:46,408 CassandraDaemon.java:911 - Exception
encountered during startup
java.lang.AssertionError: No metadata for temperatures.city_measurements_sfunc
at
org.apache.cassandra.cql3.functions.UDFunction.(UDFunction.java:240)
at
org.apache.cassandra.cql3.functions.JavaBasedUDFunction.(JavaBasedUDFunction.java:195)
at
org.apache.cassandra.cql3.functions.UDFunction.create(UDFunction.java:276)
at
org.apache.cassandra.schema.SchemaKeyspace.createUDFFromRow(SchemaKeyspace.java:1182)
at
org.apache.cassandra.schema.SchemaKeyspace.fetchUDFs(SchemaKeyspace.java:1131)
at
org.apache.cassandra.schema.SchemaKeyspace.fetchFunctions(SchemaKeyspace.java:1119)
at
org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:859)
at
org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:848)
at
org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:836)
at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:132)
at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:121)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:287)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
{{*Possible solution:*}}
*Version 4.x*
Create a KeyspaceMetadata.Builder class that uses accepts the types, tables and
views but uses a builder for the functions.
Add a KeyspaceMetadata constructor to accept the KeyspaceMetadata.Builder so
that the function builder keyspaceMetadata value can be set correctly during
construction of the KeyspaceMetadata.
Modify SchemaKeyspace.fetchKeyspace(string) so that it uses the
KeyspaceMetadata.Builder.
*Version 5.x*
Similar to 4.x except that the KeyspaceMetadata.Builder will have to have
builders for Views and Tables because the functions necessary to construct
those objects will not be available until the KeyspaceMetadata.Builder
constructs it.
was:
UDFs fail to reload properly after a rolling restart.
h3. *Symptom:*
NPE thrown when used after restart.
h3. *Steps to recreate:*
# Create a cluster as per cql file
# Populate the cluster with data.cql.
# Execute SELECT city_measurements(city, measurement, 16.5) AS m FROM current
# expect min and max values for cities.
# Performing a rolling restart on one server.
# When the server is back up
# Execute SELECT city_measurements(city, measurement, 16.5) AS m FROM current
# expect: error result with NPE message.
{*}Analysis{*}:
During system restart the SchemaKeyspace.fetchNonSystemKeyspaces() is called,
when a keyspace with a UDF is loaded the ShcemaKeyspace method
createUDFFromRow() is called, this in tern calls UDFunction.create() which
eventually calls back to UDFunction constructor where the
Schema.instance.getKeySpaceMetadata() is called with the keyspace for the UDF
name as the argument. However, the keyspace for the UDF name is being
constructed and is not yet in the instance so the method returns null for the
[jira] [Created] (CASSANDRA-18739) USF functions fail to load on rolling restart
Claude Warren created CASSANDRA-18739:
-
Summary: USF functions fail to load on rolling restart
Key: CASSANDRA-18739
URL: https://issues.apache.org/jira/browse/CASSANDRA-18739
Project: Cassandra
Issue Type: Bug
Components: Feature/UDF
Reporter: Claude Warren
Assignee: Claude Warren
Attachments: udf_error.cql, udf_error_data.cql
UDFs fail to reload properly after a rolling restart.
h3. *Symptom:*
NPE thrown when used after restart.
h3. *Steps to recreate:*
# Create a cluster as per cql file
# Populate the cluster with data.cql.
# Execute SELECT city_measurements(city, measurement, 16.5) AS m FROM current
# expect min and max values for cities.
# Performing a rolling restart on one server.
# When the server is back up
# Execute SELECT city_measurements(city, measurement, 16.5) AS m FROM current
# expect: error result with NPE message.
{*}Analysis{*}:
During system restart the SchemaKeyspace.fetchNonSystemKeyspaces() is called,
when a keyspace with a UDF is loaded the ShcemaKeyspace method
createUDFFromRow() is called, this in tern calls UDFunction.create() which
eventually calls back to UDFunction constructor where the
Schema.instance.getKeySpaceMetadata() is called with the keyspace for the UDF
name as the argument. However, the keyspace for the UDF name is being
constructed and is not yet in the instance so the method returns null for the
KeyspaceMetadata. That null KeyspaceMetadata is then used in the udfContext.
Later when the UDF method is called, if there is a need to call a method on the
keyspaceMetadata, such as udfContext.newUDTValue() where the implementation
uses keyspaceMetadata.types, a null pointer is thrown.
I have verified this affects version 4.0, 4.1 and trunk. I have not verified
3.x but I suspect it is the same there.
I modified UDFunction constructor to assert that the metadata was not null and
received the following stack trace
{{{}java.lang.AssertionError: No metadata for
temperatures.city_measurements_sfunc{}}}{{{}at
org.apache.cassandra.cql3.functions.UDFunction.(UDFunction.java:240){}}}{{{}at
org.apache.cassandra.cql3.functions.JavaBasedUDFunction.(JavaBasedUDFunction.java:195){}}}{{{}at
org.apache.cassandra.cql3.functions.UDFunction.create(UDFunction.java:276){}}}{{{}at
org.apache.cassandra.schema.SchemaKeyspace.createUDFFromRow(SchemaKeyspace.java:1182){}}}{{{}at
org.apache.cassandra.schema.SchemaKeyspace.fetchUDFs(SchemaKeyspace.java:1131){}}}{{{}at
org.apache.cassandra.schema.SchemaKeyspace.fetchFunctions(SchemaKeyspace.java:1119){}}}{{{}at
org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:859){}}}{{{}at
org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:848){}}}{{{}at
org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:836){}}}{{{}at
org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:132){}}}{{{}at
org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:121){}}}{{{}at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:287){}}}{{{}at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765){}}}{{{}at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889){}}}
{{*Possible solution:*}}
*Version 4.x*
Create a KeyspaceMetadata.Builder class that uses accepts the types, tables and
views but uses a builder for the functions.
Add a KeyspaceMetadata constructor to accept the KeyspaceMetadata.Builder so
that the function builder keyspaceMetadata value can be set correctly during
construction of the KeyspaceMetadata.
Modify SchemaKeyspace.fetchKeyspace(string) so that it uses the
KeyspaceMetadata.Builder.
*Version 5.x*
Similar to 4.x except that the KeyspaceMetadata.Builder will have to have
builders for Views and Tables because the functions necessary to construct
those objects will not be available until the KeyspaceMetadata.Builder
constructs it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-18637) As DataOutputBuffer approaches the MAX_BUFFER_SIZE limit there should be a warning.
Claude Warren created CASSANDRA-18637: - Summary: As DataOutputBuffer approaches the MAX_BUFFER_SIZE limit there should be a warning. Key: CASSANDRA-18637 URL: https://issues.apache.org/jira/browse/CASSANDRA-18637 Project: Cassandra Issue Type: Improvement Components: Local/SSTable Reporter: Claude Warren As DataOuptutBuffer fills the available buffer space it allocates more by doubling the buffer size until it reaches 64Mb (assuming DOB_DOUBLING_THRESHOLD_MB has not been set). After which it will increase the size by 1.5 until it reaches 0x7FFF_FFF8 bytes (MAX_BUFFER_SIZE) and then the reallocation will fail. 0x_5550 x 1.5 = 0x7FFF_FFF8 Once the buffer is 0x_5550 bytes or larger in size the next allocation will return 0x7FFF_FFF8 bytes and the one after that will fail. I propose adding a configurable warning that applies when the allocated buffer first approaches 0x_5550, by defining a WARNING_THRESHOLD so that the first time the buffer allocation crosses 0x_5550 - WARNING_THRESHOLD an alert is sent. The alert should be instrumented such that automated monitoring systems can report the issue before it becomes emergent. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-18625) Testing required to test upgrade and downgrade completeness
[ https://issues.apache.org/jira/browse/CASSANDRA-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Claude Warren reassigned CASSANDRA-18625: - Assignee: Claude Warren > Testing required to test upgrade and downgrade completeness > --- > > Key: CASSANDRA-18625 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18625 > Project: Cassandra > Issue Type: Improvement >Reporter: Claude Warren >Assignee: Claude Warren >Priority: Normal > > In the paper "Understanding and Detecting Software Upgrade Failures in > Distributed Systems" the authors identify several hidden issues with > Cassandra upgrade capabilities. Indeed these were submitted as bugs and > fixed. The authors also point out that a failure during upgrade (or > downgrade) is a critical failure often taking out a whole cluster. > This issue is to find/create a testing library similar to the one described > at [2] that is integrated into the Cassandra build system at appropriate > points. Those points to be determined as development of the solution > progresses. > > [1] [https://dl.acm.org/doi/10.1145/3477132.3483577] > [2] https://sysartifacts.github.io/sosp2021/summaries/duptester -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-18625) Testing required to test upgrade and downgrade completeness
Claude Warren created CASSANDRA-18625: - Summary: Testing required to test upgrade and downgrade completeness Key: CASSANDRA-18625 URL: https://issues.apache.org/jira/browse/CASSANDRA-18625 Project: Cassandra Issue Type: Improvement Reporter: Claude Warren In the paper "Understanding and Detecting Software Upgrade Failures in Distributed Systems" the authors identify several hidden issues with Cassandra upgrade capabilities. Indeed these were submitted as bugs and fixed. The authors also point out that a failure during upgrade (or downgrade) is a critical failure often taking out a whole cluster. This issue is to find/create a testing library similar to the one described at [2] that is integrated into the Cassandra build system at appropriate points. Those points to be determined as development of the solution progresses. [1] [https://dl.acm.org/doi/10.1145/3477132.3483577] [2] https://sysartifacts.github.io/sosp2021/summaries/duptester -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-18358) Fix issue with system.local.broadcast_port when downgrading to v3
[
https://issues.apache.org/jira/browse/CASSANDRA-18358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claude Warren reassigned CASSANDRA-18358:
-
Assignee: Claude Warren
> Fix issue with system.local.broadcast_port when downgrading to v3
> -
>
> Key: CASSANDRA-18358
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18358
> Project: Cassandra
> Issue Type: Bug
>Reporter: Claude Warren
>Assignee: Claude Warren
>Priority: Normal
>
> {{system.local}} table 4.0 format adds the {{broadcast_port}} column. There
> is no way to make 3.x ignore it when we try to downgrade.
> There are some tables that are defined in code. (e.g. system.local)
> When v3 sstable schema for these tables detects:
> * a column in the sstable schema that is not in the data schema it creates a
> deleted column in the data schema.
> * a column in the data schema that is not in the sstable schema it throws an
> exception.
>
> There are two basic approaches to solve this:
> * rewrite the data so that the additional column is removed. This is an
> expensive process.
> * rework the code so that columns in the data but not in the table are
> ignored for this class of tables (statically defined in code).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18358) Fix issue with system.local.broadcast_port when downgrading to v3
[
https://issues.apache.org/jira/browse/CASSANDRA-18358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724245#comment-17724245
]
Claude Warren commented on CASSANDRA-18358:
---
I managed to rewrite the sstable without the broadcase_port and other
associated changes in the pull request for CASSANDRA-8928 :
[https://github.com/apache/cassandra/pull/2045]
> Fix issue with system.local.broadcast_port when downgrading to v3
> -
>
> Key: CASSANDRA-18358
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18358
> Project: Cassandra
> Issue Type: Bug
>Reporter: Claude Warren
>Priority: Normal
>
> {{system.local}} table 4.0 format adds the {{broadcast_port}} column. There
> is no way to make 3.x ignore it when we try to downgrade.
> There are some tables that are defined in code. (e.g. system.local)
> When v3 sstable schema for these tables detects:
> * a column in the sstable schema that is not in the data schema it creates a
> deleted column in the data schema.
> * a column in the data schema that is not in the sstable schema it throws an
> exception.
>
> There are two basic approaches to solve this:
> * rewrite the data so that the additional column is removed. This is an
> expensive process.
> * rework the code so that columns in the data but not in the table are
> ignored for this class of tables (statically defined in code).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-8928) Add downgradesstables
[
https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17723930#comment-17723930
]
Claude Warren edited comment on CASSANDRA-8928 at 5/18/23 2:59 PM:
---
[~nasnousssi]
I have updated the pull request: [https://github.com/apache/cassandra/pull/2045]
The problem has been that between v3 and v4 there was a breaking change in the
system local table where the columns "broadcast_port", "listen_port", and
"rpc_port" were added. The code (in the current pull request) provides
functionality to remove those columns when the older table is written. The
code also allows for other transformations of the columns, though none are
implemented.
In order for the downgrade to work the following steps are taken (not all are
in this code, some are in a script I have for testing the process)
# Execute the standalone downgrade on the desired table(s).
# Delete the system_schema tables.
# Delete the {*}-Filter.db, *-Index.db, *-TOC.txt, *-Digest.{*}, and
*-Summary.db for the modified table(s)
# Delete the original files (e.g. nb-*)
# Start the earlier version of the software.
I tested the current code by starting 4.1 to create the tables. Downgraded all
the tables in the database to "ma" version, followed the above steps and
started 3.11 According to the logs 3.11.14 started.
The current pull request code is not as clean as I would like it, but it does
work correctly.
I would like some comments on the general approach for removing columns where
they are filtered out of the row definition during writing.
Also, I think this code should be rebased to 4.1 branch. I wonder if this
change is enough to require bumping to 4.2.
There are changes to:
* The column family store - exposing the sstable ID generator
* CompressedMetadata - adding a flag to indicate the presence of the max
compressed length variable.
* CompressedSequentialWriter - using the compressed length variable for writes.
* TableMetadata.Builder – to replace an existing column definition with
another one.
was (Author: claudenw):
[~nasnousssi]
I have updated the pull request: [https://github.com/apache/cassandra/pull/2045]
The problem has been that between v3 and v4 there was a breaking change in the
system local table where the columns "broadcast_port", "listen_port", and
"rpc_port" were added. The code (in the current pull request) provides
functionality to remove those columns when the older table is written. The
code also allows for other transformations of the columns, though none are
implemented.
In order for the downgrade to work the following steps are taken (not all are
in this code, some are in a script I have for testing the process)
# Execute the standalone downgrade on the desired table(s).
# Delete the system_schema tables.
# Delete the {*}-Filter.db, *-Index.db, *-TOC.txt, *-Digest.{*}, and
*-Summary.db for the modified table(s)
# Delete the original files (e.g. nb-*)
# Start the earlier version of the software.
I tested the current code by starting 4.1 to create the tables. Downgraded all
the tables in the database to "ma" version, followed the above steps and
started 3.11 According to the logs 3.11.14 started.
The current pull request code is not as clean as I would like it, but it does
work correctly.
I would like some comments on the general approach for removing columns where
they are filtered out of the row definition during writing.
> Add downgradesstables
> -
>
> Key: CASSANDRA-8928
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8928
> Project: Cassandra
> Issue Type: New Feature
> Components: Legacy/Tools
>Reporter: Jeremy Hanna
>Assignee: Claude Warren
>Priority: Low
> Labels: remove-reopen
> Fix For: 5.x
>
>
> As mentioned in other places such as CASSANDRA-8047 and in the wild,
> sometimes you need to go back. A downgrade sstables utility would be nice
> for a lot of reasons and I don't know that supporting going back to the
> previous major version format would be too much code since we already support
> reading the previous version.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-8928) Add downgradesstables
[
https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17723930#comment-17723930
]
Claude Warren edited comment on CASSANDRA-8928 at 5/18/23 2:52 PM:
---
[~nasnousssi]
I have updated the pull request: [https://github.com/apache/cassandra/pull/2045]
The problem has been that between v3 and v4 there was a breaking change in the
system local table where the columns "broadcast_port", "listen_port", and
"rpc_port" were added. The code (in the current pull request) provides
functionality to remove those columns when the older table is written. The
code also allows for other transformations of the columns, though none are
implemented.
In order for the downgrade to work the following steps are taken (not all are
in this code, some are in a script I have for testing the process)
# Execute the standalone downgrade on the desired table(s).
# Delete the system_schema tables.
# Delete the {*}-Filter.db, *-Index.db, *-TOC.txt, *-Digest.{*}, and
*-Summary.db for the modified table(s)
# Delete the original files (e.g. nb-*)
# Start the earlier version of the software.
I tested the current code by starting 4.1 to create the tables. Downgraded all
the tables in the database to "ma" version, followed the above steps and
started 3.11 According to the logs 3.11.14 started.
The current pull request code is not as clean as I would like it, but it does
work correctly.
I would like some comments on the general approach for removing columns where
they are filtered out of the row definition during writing.
was (Author: claudenw):
I have updated the pull request: [https://github.com/apache/cassandra/pull/2045]
The problem has been that between v3 and v4 there was a breaking change in the
system local table where the columns "broadcast_port", "listen_port", and
"rpc_port" were added. The code (in the current pull request) provides
functionality to remove those columns when the older table is written. The
code also allows for other transformations of the columns, though none are
implemented.
In order for the downgrade to work the following steps are taken (not all are
in this code, some are in a script I have for testing the process)
# Execute the standalone downgrade on the desired table(s).
# Delete the system_schema tables.
# Delete the *-Filter.db, *-Index.db, *-TOC.txt, *-Digest.*, and *-Summary.db
for the modified table(s)
# Delete the original files (e.g. nb-*)
# Start the earlier version of the software.
I tested the current code by starting 4.1 to create the tables. Downgraded all
the tables in the database to "ma" version, followed the above steps and
started 3.11 According to the logs 3.11.14 started.
The current pull request code is not as clean as I would like it, but it does
work correctly.
I would like some comments on the general approach for removing columns where
they are filtered out of the row definition during writing.
> Add downgradesstables
> -
>
> Key: CASSANDRA-8928
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8928
> Project: Cassandra
> Issue Type: New Feature
> Components: Legacy/Tools
>Reporter: Jeremy Hanna
>Assignee: Claude Warren
>Priority: Low
> Labels: remove-reopen
> Fix For: 5.x
>
>
> As mentioned in other places such as CASSANDRA-8047 and in the wild,
> sometimes you need to go back. A downgrade sstables utility would be nice
> for a lot of reasons and I don't know that supporting going back to the
> previous major version format would be too much code since we already support
> reading the previous version.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17723930#comment-17723930 ] Claude Warren commented on CASSANDRA-8928: -- I have updated the pull request: [https://github.com/apache/cassandra/pull/2045] The problem has been that between v3 and v4 there was a breaking change in the system local table where the columns "broadcast_port", "listen_port", and "rpc_port" were added. The code (in the current pull request) provides functionality to remove those columns when the older table is written. The code also allows for other transformations of the columns, though none are implemented. In order for the downgrade to work the following steps are taken (not all are in this code, some are in a script I have for testing the process) # Execute the standalone downgrade on the desired table(s). # Delete the system_schema tables. # Delete the *-Filter.db, *-Index.db, *-TOC.txt, *-Digest.*, and *-Summary.db for the modified table(s) # Delete the original files (e.g. nb-*) # Start the earlier version of the software. I tested the current code by starting 4.1 to create the tables. Downgraded all the tables in the database to "ma" version, followed the above steps and started 3.11 According to the logs 3.11.14 started. The current pull request code is not as clean as I would like it, but it does work correctly. I would like some comments on the general approach for removing columns where they are filtered out of the row definition during writing. > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Tools >Reporter: Jeremy Hanna >Assignee: Claude Warren >Priority: Low > Labels: remove-reopen > Fix For: 5.x > > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722847#comment-17722847
]
Claude Warren commented on CASSANDRA-12937:
---
[~smiklosovic] Done. I will rewrite log after approval of the code
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
> Fix For: 5.x
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8460) Make it possible to move non-compacting sstables to slow/big storage in DTCS
[ https://issues.apache.org/jira/browse/CASSANDRA-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722765#comment-17722765 ] Claude Warren commented on CASSANDRA-8460: -- Is there still interest in moving this concept forward? I am interested in exploring this option. > Make it possible to move non-compacting sstables to slow/big storage in DTCS > > > Key: CASSANDRA-8460 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8460 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction >Reporter: Marcus Eriksson >Assignee: Lerh Chuan Low >Priority: Normal > Labels: doc-impacting, dtcs > Fix For: 5.x > > > It would be nice if we could configure DTCS to have a set of extra data > directories where we move the sstables once they are older than > max_sstable_age_days. > This would enable users to have a quick, small SSD for hot, new data, and big > spinning disks for data that is rarely read and never compacted. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17719800#comment-17719800
]
Claude Warren commented on CASSANDRA-12937:
---
I did not modify the code path for the commit_log_compression (or any other
compression other than sstable_compression) . CommitLog does not create a
CompressionParams object from the configuration it just constructs an
ICompression instance. That path is not affected by the changes in the pull
request. We can add that later but it was not included in this change.
There was a path where unrecognised variables were not detected. That has been
corrected and tests added.
There was an issue with chunk_length_in_kb not being serialized to the map
correctly in some cases. That has been corrected and tests added.
The commented out configuration in the yaml file is not functional if
uncommented.
Additional comments were added to explain the items that were previously
explained by the broken configuration. (e.g. max_compressed_length needing to
have the GiB, MiB, KiB suffix)
Additional comment was added to specify that all parameter values must be
strings.
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
> Fix For: 5.x
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17718870#comment-17718870
]
Claude Warren commented on CASSANDRA-12937:
---
[~smiklosovic], since the aliases are defined in the CompressionParams class
they could be used anywhere. Good observation.
The crcCheckChance parameter was removed but the crcCheckChance variable still
exists because there are getter/setter functions that are used in other code.
Is the intention to remove the crcCheckChance from all code, just the
CompressionParams or ???
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
> Fix For: 5.x
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17718819#comment-17718819 ] Claude Warren commented on CASSANDRA-8928: -- I also encountered the bigV3 problem and fixed that in a later push. The problem I am trying to solve is to remove columns that were added to system tables between 3 and 4. I also think there were some format changes that need to be undone. > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Tools >Reporter: Jeremy Hanna >Assignee: Claude Warren >Priority: Low > Labels: remove-reopen > Fix For: 5.x > > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[ https://issues.apache.org/jira/browse/CASSANDRA-12937 ]
Claude Warren deleted comment on CASSANDRA-12937:
---
was (Author: claudenw):
Code rebased to trunk
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
> Fix For: 5.x
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714856#comment-17714856
]
Claude Warren edited comment on CASSANDRA-12937 at 4/21/23 1:57 PM:
[~smiklosovic], [~mck]
Please review [Github Pull Request
#2254|https://github.com/apache/cassandra/pull/2254] there are a number of
changes.
* switched to ParameterizedClass for yaml configuration
* ensured processing to support CQL compression parameters
* added extensive yaml file documentation.
* moved methods used only in testing from CompressionParams to a new
TestingCompressionParamsFactory class.
* updated tests to use TestingCompressionFactory.
* added extensive testing to ensure that all combinations of Map and
ParameterizedClass construct the same CompressionParams.
* added additional testing coverage for pre-existing methods
* unified error reporting so that the same error on different paths reports
with the same text.
was (Author: claudenw):
[~smiklosovic], [~mck]
Please review [Github Pull Request
#2254|https://github.com/apache/cassandra/pull/2254] there are a number of
changes.
* switched to ParameterizedClass for yaml configuration
* ensured processing to support CQL compression parameters
* added extensive yaml file documentation.
* moved methods used only in testing from CompressionParams to a new
TestingCompressionParamsFactory class.
* updated tests to use TestingCompressionFactory.
* added extensive testing to ensure that all combinations of Map and
ParameterixedClass construct the same CompressionParams.
* added additional testing coverage or pre-existing methods
* unified error reporting so that the same error on different paths reports
with the same text.
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
> Fix For: 5.x
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714856#comment-17714856
]
Claude Warren commented on CASSANDRA-12937:
---
[~smiklosovic], [~mck]
Please review [Github Pull Request
#2254|https://github.com/apache/cassandra/pull/2254] there are a number of
changes.
* switched to ParameterizedClass for yaml configuration
* ensured processing to support CQL compression parameters
* added extensive yaml file documentation.
* moved methods used only in testing from CompressionParams to a new
TestingCompressionParamsFactory class.
* updated tests to use TestingCompressionFactory.
* added extensive testing to ensure that all combinations of Map and
ParameterixedClass construct the same CompressionParams.
* added additional testing coverage or pre-existing methods
* unified error reporting so that the same error on different paths reports
with the same text.
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
> Fix For: 5.x
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714848#comment-17714848
]
Claude Warren commented on CASSANDRA-12937:
---
Code rebased to trunk
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
> Fix For: 5.x
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713920#comment-17713920 ] Claude Warren commented on CASSANDRA-8928: -- [~nasnousssi] , Yes. it is one of the 3 tickets I am working on. There is currently an issue with downgrading from 4.x to 3.x You said you were able to get the downgrade to work on a single node. Can you tell me what versions you downgraded from and to? > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Tools >Reporter: Jeremy Hanna >Assignee: Claude Warren >Priority: Low > Labels: remove-reopen > Fix For: 5.x > > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713592#comment-17713592
]
Claude Warren commented on CASSANDRA-12937:
---
OK. Here is my proposal
||Parameter||CompressionParam
Map (input)||CompressionParam
Map (output)||CompressionParam as Serializer||CQL||Notes||
|chunk_length_in_kb |X|X|X|X| |
|chunk_length_kb|X| | | | |
|chunk_length|X| | | |chunk length with DataStorageSpec suffix|
|crc_check_chance|X| | |X|Accepted but ignored and removed from map.
This is the current operation|
|min_compress_ratio|X|X| | | |
|max_compressed_length|X| |X| |DataStorageSpec suffix|
|lz4_compressor_type|X|X|X|X|left in options for ICompressor construction|
|{{lz4_high_compressor_level}}|X|X|X|X|left in options for ICompressor
construction|
|{{compression_level}}|X|X|X|X|left in options for ICompressor construction|
|(arbitrary parameter)|X|X|X| |left in options for ICompressor construction|
All of the parameters will be accepted.
Conflicts of input will be handled as follows:
* multiple chunk_length type parameters (chunk_length, chunk_length_kb,
chunk_length_in_kb) will result in a ConfigurationException
* Specifying both min_compress_ratio and max_compressed_length will result in
a ConfigurationException.
* crc_check_chance processing will not change.
I think this meets the requirements for CQL configuration copy paste as well as
YAML properties expressed with DataStoreageSpec suffix.
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
> Fix For: 5.x
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713549#comment-17713549
]
Claude Warren commented on CASSANDRA-12937:
---
I am trying to follow the guidance I was given. At this point I think we need
to have a discussion on the dev mailing list to arrive at a consensus of how
this should be done.
Currently CompressionParams takes the class name and the parameters. It
extracts chunk_length_kb (or chunk_length_in_kb) and min_compress_ratio from
the parameters and uses them to build the CompressionParams instance.
The table below outlines the parameters and where they are used. Blue cells
indicate proposed changes.
||Parameter||CompressionParam as map||CompressionParam as
Serializer||CQL||Notes||
|chunk_length_in_kb |X|X|X| |
|chunk_length_kb|(deprecated - read not written)| | | |
|chunk_length|X( proposed)| | |chunk length with DataStorageSpec suffix|
|crc_check_chance|(deprecated - read not written )| |X|crc_check_chance is not
used in CompressionParam|
|min_compress_ratio|X| | | |
|max_compressed_length| |X| |Proposed to add to map input as a string with
DataStorageSpec suffix|
|lz4_compressor_type|X|X|X| |
|{{lz4_high_compressor_level}}|X|X|X| |
|{{compression_level}}|X|X|X|Zstd compressor param|
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
> Fix For: 5.x
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713524#comment-17713524
]
Claude Warren edited comment on CASSANDRA-12937 at 4/18/23 10:09 AM:
-
hints_compression and commitlog_compression use the standard ParameterizedClass.
The CompressionParams has 3 parameters that it extracts or creates from the
parameters in the ParameterizedClass. The parameters in CompressionParams are
{code:java}
private final int chunkLength;
private final int maxCompressedLength; // In content we store max length to
avoid rounding errors causing compress/decompress mismatch.
private final double minCompressRatio; // In configuration we store min ratio,
the input parameter.
{code}
The ParameterizedClass constructor that accepts the Map of
options expects a key of "chunk_length_in_kb" or "chunk_length_kb" as well as
a "min_compress_ratio".
This change I made does not change the hints_compression or
commitlog_compression options.
The yaml file has an additional set of requirements:
* The chunkLength (yaml: chunk_length) should be specified with the
DataStorageSpec suffix (e.g. KiB).
* The maxCompressedLength should be accepted as a parameter.
* The maxCompressedLength (yaml: max_compressed_length) should be specified
with the DataStorageSpec suffix (e.g. KiB).
* maxCompressedLength and minCompressRatio are related to each other via
chunk_length; so only one can be specified.
I could work chunkLength and maxCompressedLength into the class_name
parameters, however, I believe this will result in adding 2 more reserved words
both of which will need to be removed from the parameter list. This change
will affect all CompressionParams constructions that use the
Map format.
I will make the change with the following processes for determining collision
values:
* If both max_compressed_length and min_compress_ratio are specified an
ConfigurationException will be thrown.
* if both chunk_length and either chunk_length_in_kb or chunk_length_kb are
specified and they are not equal ConfiguraitonException will be thrown.
* if chunk_length or max_compressed_length are specified and do not use the
DataStorageSpec suffix a ConfigurationException will be thrown
I will also ensure that the short names: lz4, none, noop, snappy, deflate, and
zstd will work as class names and use the defaults specified by the
CompressionParams methods of the same names.
was (Author: claudenw):
hints_compression and commitlog_compression use the standard ParameterizedClass.
The CompressionParams has 3 parameters that it extracts or creates from the
parameters in the ParameterizedClass. The parameters in CompressionParams are
{code:java}
private final int chunkLength;
private final int maxCompressedLength; // In content we store max length to
avoid rounding errors causing compress/decompress mismatch.
private final double minCompressRatio; // In configuration we store min ratio,
the input parameter.
{code}
The ParameterizedClass constructor that accepts the Map of
options expects a key of "chunk_length_in_kb" or "chunk_length_kb" as well as
a "min_compress_ratio".
This change I made does not change the hints_compression or
commitlog_compression options.
The yaml file has an additional set of requirements:
* The chunkLength (yaml: chunk_length) should be specified with the
DataStorageSpec suffix (e.g. KiB).
* The maxCompressedLength should be accepted as a parameter.
* The maxCompressedLength (yaml: max_compressed_length) should be specified
with the DataStorageSpec extensions (e.g. KiB).
* maxCompressedLength and minCompressRatio are related to each other via
chunk_length; so only one can be specified.
I could work chunkLength and maxCompressedLength into the class_name
parameters, however, I believe this will result in adding 2 more reserved words
both of which will need to be removed from the parameter list. This change
will affect all CompressionParams constructions that use the
Map format.
I will make the change with the following processes for determining collision
values:
* If both max_compressed_length and min_compress_ratio are specified an
ConfigurationException will be thrown.
* if both chunk_length and either chunk_length_in_kb or chunk_length_kb are
specified and they are not equal ConfiguraitonException will be thrown.
* if chunk_length or max_compressed_length are specified and do not use the
DataStorageSpec suffix a ConfigurationException will be thrown
I will also ensure that the short names: lz4, none, noop, snappy, deflate, and
zstd will work as class names and use the defaults specified by the
CompressionParams methods of the same names.
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL:
[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713524#comment-17713524
]
Claude Warren commented on CASSANDRA-12937:
---
hints_compression and commitlog_compression use the standard ParameterizedClass.
The CompressionParams has 3 parameters that it extracts or creates from the
parameters in the ParameterizedClass. The parameters in CompressionParams are
{code:java}
private final int chunkLength;
private final int maxCompressedLength; // In content we store max length to
avoid rounding errors causing compress/decompress mismatch.
private final double minCompressRatio; // In configuration we store min ratio,
the input parameter.
{code}
The ParameterizedClass constructor that accepts the Map of
options expects a key of "chunk_length_in_kb" or "chunk_length_kb" as well as
a "min_compress_ratio".
This change I made does not change the hints_compression or
commitlog_compression options.
The yaml file has an additional set of requirements:
* The chunkLength (yaml: chunk_length) should be specified with the
DataStorageSpec suffix (e.g. KiB).
* The maxCompressedLength should be accepted as a parameter.
* The maxCompressedLength (yaml: max_compressed_length) should be specified
with the DataStorageSpec extensions (e.g. KiB).
* maxCompressedLength and minCompressRatio are related to each other via
chunk_length; so only one can be specified.
I could work chunkLength and maxCompressedLength into the class_name
parameters, however, I believe this will result in adding 2 more reserved words
both of which will need to be removed from the parameter list. This change
will affect all CompressionParams constructions that use the
Map format.
I will make the change with the following processes for determining collision
values:
* If both max_compressed_length and min_compress_ratio are specified an
ConfigurationException will be thrown.
* if both chunk_length and either chunk_length_in_kb or chunk_length_kb are
specified and they are not equal ConfiguraitonException will be thrown.
* if chunk_length or max_compressed_length are specified and do not use the
DataStorageSpec suffix a ConfigurationException will be thrown
I will also ensure that the short names: lz4, none, noop, snappy, deflate, and
zstd will work as class names and use the defaults specified by the
CompressionParams methods of the same names.
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
> Fix For: 5.x
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713416#comment-17713416
]
Claude Warren commented on CASSANDRA-12937:
---
Because if I have some compressor X that has a property called "chunk_length"
then
{code:java}
sstable_compression:
- class_name: org.apache.cassandra.io.compress.X
parameters:
- chunk_length: 15
{code}
Will collide with the "chunk_length" for the CompressionParams. It is
semantically clearer what the chunk_length is for.
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
> Fix For: 5.x
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713051#comment-17713051
]
Claude Warren commented on CASSANDRA-12937:
---
[https://github.com/apache/cassandra/pull/2254/files] is the correct one.
Yes, it still has SSTableCompressionOptions, SSTableCompressionOptions are
used to store the non class-name configuration options. The "properties"
within the SSTableCompressionOptions contains the properties for the
"class-name" instantiation. In this way we don't have any property name
collisions.
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
> Fix For: 5.x
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17712982#comment-17712982
]
Claude Warren commented on CASSANDRA-12937:
---
[~mck] [~smiklosovic] I posted updates to the pull requests that I think
resolve all the issues. Please take a look.
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
> Fix For: 5.x
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710853#comment-17710853
]
Claude Warren commented on CASSANDRA-12937:
---
I reworked [https://github.com/apache/cassandra/pull/2254] Please take a
look. It has all the options from the earlier fix but not the removal of
"type" and the defaulting of additional options as parameters to the compressor
class.
those changes should now be limited to CompressionParams.java and
CompressionParamsTest.java
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
> Fix For: 5.x
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710441#comment-17710441
]
Claude Warren commented on CASSANDRA-12937:
---
[~smiklosovic] your alias solution sounds good to me.
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
> Fix For: 5.x
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709354#comment-17709354
]
Claude Warren commented on CASSANDRA-12937:
---
How about we settle on something like this:
sstable_compressor has several standard parameters:
* enabled - defaults to true. If false no other parameters may be specified.
(original operation for flag)
* chunk_length – defaults to 16KiB. Converts to chunk_length_in_kb in the
CompressorParams (optional)
* type - A standard type of compression: must be one of none, noop, lz4,
snappy, deflate zstd (optional)
* class_name – standard class_name for compressor class to load. (optional)
* min_compress_ratio – defaults to 0.0 (optional)
* max_compressed_length – defaults to Integer.MAX_VALUE (optional)
type and class_name are mutually exclusive and one must be specified. Type
will define the standard types and class_name specify class to load. Class
must accept a Map constructor.
min_compress_ratio and max_compressed_length are mutually exclusive. If
neither is specified a min_compress_ratio of 0.0 is used (current default).
Any other parameters specified are assumed to be arguments passed to the
class_name constructor.
{code:java}sstable_compressor:
class_name: org.apache.cassandra.io.compress.LZ4Compressor.class
lz4_compressor_type: fast
{code}
and
{code:java}sstable_compressor:
type: lz4
lz4_compressor_type: fast
{code}
are the same
We also need to move the {{SSTableCompressionOptions.getCompressionParams()}}
functionality into the {{CompressionParams.fromOptions()}} method to remove the
schema package dependency from the config class.
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
> Fix For: 5.x
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709226#comment-17709226
]
Claude Warren edited comment on CASSANDRA-12937 at 4/6/23 6:53 AM:
---
Looking at CompressionParams there are a number of default configurations e.g.
snappy(), lz4(), and noCompression() that I thought would be in common use.
What I wanted to do was to provide an easy way to get to call those methods as
well as provide the ability to load any compressor via the map.
Also, early on the idea of putting chunk_length_in_kb was rejected with the
"16KiB" form for the input requested.
If it is agreed to remove the shortcuts and use the simple map form with the
parameters I'll make those changes.
I did come across a note that says that configuration file and CQL use
different parameters for compression, thus I onluy implemented
min_compress_ratio and used it to calculate max_compression_length.
So I got to where the code is by trying to support the defaults in
CompressonParams and following the min_compress_ratio not
max_compression_length in the config files.
you can configure the ztsd with 12Kib chunks by setting:
{code:java}
sstable_compressor:
chunk_length: 12KiB
type: zstd
{code}
was (Author: claudenw):
Looking at CompressionParams there are a number of default configurations e.g.
snappy(), lz4(), and noCompression() that I thought would be in common use.
What I wanted to do was to provide an easy way to get to call those methods as
well as provide the ability to load any compressor via the map.
Also, early on the idea of putting chunk_length_in_kb was rejected with the
"16KiB" form for the input requested.
If it is agreed to remove the shortcuts and use the simple map form with the
parameters I'll make those changes.
I did come across a note that says that configuration file and CQL use
different parameters for compression, thus I onluy implemented
min_compress_ratio and used it to calculate max_compression_length.
So I got to where the code is by trying to support the defaults in
CompressonParams and following the min_compress_ratio not
max_compression_length in the config files.
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
> Fix For: 5.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709226#comment-17709226
]
Claude Warren commented on CASSANDRA-12937:
---
Looking at CompressionParams there are a number of default configurations e.g.
snappy(), lz4(), and noCompression() that I thought would be in common use.
What I wanted to do was to provide an easy way to get to call those methods as
well as provide the ability to load any compressor via the map.
Also, early on the idea of putting chunk_length_in_kb was rejected with the
"16KiB" form for the input requested.
If it is agreed to remove the shortcuts and use the simple map form with the
parameters I'll make those changes.
I did come across a note that says that configuration file and CQL use
different parameters for compression, thus I onluy implemented
min_compress_ratio and used it to calculate max_compression_length.
So I got to where the code is by trying to support the defaults in
CompressonParams and following the min_compress_ratio not
max_compression_length in the config files.
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
> Fix For: 5.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709018#comment-17709018
]
Claude Warren commented on CASSANDRA-12937:
---
Your's is tested and fixed so let's go with that.
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
> Fix For: 5.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17707961#comment-17707961
]
Claude Warren commented on CASSANDRA-12937:
---
I did a rework of [https://github.com/apache/cassandra/pull/2199] to fix the
issues. [~smiklosovic] , I also created a new PR with the code rebased against
trunk. That PR is https://github.com/apache/cassandra/pull/2254
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
> Fix For: 5.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17704164#comment-17704164 ] Claude Warren commented on CASSANDRA-8928: -- Downgrading from V4 to V3 is not possible until CASSANDRA-18358 is resolved > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Tools >Reporter: Jeremy Hanna >Assignee: Claude Warren >Priority: Low > Labels: remove-reopen > Fix For: 5.x > > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-18358) Fix issue with system.local.broadcast_port when downgrading to v3
Claude Warren created CASSANDRA-18358:
-
Summary: Fix issue with system.local.broadcast_port when
downgrading to v3
Key: CASSANDRA-18358
URL: https://issues.apache.org/jira/browse/CASSANDRA-18358
Project: Cassandra
Issue Type: Bug
Reporter: Claude Warren
{{system.local}} table 4.0 format adds the {{broadcast_port}} column. There is
no way to make 3.x ignore it when we try to downgrade.
There are some tables that are defined in code. (e.g. system.local)
When v3 sstable schema for these tables detects:
* a column in the sstable schema that is not in the data schema it creates a
deleted column in the data schema.
* a column in the data schema that is not in the sstable schema it throws an
exception.
There are two basic approaches to solve this:
* rewrite the data so that the additional column is removed. This is an
expensive process.
* rework the code so that columns in the data but not in the table are ignored
for this class of tables (statically defined in code).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17697448#comment-17697448
]
Claude Warren commented on CASSANDRA-12937:
---
Created pull request: https://github.com/apache/cassandra/pull/2199
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-12937) Default setting (yaml) for SSTable compression
[
https://issues.apache.org/jira/browse/CASSANDRA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claude Warren reassigned CASSANDRA-12937:
-
Assignee: Claude Warren
> Default setting (yaml) for SSTable compression
> --
>
> Key: CASSANDRA-12937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12937
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Config
>Reporter: Michael Semb Wever
>Assignee: Claude Warren
>Priority: Low
> Labels: AdventCalendar2021, lhf
>
> In many situations the choice of compression for sstables is more relevant to
> the disks attached than to the schema and data.
> This issue is to add to cassandra.yaml a default value for sstable
> compression that new tables will inherit (instead of the defaults found in
> {{CompressionParams.DEFAULT}}.
> Examples where this can be relevant are filesystems that do on-the-fly
> compression (btrfs, zfs) or specific disk configurations or even specific C*
> versions (see CASSANDRA-10995 ).
> +Additional information for newcomers+
> Some new fields need to be added to {{cassandra.yaml}} to allow specifying
> the field required for defining the default compression parameters. In
> {{DatabaseDescriptor}} a new {{CompressionParams}} field should be added for
> the default compression. This field should be initialized in
> {{DatabaseDescriptor.applySimpleConfig()}}. At the different places where
> {{CompressionParams.DEFAULT}} was used the code should call
> {{DatabaseDescriptor#getDefaultCompressionParams}} that should return some
> copy of configured {{CompressionParams}}.
> Some unit test using {{OverrideConfigurationLoader}} should be used to test
> that the table schema use the new default when a new table is created (see
> CreateTest for some example).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17773) Incorrect cassandra.logdir on Debian systems
[ https://issues.apache.org/jira/browse/CASSANDRA-17773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17693145#comment-17693145 ] Claude Warren commented on CASSANDRA-17773: --- This has been fixed in the latest update to the pull request. > Incorrect cassandra.logdir on Debian systems > > > Key: CASSANDRA-17773 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17773 > Project: Cassandra > Issue Type: Bug > Components: Packaging >Reporter: Eric Evans >Assignee: Claude Warren >Priority: Normal > Labels: lhf > Fix For: 3.0.x, 3.11.x, 4.0.x, 4.1.x, 4.x > > Time Spent: 6h 10m > Remaining Estimate: 0h > > The Debian packaging patches bin/cassandra to use /var/log/cassandra for > logs, it does so conditionally however, only if CASSANDRA_LOG_DIR is unset. > This occurs _after_ cassandra-env.sh is sourced though, which also sets > CASSANDRA_LOG_DIR if unset (to $CASSANDRA_HOME/logs). The result is that > -Dcassandra.lodir is set to /usr/share/cassandra/logs on Debian systems. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17773) Incorrect cassandra.logdir on Debian systems
[
https://issues.apache.org/jira/browse/CASSANDRA-17773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683410#comment-17683410
]
Claude Warren commented on CASSANDRA-17773:
---
Error at linke 25 of the new nodetool script.
it currently reads:
{{ENV_PRESERVE="$ENV_PRESERVE MAX_HEAP_SIZE JVM_OPTS"}}
but should read:
{{CASSANDRA_ENV_PRESERVE="$CASSENDRA_ENV_PRESERVE MAX_HEAP_SIZE JVM_OPTS"}}
> Incorrect cassandra.logdir on Debian systems
>
>
> Key: CASSANDRA-17773
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17773
> Project: Cassandra
> Issue Type: Bug
> Components: Packaging
>Reporter: Eric Evans
>Assignee: Claude Warren
>Priority: Normal
> Labels: lhf
> Fix For: 3.0.x, 3.11.x, 4.0.x, 4.1.x, 4.x
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> The Debian packaging patches bin/cassandra to use /var/log/cassandra for
> logs, it does so conditionally however, only if CASSANDRA_LOG_DIR is unset.
> This occurs _after_ cassandra-env.sh is sourced though, which also sets
> CASSANDRA_LOG_DIR if unset (to $CASSANDRA_HOME/logs). The result is that
> -Dcassandra.lodir is set to /usr/share/cassandra/logs on Debian systems.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-17773) Incorrect cassandra.logdir on Debian systems
[ https://issues.apache.org/jira/browse/CASSANDRA-17773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683339#comment-17683339 ] Claude Warren edited comment on CASSANDRA-17773 at 2/2/23 10:58 AM: I think we must be talking past each other. In the pull request the nodetool file shows the differences. ([https://github.com/apache/cassandra/pull/1950/files#diff-f7131d9a4f739662e81931739eff926bd82a3a856e9e8ed790ee1aca4458d98dL25-R26)] As you can see the new file is sourced as you suggest, though without the $CASSANDRA_HOME. This replaces a fair amount of boiler plate. The process is: # setup any required environment vars: ** $CASSANDRA_INCLUDE – (optional) ** $CASSANDRA_CONF ** $CLASSPATH ** $CASSANDRA_ENV_PRESERVE – (optional) a list of environment vars that should not be modified by cassandra-env.sh # source cassandra-conf.sh # set up any additional environment vars e.g. JMX_PORT and HEAP_SIZE in nodetool, or CASSANDRA_ENV_SHOW to debug other environment variables. # call validate_env passing any variables to be dumped during debug (e.g. JMX_PORT in nodetool). – validate_env defined in cassandra-conf.sh # call whatever program is executed by the script. validate_env function will validate that: * CASSANDRA_INCLUDE was set or located by default mechanism * CASSANDRA_HOME is set * CASSANDRA_LOG_DIR is set * if CASSANDRA_ENV_DEBUG is set the script will ** display the above variables ** display PWD, ** display he location of cassandra-conf.sh ** notify if numactl was not found ** dump the arguments to the validate_env function ** display the name and value of every variable listed in CASSANDRA_ENV_SHOW * if any required variable is not set validate_env will abort the script. This does not change the current program flow. It * Removes boiler plate * Adds debugging features to see what the environment looks like at runtime. was (Author: claudenw): I think we must be talking past each other. In the pull request the nodetool file shows the differences. ([https://github.com/apache/cassandra/pull/1950/files#diff-f7131d9a4f739662e81931739eff926bd82a3a856e9e8ed790ee1aca4458d98dL25-R26)] As you can see the new file is sourced as you suggest, though without the $CASSANDRA_HOME. This replaces a fair amount of boiler plate. The process is: # setup any required environment vars: ** $CASSANDRA_INCLUDE – (optional) ** $CASSANDRA_CONF ** $CLASSPATH ** $CASSANDRA_ENV_PRESERVE – (optional) a list of environment vars that should not be modified by cassandra-env.sh # source cassandra-conf.sh # set up any additional environment vars e.g. JMX_PORT and HEAP_SIZE in nodetool, or CASSANDRA_ENV_SHOW to debug other environment variables. # call validate_env passing any variables to be dumped during debug (e.g. JMX_PORT in nodetool). – validate_env defined in cassandra-conf.sh # call whatever program is executed by the script. validate_env function will validate that: * CASSANDRA_INCLUDE was set or located by default mechanism * CASSANDRA_HOME is set * CASSANDRA_LOG_DIR is set * if CASSANDRA_ENV_DEBUG is set the script will ** display the above variables ** display PWD, ** display he location of cassandra-conf.sh ** notify if numactl was not found ** dump the arguments to the validate_env function ** display the name and value of every variable listed in CASSANDRA_ENV_SHOW * if any required variable is not set validate_env will abort the script. > Incorrect cassandra.logdir on Debian systems > > > Key: CASSANDRA-17773 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17773 > Project: Cassandra > Issue Type: Bug > Components: Packaging >Reporter: Eric Evans >Assignee: Claude Warren >Priority: Normal > Labels: lhf > Fix For: 3.0.x, 3.11.x, 4.0.x, 4.1.x, 4.x > > Time Spent: 2h 10m > Remaining Estimate: 0h > > The Debian packaging patches bin/cassandra to use /var/log/cassandra for > logs, it does so conditionally however, only if CASSANDRA_LOG_DIR is unset. > This occurs _after_ cassandra-env.sh is sourced though, which also sets > CASSANDRA_LOG_DIR if unset (to $CASSANDRA_HOME/logs). The result is that > -Dcassandra.lodir is set to /usr/share/cassandra/logs on Debian systems. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17773) Incorrect cassandra.logdir on Debian systems
[ https://issues.apache.org/jira/browse/CASSANDRA-17773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683339#comment-17683339 ] Claude Warren commented on CASSANDRA-17773: --- I think we must be talking past each other. In the pull request the nodetool file shows the differences. ([https://github.com/apache/cassandra/pull/1950/files#diff-f7131d9a4f739662e81931739eff926bd82a3a856e9e8ed790ee1aca4458d98dL25-R26)] As you can see the new file is sourced as you suggest, though without the $CASSANDRA_HOME. This replaces a fair amount of boiler plate. The process is: # setup any required environment vars: ** $CASSANDRA_INCLUDE – (optional) ** $CASSANDRA_CONF ** $CLASSPATH ** $CASSANDRA_ENV_PRESERVE – (optional) a list of environment vars that should not be modified by cassandra-env.sh # source cassandra-conf.sh # set up any additional environment vars e.g. JMX_PORT and HEAP_SIZE in nodetool, or CASSANDRA_ENV_SHOW to debug other environment variables. # call validate_env passing any variables to be dumped during debug (e.g. JMX_PORT in nodetool). – validate_env defined in cassandra-conf.sh # call whatever program is executed by the script. validate_env function will validate that: * CASSANDRA_INCLUDE was set or located by default mechanism * CASSANDRA_HOME is set * CASSANDRA_LOG_DIR is set * if CASSANDRA_ENV_DEBUG is set the script will ** display the above variables ** display PWD, ** display he location of cassandra-conf.sh ** notify if numactl was not found ** dump the arguments to the validate_env function ** display the name and value of every variable listed in CASSANDRA_ENV_SHOW * if any required variable is not set validate_env will abort the script. > Incorrect cassandra.logdir on Debian systems > > > Key: CASSANDRA-17773 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17773 > Project: Cassandra > Issue Type: Bug > Components: Packaging >Reporter: Eric Evans >Assignee: Claude Warren >Priority: Normal > Labels: lhf > Fix For: 3.0.x, 3.11.x, 4.0.x, 4.1.x, 4.x > > Time Spent: 2h 10m > Remaining Estimate: 0h > > The Debian packaging patches bin/cassandra to use /var/log/cassandra for > logs, it does so conditionally however, only if CASSANDRA_LOG_DIR is unset. > This occurs _after_ cassandra-env.sh is sourced though, which also sets > CASSANDRA_LOG_DIR if unset (to $CASSANDRA_HOME/logs). The result is that > -Dcassandra.lodir is set to /usr/share/cassandra/logs on Debian systems. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17773) Incorrect cassandra.logdir on Debian systems
[ https://issues.apache.org/jira/browse/CASSANDRA-17773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683303#comment-17683303 ] Claude Warren commented on CASSANDRA-17773: --- No, the intention here was for the nodetool to be an example of how to use the script and the changes it brings. My plan is to update all relevant script to use the new file. Rather than do all the tools first I wanted to provide the new file and an example to get feedback before proceeding. It would also be nice to get feedback regarding how to proceed. I see options being: # Do all the changes in one go. # accept this pull request and do other changes in some sort of grouping (not sure what that grouping would be but...) # accept this pull request and do the other changes as individual pull requests. There is probably a middle ground as well. > Incorrect cassandra.logdir on Debian systems > > > Key: CASSANDRA-17773 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17773 > Project: Cassandra > Issue Type: Bug > Components: Packaging >Reporter: Eric Evans >Assignee: Claude Warren >Priority: Normal > Labels: lhf > Fix For: 3.0.x, 3.11.x, 4.0.x, 4.1.x, 4.x > > Time Spent: 2h 10m > Remaining Estimate: 0h > > The Debian packaging patches bin/cassandra to use /var/log/cassandra for > logs, it does so conditionally however, only if CASSANDRA_LOG_DIR is unset. > This occurs _after_ cassandra-env.sh is sourced though, which also sets > CASSANDRA_LOG_DIR if unset (to $CASSANDRA_HOME/logs). The result is that > -Dcassandra.lodir is set to /usr/share/cassandra/logs on Debian systems. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-18198) "AttributeError: module 'py' has no attribute 'io'" reported in multiple tests
[
https://issues.apache.org/jira/browse/CASSANDRA-18198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claude Warren reassigned CASSANDRA-18198:
-
Assignee: Brandon Williams
> "AttributeError: module 'py' has no attribute 'io'" reported in multiple tests
> --
>
> Key: CASSANDRA-18198
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18198
> Project: Cassandra
> Issue Type: Bug
>Reporter: Claude Warren
>Assignee: Brandon Williams
>Priority: Normal
>
> {{title = 'Timeout'}}
> {{stream = <_io.TextIOWrapper name='' mode='w' encoding='utf-8'>}}
> {{{}sep = '+'{}}}{{{}def write_title(title, stream=None, sep="~"):{}}}
> {{{}"""Write a section title.{}}}{{{}If *stream* is None sys.stderr will be
> used, *sep* is used to{}}}
> {{draw the line.}}
> {{"""}}
> {{if stream is None:}}
> {{stream = sys.stderr}}
> {{> width = py.io.get_terminal_width()}}
> {{E AttributeError: module 'py' has no attribute 'io}}
>
> is reported in multiple tests as noted below.
> possibly a class loader issue associated with CASSANDRA-18150
> 4.1
> [https://ci-cassandra.apache.org/job/Cassandra-4.1/256/testReport/dtest-offheap.repair_tests.incremental_repair_test/TestIncRepair/test_multiple_full_repairs_lcs]
> 3.11
> [https://ci-cassandra.apache.org/job/Cassandra-3.11/424/testReport/dtest.bootstrap_test/TestBootstrap/test_simultaneous_bootstrap/]
> 3.0
> [https://ci-cassandra.apache.org/job/Cassandra-3.0/328/testReport/dtest-upgrade.upgrade_tests.upgrade_supercolumns_test/TestSCUpgrade/test_upgrade_super_columns_through_all_versions/]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-18198) "AttributeError: module 'py' has no attribute 'io'" reported in multiple tests
Claude Warren created CASSANDRA-18198:
-
Summary: "AttributeError: module 'py' has no attribute 'io'"
reported in multiple tests
Key: CASSANDRA-18198
URL: https://issues.apache.org/jira/browse/CASSANDRA-18198
Project: Cassandra
Issue Type: Bug
Reporter: Claude Warren
{{title = 'Timeout'}}
{{stream = <_io.TextIOWrapper name='' mode='w' encoding='utf-8'>}}
{{{}sep = '+'{}}}{{{}def write_title(title, stream=None, sep="~"):{}}}
{{{}"""Write a section title.{}}}{{{}If *stream* is None sys.stderr will be
used, *sep* is used to{}}}
{{draw the line.}}
{{"""}}
{{if stream is None:}}
{{stream = sys.stderr}}
{{> width = py.io.get_terminal_width()}}
{{E AttributeError: module 'py' has no attribute 'io}}
is reported in multiple tests as noted below.
possibly a class loader issue associated with CASSANDRA-18150
4.1
[https://ci-cassandra.apache.org/job/Cassandra-4.1/256/testReport/dtest-offheap.repair_tests.incremental_repair_test/TestIncRepair/test_multiple_full_repairs_lcs]
3.11
[https://ci-cassandra.apache.org/job/Cassandra-3.11/424/testReport/dtest.bootstrap_test/TestBootstrap/test_simultaneous_bootstrap/]
3.0
[https://ci-cassandra.apache.org/job/Cassandra-3.0/328/testReport/dtest-upgrade.upgrade_tests.upgrade_supercolumns_test/TestSCUpgrade/test_upgrade_super_columns_through_all_versions/]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18191) Native Transport SSL tests failing
[
https://issues.apache.org/jira/browse/CASSANDRA-18191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claude Warren updated CASSANDRA-18191:
--
Summary: Native Transport SSL tests failing (was:
TestNativeTransportSSL.test_connect_to_ssl test failing)
> Native Transport SSL tests failing
> --
>
> Key: CASSANDRA-18191
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18191
> Project: Cassandra
> Issue Type: Bug
>Reporter: Claude Warren
>Assignee: Brandon Williams
>Priority: Normal
>
>
> Cassandra 3.0
>
> TestNativeTransportSSL.test_connect_to_ssl and
> TestNativeTransportSSL.test_connect_to_ssl (novnode)
> [https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-3.0/failure/native_transport_ssl_test/TestNativeTransportSSL/test_connect_to_ssl]
> As well as
> TestNativeTransportSSL.test_connect_to_ssl_optional and
> TestNativeTransportSSL.test_connect_to_ssl_optional (nvnode)
> [https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-3.0/failure/native_transport_ssl_test/TestNativeTransportSSL/test_use_custom_ssl_port]
> are failing after 2 merges that don't seem like they would impact SSL or
> server availaibility.
> Most recent error message is:
> cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers',
> {'127.0.0.1:9042': PermissionError(1, "Tried connecting to [('127.0.0.1',
> 9042)]. Last error: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert
> handshake failure (_ssl.c:992)")}
> )
>
> Most recent changes made by
> [@smiklosovic|https://github.com/apache/cassandra/commits?author=smiklosovic]
> and [@driftx|https://github.com/apache/cassandra/commits?author=driftx] .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18191) TestNativeTransportSSL.test_connect_to_ssl test failing
[
https://issues.apache.org/jira/browse/CASSANDRA-18191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claude Warren updated CASSANDRA-18191:
--
Description:
Cassandra 3.0
TestNativeTransportSSL.test_connect_to_ssl and
TestNativeTransportSSL.test_connect_to_ssl (novnode)
[https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-3.0/failure/native_transport_ssl_test/TestNativeTransportSSL/test_connect_to_ssl]
As well as
TestNativeTransportSSL.test_connect_to_ssl_optional and
TestNativeTransportSSL.test_connect_to_ssl_optional (nvnode)
[https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-3.0/failure/native_transport_ssl_test/TestNativeTransportSSL/test_use_custom_ssl_port]
are failing after 2 merges that don't seem like they would impact SSL or server
availaibility.
Most recent error message is:
cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers',
{'127.0.0.1:9042': PermissionError(1, "Tried connecting to [('127.0.0.1',
9042)]. Last error: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake
failure (_ssl.c:992)")}
)
Most recent changes made by
[@smiklosovic|https://github.com/apache/cassandra/commits?author=smiklosovic]
and [@driftx|https://github.com/apache/cassandra/commits?author=driftx] .
was:
Cassandra 3.0
[TestNativeTransportSSL.test_connect_to_ssl
|https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-3.0/failure/native_transport_ssl_test/TestNativeTransportSSL/test_connect_to_ssl]
and
[TestNativeTransportSSL.test_connect_to_ssl (novnode)
|https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-3.0/failure/native_transport_ssl_test/TestNativeTransportSSL/test_connect_to_ssl]
TestNativeTransportSSL.test_connect_to_ssl (novnode)
are both failing after 2 merges that don't seem like they would impact SSL or
server availaibility.
Most recent error message is:
cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers',
{'127.0.0.1:9042': PermissionError(1, "Tried connecting to [('127.0.0.1',
9042)]. Last error: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake
failure (_ssl.c:992)")})
Most recent changes made by
[@smiklosovic|https://github.com/apache/cassandra/commits?author=smiklosovic]
and [@driftx|https://github.com/apache/cassandra/commits?author=driftx] .
> TestNativeTransportSSL.test_connect_to_ssl test failing
> ---
>
> Key: CASSANDRA-18191
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18191
> Project: Cassandra
> Issue Type: Bug
>Reporter: Claude Warren
>Assignee: Brandon Williams
>Priority: Normal
>
>
> Cassandra 3.0
>
> TestNativeTransportSSL.test_connect_to_ssl and
> TestNativeTransportSSL.test_connect_to_ssl (novnode)
> [https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-3.0/failure/native_transport_ssl_test/TestNativeTransportSSL/test_connect_to_ssl]
> As well as
> TestNativeTransportSSL.test_connect_to_ssl_optional and
> TestNativeTransportSSL.test_connect_to_ssl_optional (nvnode)
> [https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-3.0/failure/native_transport_ssl_test/TestNativeTransportSSL/test_use_custom_ssl_port]
> are failing after 2 merges that don't seem like they would impact SSL or
> server availaibility.
> Most recent error message is:
> cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers',
> {'127.0.0.1:9042': PermissionError(1, "Tried connecting to [('127.0.0.1',
> 9042)]. Last error: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert
> handshake failure (_ssl.c:992)")}
> )
>
> Most recent changes made by
> [@smiklosovic|https://github.com/apache/cassandra/commits?author=smiklosovic]
> and [@driftx|https://github.com/apache/cassandra/commits?author=driftx] .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-18191) TestNativeTransportSSL.test_connect_to_ssl test failing
Claude Warren created CASSANDRA-18191:
-
Summary: TestNativeTransportSSL.test_connect_to_ssl test failing
Key: CASSANDRA-18191
URL: https://issues.apache.org/jira/browse/CASSANDRA-18191
Project: Cassandra
Issue Type: Bug
Reporter: Claude Warren
Assignee: Brandon Williams
Cassandra 3.0
[TestNativeTransportSSL.test_connect_to_ssl
|https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-3.0/failure/native_transport_ssl_test/TestNativeTransportSSL/test_connect_to_ssl]
and
[TestNativeTransportSSL.test_connect_to_ssl (novnode)
|https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-3.0/failure/native_transport_ssl_test/TestNativeTransportSSL/test_connect_to_ssl]
TestNativeTransportSSL.test_connect_to_ssl (novnode)
are both failing after 2 merges that don't seem like they would impact SSL or
server availaibility.
Most recent error message is:
cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers',
{'127.0.0.1:9042': PermissionError(1, "Tried connecting to [('127.0.0.1',
9042)]. Last error: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake
failure (_ssl.c:992)")})
Most recent changes made by
[@smiklosovic|https://github.com/apache/cassandra/commits?author=smiklosovic]
and [@driftx|https://github.com/apache/cassandra/commits?author=driftx] .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18143) upgradesstables does not always upgrade tables in proper order.
[ https://issues.apache.org/jira/browse/CASSANDRA-18143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17676257#comment-17676257 ] Claude Warren commented on CASSANDRA-18143: --- Pull request is at https://github.com/apache/cassandra/pull/2081 > upgradesstables does not always upgrade tables in proper order. > --- > > Key: CASSANDRA-18143 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18143 > Project: Cassandra > Issue Type: Bug > Components: Tool/sstable >Reporter: Claude Warren >Assignee: Claude Warren >Priority: Normal > > The SSTableUpgrader accepts tools in the hash order provided by > Directories.SSTableLister rather than ordering them to ensure that they are > upgraded in the proper order. > They should be ordered by their id. The comparator for SSTableId is available > in SSTableIdFactory.COMPARATOR. > > Dev discussion thread: > https://lists.apache.org/thread/w6pm5hbdxt295mtvlckv0joyk8x4o8nf -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-18143) upgradesstables does not always upgrade tables in proper order.
Claude Warren created CASSANDRA-18143: - Summary: upgradesstables does not always upgrade tables in proper order. Key: CASSANDRA-18143 URL: https://issues.apache.org/jira/browse/CASSANDRA-18143 Project: Cassandra Issue Type: Bug Components: Tool/sstable Reporter: Claude Warren Assignee: Claude Warren The SSTableUpgrader accepts tools in the hash order provided by Directories.SSTableLister rather than ordering them to ensure that they are upgraded in the proper order. They should be ordered by their id. The comparator for SSTableId is available in SSTableIdFactory.COMPARATOR. Dev discussion thread: https://lists.apache.org/thread/w6pm5hbdxt295mtvlckv0joyk8x4o8nf -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18012) Remove -l / -m / -h designation and have two options: free or paid tier circle config
[ https://issues.apache.org/jira/browse/CASSANDRA-18012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17629829#comment-17629829 ] Claude Warren commented on CASSANDRA-18012: --- >From the description it seems like all that needs to happen is to remove the >-h and -f from the script. This would include removing the check for $hires >and $check_env_vars variables as the code blocks associated with same. This means removing the code block that setup the hires yml file. Should the config-2_1.yml.high_res.patch and the config.yml.HIRES files be removed as well? > Remove -l / -m / -h designation and have two options: free or paid tier > circle config > - > > Key: CASSANDRA-18012 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18012 > Project: Cassandra > Issue Type: Task > Components: CI >Reporter: Josh McKenzie >Priority: Normal > > Currently the -h designation is wasteful and should not be used, and the -f > designation won't actually successfully run to completion. > We should default to a "free tier" config (probably print a warning that it's > generating config w/subset of full tests that should not be used to validate > commits in big warning letters) that runs a subset of the test suites for > users working on patches to validate their work (unit test only? unit + > in-jvm dtest? TBD), and add a flag to generate a config using larger > containers + parallelization for the "paid" tier for folks w/paid circleci > accounts., > It looks like -p is available right now fwiw. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17773) Incorrect cassandra.logdir on Debian systems
[ https://issues.apache.org/jira/browse/CASSANDRA-17773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625147#comment-17625147 ] Claude Warren commented on CASSANDRA-17773: --- I have created a pull request as an example of how this could be made better. This is a work in progress and I am looking for comments. https://github.com/apache/cassandra/pull/1950 > Incorrect cassandra.logdir on Debian systems > > > Key: CASSANDRA-17773 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17773 > Project: Cassandra > Issue Type: Bug > Components: Packaging >Reporter: Eric Evans >Priority: Normal > Labels: lhf > Fix For: 3.0.x, 3.11.x, 4.0.x, 4.1.x > > > The Debian packaging patches bin/cassandra to use /var/log/cassandra for > logs, it does so conditionally however, only if CASSANDRA_LOG_DIR is unset. > This occurs _after_ cassandra-env.sh is sourced though, which also sets > CASSANDRA_LOG_DIR if unset (to $CASSANDRA_HOME/logs). The result is that > -Dcassandra.lodir is set to /usr/share/cassandra/logs on Debian systems. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-17978) Aiven offerings on ecosystem page is outdated.
Claude Warren created CASSANDRA-17978: - Summary: Aiven offerings on ecosystem page is outdated. Key: CASSANDRA-17978 URL: https://issues.apache.org/jira/browse/CASSANDRA-17978 Project: Cassandra Issue Type: Task Components: Documentation/Website Reporter: Claude Warren Aiven has updated their offering to 4.0.4 and they have a backup tool, Astacus, that should be listed in the tools section -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17773) Incorrect cassandra.logdir on Debian systems
[
https://issues.apache.org/jira/browse/CASSANDRA-17773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17619358#comment-17619358
]
Claude Warren commented on CASSANDRA-17773:
---
The problem in this case was an old {{cassandra-env.sh}} (which does not get
patched). But it could just as easily been a {{cassandra.in.sh}} script.
It seem to me that there is at least 1 and possibly more environment vars that
are critical to understanding how the environment is configured and have
default values. I think that perhaps we should have a block of script that
checks the set value against the default and logs a warning (similar to mck's
example above) whenever they don't match. This will provide a warning in cases
where it is accidental and confirmation in cases where it is desired.
I also note that not all scripts include {{cassandra-env.sh }}only
{{cassandra}}, {{debug-cql}}, and {{nodetool}} do.
The CASSANDRA_INCLUDE block is found in {{cassandra}}, {{debug-cql}},
{{nodetool}}, {{sstableloader}}, {{sstablescrub}}, {{sstableupgrade}},
{{sstableutil}}, {{sstableverify}}.
It seems like this should be consistent.
I think that an include file (cassandra-conf) in the CASSANDRA_HOME/bin
directory containing:
{noformat}
CASSANDRA_CONF=Y
if [ "x$CASSANDRA_HOME" = "x" ]; then
CASSANDRA_HOME="`dirname "$0"`/.."
fi
# The directory where Cassandra's configs live (required)
if [ "x$CASSANDRA_CONF" = "x" ]; then
CASSANDRA_CONF="$CASSANDRA_HOME/conf"
fi
if [ "x$CASSANDRA_INCLUDE" = "x" ]; then
# Locations (in order) to use when searching for an include file.
for include in "`dirname "$0"`/cassandra.in.sh" \
"$HOME/.cassandra.in.sh" \
/usr/share/cassandra/cassandra.in.sh \
/usr/local/share/cassandra/cassandra.in.sh \
/opt/cassandra/cassandra.in.sh; do
if [ -r "$include" ]; then
CASSANDRA_INCLUDE=$include
. "$include"
break
fi
done
# ...otherwise, source the specified include.
elif [ -r "$CASSANDRA_INCLUDE" ]; then
. "$CASSANDRA_INCLUDE"
fi
if [ -z "$CASSANDRA_CONF " ]; then
echo "You must set the CASSANDRA_CONF var" >&2
exit 1
fi
## Add variable checks here like
if [ -z "$CASSANDRA_LOG_DIR" ]; then
echo "CASSANDRA_LOG_DIR should have been set to a default value in
$CASSANDRA_INCLUDE some log files will be found elsewhere, please fix
$CASSANDRA_INCLUDE"
CASSANDRA_LOG_DIR=$CASSANDRA_HOME/logs
fi
{noformat}
Finally, modify conf/cassandra-env.sh to change:
{noformat}
# Sets the path where logback and GC logs are written.
if [ "x$CASSANDRA_LOG_DIR" = "x" ] ; then
CASSANDRA_LOG_DIR="$CASSANDRA_HOME/logs"
fi
{noformat}
to
{noformat}
# Sets the path where logback and GC logs are written.
if [ "x$CASSANDRA_CONF" = "x" ] ; then
. ../bin/cassandra-conf
fi
{noformat}
> Incorrect cassandra.logdir on Debian systems
>
>
> Key: CASSANDRA-17773
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17773
> Project: Cassandra
> Issue Type: Bug
> Components: Packaging
>Reporter: Eric Evans
>Priority: Normal
> Labels: lhf
> Fix For: 3.0.x, 3.11.x, 4.0.x, 4.1-rc
>
>
> The Debian packaging patches bin/cassandra to use /var/log/cassandra for
> logs, it does so conditionally however, only if CASSANDRA_LOG_DIR is unset.
> This occurs _after_ cassandra-env.sh is sourced though, which also sets
> CASSANDRA_LOG_DIR if unset (to $CASSANDRA_HOME/logs). The result is that
> -Dcassandra.lodir is set to /usr/share/cassandra/logs on Debian systems.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
