Re: [DISCUSS] CEP-42: Constraints Framework

[Claude Warren, Jr via dev]

>
> 2)
> Is part of an enum is somehow suplying the lack of enum types. Constraint
> could be something like CONSTRAINT belongsToEnum([list of valid values],
> field):
> CREATE TABLE keyspace.table (
>   field text CONSTRAINT belongsToEnum(['foo', 'foo2'], field),
>   ...
> );
> 3)
> Similarly, we can check and reject if a term is part of a list of blocked
> terms:
> CREATE TABLE keyspace.table (
>   field text CONSTRAINT isNotBlocked(['blocked_foo', 'blocked_foo2'],
> field),
>   ...
> );


Are these not just "CONSTRAINT inList([List of valid values], field);"  and
"CONSTRAINT not inList([List of valid values], field);"?
At this point doesn't "CONSTRAINT p1 != p2" devolve to "CONSTRAINT not
inList([p1], p2);"?

Can "[List of values]" point to a variable containing a list?  Or does it
require hard coding in the constraint itself?



On Tue, Jun 11, 2024 at 6:23 PM Bernardo Botella <
conta...@bernardobotella.com> wrote:

> Hi Štephan
>
> I'll address the different points:
> 1)
> An example (possibly a stretch) of use case for != constraint would be:
> Let's say you have a table in which you want to record a movement, from
> position p1 to position p2. You may want to check that those two are
> different to make sure there is actual movement.
>
> CREATE TABLE keyspace.table (
>   p1 int,
>   p2 int,
>   ...,
>   CONSTRAINT p1 != p2
> );
>
> For the case of ==, I agree that it is harder to come up with a valid use
> case, and I added it for completion.
>
> 2)
> Is part of an enum is somehow suplying the lack of enum types. Constraint
> could be something like CONSTRAINT belongsToEnum([list of valid values],
> field):
> CREATE TABLE keyspace.table (
>   field text CONSTRAINT belongsToEnum(['foo', 'foo2'], field),
>   ...
> );
>
> 3)
> Similarly, we can check and reject if a term is part of a list of blocked
> terms:
> CREATE TABLE keyspace.table (
>   field text CONSTRAINT isNotBlocked(['blocked_foo', 'blocked_foo2'],
> field),
>   ...
> );
>
> Please let me know if this helps,
> Bernardo
>
>
>
> On Jun 11, 2024, at 6:29 AM, Štefan Miklošovič <
> stefan.mikloso...@gmail.com> wrote:
>
> Hi Bernardo,
>
> 1) Could you elaborate on these two constraints?
>
> == and != ?
>
> What is the use case? Why would I want to have data in a database stored
> in some column which would need to be _same as my constraint_ and which
> _could not_ be same as my constraint? Can you give me at least one example
> of each? It looks like I am going to put a constant into a database in case
> of ==, wouldn't a static column be better?
>
> 2) For examples of text based types you mentioned: "is part of an enum" -
> how would you enforce this in Cassandra? What enum do we have in CQL?
> 3) What does "is it block listed" mean?
>
> In the meanwhile, I made changes to CEP-24 to move transactionality into
> optional features.
>
> On Tue, Jun 11, 2024 at 12:18 AM Bernardo Botella <
> conta...@bernardobotella.com> wrote:
>
>> Hi everyone,
>>
>> After the feedback, I'd like to make a recap of what we have discussed in
>> this thread and try to move forward with the conversation.
>>
>> I made some clarifications:
>> - Constraints are only applied at write time.
>> - Guardrail configurations should maintain preference over what's being
>> defined as a constraint.
>>
>> *Specify constraints:*
>> There is a general feedback around adding more concrete examples than the
>> ones that can be found on the CEP document.
>> Basically, the initial constraints I am proposing are:
>> - SizeOf Constraint for String types, as in
>> name text CONSTRAINT sizeOf(name) < 256
>>
>> - Value Constraint for numeric types
>> number_of_items int CONSTRAINT number_of_items < 1000
>>
>> Those two alone and combined provide a lot of flexibility, and allow
>> complex validations that enable "new types" such as:
>>
>> CREATE TYPE keyspace.cidr_address_ipv4 (
>>   ip_adress inet,
>>   subnet_mask int,
>>   CONSTRAINT subnet_mask > 0,
>>   CONSTRAINT subnet_mask < 32
>> )
>>
>> CREATE TYPE keyspace.color (
>>   r int,
>>   g int,
>>   b int,
>>   CONSTRAINT r >= 0,
>>   CONSTRAINT r < 255,
>>   CONSTRAINT g >= 0,
>>   CONSTRAINT g < 255,
>>   CONSTRAINT b >= 0,
>>   CONSTRAINT b < 255,
>> )
>>
>>
>> Those two initial Constraints are de fundamental constraints that would
>> give value to the feature. The framework can (and will) be extended with
>> other Constraints, leaving us with the following:
>>
>> For numeric types:
>> - Max (<)
>> - Min (>)
>> - Equality ( = = )
>> - Difference (!=)
>>
>> For date types:
>> - Before (<)
>> - After (>)
>>
>> For text based types:
>> - Size (sizeOf)
>> - isJson (is the text a json?)
>> - complies with a given pattern
>> - Is it block listed?
>> - Is it part of an enum?
>>
>> General table constraints (including more than one column):
>> - Compare between numeric types (a < b, a > b, a != b, …)
>> - Compare between date types (date1 < date2, date1>date2, date1!=date2, …)
>>
>> I have updated the CEP with this information.
>>
>> *Potential 

Re: [DISCUSS] CEP-40: Data Transfer Using Cassandra Sidecar for Live Migrating Instances

[Claude Warren, Jr via dev]

Alex,

 you write:

> We can implement CEP-40 using a similar approach: we can leave the source
> node as both a read and write target, and allow the new node to be a target
> for (pending) writes. Unfortunately, this does not help with availability
> (in fact, it decreases write availability, since we will have to collect
> 2+1 mandatory write responses instead of just 2), but increases durability,
> and I think helps to fully eliminate the second phase. This also increases
> read availability when the source node is up, since we can still use the
> source node as a part of read quorum.


Would it be possible to create a new type of write target node?  The new
write target node is notified of writes (like any other write node) but
does not participate in the write availability calculation.  In this way a
node this is being migrated to could receive writes and have minimal impact
on the current operation of the cluster?

Claude



On Wed, May 1, 2024 at 12:33 PM Alex Petrov  wrote:

> Thank you for submitting this CEP!
>
> Wanted to discuss this point from the description:
>
> > How to bring up/down Cassandra/Sidecar instances or making/applying
> config changes are outside the scope of this document.
>
> One advantage of doing migration via sidecar is the fact that we can
> stream sstables to the target node from the source node while the source
> node is down. Also if the source node is down, it does not matter if we
> can’t use it as a write target However, if we are replacing a live node, we
> do lose both durability and availability during the second copy phase.
> There are copious other advantages described by others in the thread above.
>
> For example, we have three adjacent nodes A,B,C and simple RF 3. C
> (source) is up and is being replaced with live-migrated D (destination).
> According to the described process in CEP-40, we perform streaming in 2
> phases: first one is a full copy (similar to bootstrap/replacement in
> cassandra), and the second one is just a diff. The second phase is still
> going to take a non-trivial amount of time, and is likely to last at very
> least minutes. During this time, we only have nodes A and B as both read
> and write targets, with no alternatives: we have to have both of them
> present for any operation, and losing either one of them leaves us with
> only one copy of data.
>
> To contrast this, TCM bootstrap process is 4-step: between the old owner
> being phased out and the new owner brought in, we always ensure r/w quorum
> consistency and liveness of at least 2 nodes for the read quorum, 3 nodes
> available for reads in best case, and 2+1 pending replica for the write
> quorum, with 4 nodes (3 existing owners + 1 pending) being available for
> writes in best case. Replacement in TCM is implemented similarly, with the
> old node remaining an (unavailable) read target, but new node already being
> the target for (pending) writes.
>
> We can implement CEP-40 using a similar approach: we can leave the source
> node as both a read and write target, and allow the new node to be a target
> for (pending) writes. Unfortunately, this does not help with availability
> (in fact, it decreases write availability, since we will have to collect
> 2+1 mandatory write responses instead of just 2), but increases durability,
> and I think helps to fully eliminate the second phase. This also increases
> read availability when the source node is up, since we can still use the
> source node as a part of read quorum.
>
> I think if we want to call this feature "live migration", since this term
> is used in hypervisor community to describe an instant and uninterrupted
> instance migration from one host to the other without guest instance being
> able to notice as much as the time jump, we may want to provide similar
> guarantees.
>
> I am also not against to have this to be done post-factum, after
> implementation of CEP in its current form, but I think it would be good to
> have good understanding of availability and durability guarantees we want
> to provide with it, and have it stated explicitly, for both "source node
> down" and "source node up" cases. That said, since we will have to
> integrate CEP-40 with TCM, and will have to ensure correctness of sstable
> diffing for the second phase, it might make sense to consider reusing some
> of the existing replacement logic from TCM. Just to make sure this is
> mentioned explicitly, my proposal is only concerned with the second copy
> phase, without any implications about the first.
>
> Thank you,
> --Alex
>
> On Fri, Apr 5, 2024, at 12:46 PM, Venkata Hari Krishna Nukala wrote:
>
> Hi all,
>
> I have filed CEP-40 [1] for live migrating Cassandra instances using the
> Cassandra Sidecar.
>
> When someone needs to move all or a portion of the Cassandra nodes
> belonging to a cluster to different hosts, the traditional approach of
> Cassandra node replacement can be time-consuming due to repairs and the
> bootstrapping of new nodes. 

Re: discuss: add to_human_size function

[Claude Warren, Jr via dev]

TiB is not yet in DataStorageSpec (perhaps we should add it).

A quick review tells me that all the units are unique across the 3 specs.
As long as we guarantee that in the future the method you propose should be
easily expandable to the other specs.

+1 to this idea.

On Thu, Apr 25, 2024 at 12:26 PM Štefan Miklošovič <
stefan.mikloso...@gmail.com> wrote:

> That is a very interesting point, Claude. My so-far implementation is
> using FileUtils.stringifyFileSize which is just dividing a value by a
> respective divisor based on how big a value is. While this works, it will
> prevent you from specifying what unit you want that value to be converted
> to as well as it will prevent you from specifying what unit a value you
> provided is of. So, for example, if a column is known to be in kibibytes
> and we want that to be converted into gibibytes, that won't be possible
> because that function will think that a value is in bytes.
>
> It would be more appropriate to have something like this:
>
> to_human_size(val) -> alias to FileUtils.stringifyFileSize, without any
> source nor target unit, it will consider it to be in bytes and it will
> convert it like in FileUtils.stringifyFileSize
>
> to_human_size(val, 'MiB') -> alias for to_human_size(val, 'B', 'MiB')
> to_human_size(val, 'GiB') -> alias for to_human_size(val, 'B', 'GiB')
>
> the first argument is the source unit, the second argument is target unit
>
> to_human_size(val, 'B', 'MiB')
> to_human_size(val, 'B', 'GiB')
> to_human_size(val, 'KiB', 'GiB')
> to_human_size(val, 'KiB', 'TiB')
>
> I think this is more flexible and we should funnel this via
> DataStorageSpec and similar as you mentioned.
>
> In the future, we might also add to_human_duration which would be
> implemented against DurationSpec so similar conversions are possible.
>
> On Fri, Apr 19, 2024 at 10:53 AM Claude Warren, Jr via dev <
> dev@cassandra.apache.org> wrote:
>
>> I like the idea.  Is the intention to have the of the function be
>> parsable by the config  parsers like DataRateSpec, DataStorageSpec, or
>> DurationSpec?
>>
>> Claude
>>
>> On Thu, Apr 18, 2024 at 9:47 PM Ariel Weisberg  wrote:
>>
>>> Hi,
>>>
>>> I think it’s a good quality of life improvement, but I am someone who
>>> believes in a rich set of built-in functions being a good thing.
>>>
>>> A format function is a bit more scope and kind of orthogonal. It would
>>> still be good to have shorthand functions for things like size.
>>>
>>> Ariel
>>>
>>> On Tue, Apr 9, 2024, at 8:09 AM, Štefan Miklošovič wrote:
>>>
>>> Hi,
>>>
>>> I want to propose CASSANDRA-19546. It would be possible to convert raw
>>> numbers to something human-friendly.
>>> There are cases when we write just a number of bytes in our system
>>> tables but these numbers are just hard to parse visually. Users can indeed
>>> use this for their tables too if they find it useful.
>>>
>>> Also, a user can indeed write a UDF for this but I would prefer if we
>>> had something baked in.
>>>
>>> Does this make sense to people? Are there any other approaches to do
>>> this?
>>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-19546
>>> https://github.com/apache/cassandra/pull/3239/files
>>>
>>> Regards
>>>
>>>
>>>

Re: discuss: add to_human_size function

[Claude Warren, Jr via dev]

I like the idea.  Is the intention to have the of the function be parsable
by the config  parsers like DataRateSpec, DataStorageSpec, or DurationSpec?

Claude

On Thu, Apr 18, 2024 at 9:47 PM Ariel Weisberg  wrote:

> Hi,
>
> I think it’s a good quality of life improvement, but I am someone who
> believes in a rich set of built-in functions being a good thing.
>
> A format function is a bit more scope and kind of orthogonal. It would
> still be good to have shorthand functions for things like size.
>
> Ariel
>
> On Tue, Apr 9, 2024, at 8:09 AM, Štefan Miklošovič wrote:
>
> Hi,
>
> I want to propose CASSANDRA-19546. It would be possible to convert raw
> numbers to something human-friendly.
> There are cases when we write just a number of bytes in our system tables
> but these numbers are just hard to parse visually. Users can indeed use
> this for their tables too if they find it useful.
>
> Also, a user can indeed write a UDF for this but I would prefer if we had
> something baked in.
>
> Does this make sense to people? Are there any other approaches to do this?
>
> https://issues.apache.org/jira/browse/CASSANDRA-19546
> https://github.com/apache/cassandra/pull/3239/files
>
> Regards
>
>
>

Re: [DISCUSS] CEP-40: Data Transfer Using Cassandra Sidecar for Live Migrating Instances

[Claude Warren, Jr via dev]

I think this solution would solve one of the problems that Aiven has with
node replacement currently.  Though TCM will probably help as well.

On Mon, Apr 15, 2024 at 11:47 PM German Eichberger via dev <
dev@cassandra.apache.org> wrote:

> Thanks for the proposal. I second Jordan that we need more abstraction in
> (1), e.g. most cloud provider allow for disk snapshots and starting nodes
> from a snapshot which would be a good mechanism if you find yourself there.
>
> German
> --
> *From:* Jordan West 
> *Sent:* Sunday, April 14, 2024 12:27 PM
> *To:* dev@cassandra.apache.org 
> *Subject:* [EXTERNAL] Re: [DISCUSS] CEP-40: Data Transfer Using Cassandra
> Sidecar for Live Migrating Instances
>
> Thanks for proposing this CEP! We have something like this internally so I
> have some familiarity with the approach and the challenges. After reading
> the CEP a couple things come to mind:
>
> 1. I would like to see more abstraction of how the files get moved / put
> in place with the proposed solution being the default implementation. That
> would allow others to plug in alternatives means of data movement like
> pulling down backups from S3 or rsync, etc.
>
> 2. I do agree with Jon’s last email that the lifecycle / orchestration
> portion is the more challenging aspect. It would be nice to address that as
> well so we don’t end up with something like repair where the building
> blocks are there but the hard parts are left to the operator. I do,
> however, see that portion being done in a follow-on CEP to limit the scope
> of CEP-40 and have a higher chance for success by incrementally adding
> these features.
>
> Jordan
>
> On Thu, Apr 11, 2024 at 12:31 Jon Haddad  wrote:
>
> First off, let me apologize for my initial reply, it came off harsher than
> I had intended.
>
> I know I didn't say it initially, but I like the idea of making it easier
> to replace a node.  I think it's probably not obvious to folks that you can
> use rsync (with stunnel, or alternatively rclone), and for a lot of teams
> it's intimidating to do so.  Whether it actually is easy or not to do with
> rsync is irrelevant.  Having tooling that does it right is better than duct
> taping things together.
>
> So with that said, if you're looking to get feedback on how to make the
> CEP more generally useful, I have a couple thoughts.
>
> > Managing the Cassandra processes like bringing them up or down while
> migrating the instances.
>
> Maybe I missed this, but I thought we already had support for managing the
> C* lifecycle with the sidecar?  Maybe I'm misremembering.  It seems to me
> that adding the ability to make this entire workflow self managed would be
> the biggest win, because having a live migrate *feature* instead of what's
> essentially a runbook would be far more useful.
>
> > To verify whether the desired file set matches with source, only file
> path and size is considered at the moment. Strict binary level verification
> is deferred for later.
>
> Scott already mentioned this is a problem and I agree, we cannot simply
> rely on file path and size.
>
> TL;DR: I like the intention of the CEP.  I think it would be better if it
> managed the entire lifecycle of the migration, but you might not have an
> appetite to implement all that.
>
> Jon
>
>
> On Thu, Apr 11, 2024 at 10:01 AM Venkata Hari Krishna Nukala <
> n.v.harikrishna.apa...@gmail.com> wrote:
>
> Thanks Jon & Scott for taking time to go through this CEP and providing
> inputs.
>
> I am completely with what Scott had mentioned earlier (I would have added
> more details into the CEP). Adding a few more points to the same.
>
> Having a solution with Sidecar can make the migration easy without
> depending on rsync. At least in the cases I have seen, rsync is not enabled
> by default and most of them want to run OS/images with as minimal
> requirements as possible. Installing rsync requires admin privileges and
> syncing data is a manual operation. If an API is provided with Sidecar,
> then tooling can be built around it reducing the scope for manual errors.
>
> From performance wise, at least in the cases I had seen, the File
> Streaming API in Sidecar performs a lot better. To give an idea on the
> performance, I would like to quote "up to 7 Gbps/instance writes (depending
> on hardware)" from CEP-28 as this CEP proposes to leverage the same.
>
> For:
>
> >When enabled for LCS, single sstable uplevel will mutate only the level
> of an SSTable in its stats metadata component, which wouldn't alter the
> filename and may not alter the length of the stats metadata component. A
> change to the level of an SSTable on the source via single sstable uplevel
> may not be caught by a digest based only on filename and length.
>
> In this case file size may not change, but the timestamp of last modified
> time would change, right? It is addressed in section MIGRATING ONE
> INSTANCE, point 2.b.ii which says "If a file is present at the destination
> but did not 

Re: Default table compression defined in yaml.

[Claude Warren, Jr via dev]

The text I posted above is directly from the yaml.  Is it intended that
"sstable:" is to be first segment of the yaml key for
"default_compaction"?  If so, it won't because column_index_cache starts in
the first column.

I am happy to move the new configuration section, but I don't follow how
this is to work.



On Thu, Mar 21, 2024 at 1:23 PM Jacek Lewandowski <
lewandowski.ja...@gmail.com> wrote:

> Only indented items below "sstable" belong to "sstable". It is commented
> out by default to make it clear that it is not required and the default
> values apply.
>
> There are a number of sstable parameters which are historically spread
> across the yaml with no structure. The point is that we should not add to
> that mess and try to group the new stuff.
>
> "default_compression" under ""sstable" key sounds good to me.
>
> - - -- --- -----  -
> Jacek Lewandowski
>
>
> czw., 21 mar 2024 o 08:32 Claude Warren, Jr via dev <
> dev@cassandra.apache.org> napisał(a):
>
>> Jacek,
>>
>> I am a bit confused here.  I find a key for "sstable" in the yaml but it
>> is commented out by default.  There are a number of options under it that
>> are commented out and then one that is not and then the
>> "default_compaction" section, which I assume is supposed to apply to the
>> "sstable" section.  Are you saying that the "sstable_compression" section
>> that we introduced should be placed as a child to the "sstable" key (and
>> probably renamed to default_compression"?
>>
>> I have included the keys from the trunk yaml below with non-key comments
>> excluded.  The way I read it either the "sstable" key is not required and a
>> user can just uncomment "column_index_size"; or "column_index_cache_size"
>> is not really used because it would be under
>> "sstable/column_index_cache_size" in the Config; or the "sstable:" is only
>> intended to be a visual break / section for the human editor.
>>
>> Can you or someone clarify this form me?
>>
>> #sstable:
>> #  selected_format: big
>> # column_index_size: 4KiB
>> column_index_cache_size: 2KiB
>> # default_compaction:
>> #   class_name: SizeTieredCompactionStrategy
>> #   parameters:
>> # min_threshold: 4
>> # max_threshold: 32
>>
>> On Wed, Mar 20, 2024 at 10:31 PM Jacek Lewandowski <
>> lewandowski.ja...@gmail.com> wrote:
>>
>>> Compression params for sstables should be under the "sstable" key.
>>>
>>>
>>> - - -- --- -  -
>>> Jacek Lewandowski
>>>
>>>
>>> wt., 19 mar 2024 o 13:10 Ekaterina Dimitrova 
>>> napisał(a):
>>>
>>>> Any new settings are expected to be added in the new format
>>>>
>>>> On Tue, 19 Mar 2024 at 5:52, Bowen Song via dev <
>>>> dev@cassandra.apache.org> wrote:
>>>>
>>>>> I believe the `foobar_in_kb: 123` format in the cassandra.yaml file is
>>>>> deprecated, and the new format is `foobar: 123KiB`. Is there a need to
>>>>> introduce new settings entries with the deprecated format only to be
>>>>> removed at a later version?
>>>>>
>>>>>
>>>>> On 18/03/2024 14:39, Claude Warren, Jr via dev wrote:
>>>>>
>>>>> After much work by several people, I have pulled together the changes
>>>>> to define the default compression in the cassandra.yaml file and have
>>>>> created a pull request [1].
>>>>>
>>>>> If you are interested this in topic, please take a look at the changes
>>>>> and give at least a cursory review.
>>>>>
>>>>> [1]  https://github.com/apache/cassandra/pull/3168
>>>>>
>>>>> Thanks,
>>>>> Claude
>>>>>
>>>>>

Re: Default table compression defined in yaml.

[Claude Warren, Jr via dev]

Jacek,

I am a bit confused here.  I find a key for "sstable" in the yaml but it is
commented out by default.  There are a number of options under it that are
commented out and then one that is not and then the "default_compaction"
section, which I assume is supposed to apply to the "sstable" section.  Are
you saying that the "sstable_compression" section that we introduced should
be placed as a child to the "sstable" key (and probably renamed to
default_compression"?

I have included the keys from the trunk yaml below with non-key comments
excluded.  The way I read it either the "sstable" key is not required and a
user can just uncomment "column_index_size"; or "column_index_cache_size"
is not really used because it would be under
"sstable/column_index_cache_size" in the Config; or the "sstable:" is only
intended to be a visual break / section for the human editor.

Can you or someone clarify this form me?

#sstable:
#  selected_format: big
# column_index_size: 4KiB
column_index_cache_size: 2KiB
# default_compaction:
#   class_name: SizeTieredCompactionStrategy
#   parameters:
# min_threshold: 4
# max_threshold: 32

On Wed, Mar 20, 2024 at 10:31 PM Jacek Lewandowski <
lewandowski.ja...@gmail.com> wrote:

> Compression params for sstables should be under the "sstable" key.
>
>
> - - -- --- -  -
> Jacek Lewandowski
>
>
> wt., 19 mar 2024 o 13:10 Ekaterina Dimitrova 
> napisał(a):
>
>> Any new settings are expected to be added in the new format
>>
>> On Tue, 19 Mar 2024 at 5:52, Bowen Song via dev 
>> wrote:
>>
>>> I believe the `foobar_in_kb: 123` format in the cassandra.yaml file is
>>> deprecated, and the new format is `foobar: 123KiB`. Is there a need to
>>> introduce new settings entries with the deprecated format only to be
>>> removed at a later version?
>>>
>>>
>>> On 18/03/2024 14:39, Claude Warren, Jr via dev wrote:
>>>
>>> After much work by several people, I have pulled together the changes to
>>> define the default compression in the cassandra.yaml file and have created
>>> a pull request [1].
>>>
>>> If you are interested this in topic, please take a look at the changes
>>> and give at least a cursory review.
>>>
>>> [1]  https://github.com/apache/cassandra/pull/3168
>>>
>>> Thanks,
>>> Claude
>>>
>>>

Re: Default table compression defined in yaml.

[Claude Warren, Jr via dev]

We can not support both the "rule" that new settings have the new format,
and the design goal that the CQL statement format be accepted.

We came to a compromise.
We introduced the new chunk_length parameter that requires a DataStorageSpec
We reused the CQL chunk_length_in_kb parameter to accept the CQL format.

I think this is a reasonable compromise.  We could emphasize the
chunk_length parameter in documentation changes and leave the
chunk_length_in_kb parameter to a discussion of converting CQL to YAML
configuration.
We could put in a log message that recommends the correct chunk_length
parameter based on chunk_length_in_kb value.  But I do not see a way to
support both requirements for the new format and the CQL format support.

We can deprecate the chunk_length_in_kb as soon as CQL changes to use a
DataStorageSpec for its parameter, but I do not know of any proposals to
change CQL.


On Tue, Mar 19, 2024 at 1:09 PM Ekaterina Dimitrova 
wrote:

> Any new settings are expected to be added in the new format
>
> On Tue, 19 Mar 2024 at 5:52, Bowen Song via dev 
> wrote:
>
>> I believe the `foobar_in_kb: 123` format in the cassandra.yaml file is
>> deprecated, and the new format is `foobar: 123KiB`. Is there a need to
>> introduce new settings entries with the deprecated format only to be
>> removed at a later version?
>>
>>
>> On 18/03/2024 14:39, Claude Warren, Jr via dev wrote:
>>
>> After much work by several people, I have pulled together the changes to
>> define the default compression in the cassandra.yaml file and have created
>> a pull request [1].
>>
>> If you are interested this in topic, please take a look at the changes
>> and give at least a cursory review.
>>
>> [1]  https://github.com/apache/cassandra/pull/3168
>>
>> Thanks,
>> Claude
>>
>>

Re: Default table compression defined in yaml.

[Claude Warren, Jr via dev]

The earlier request was to be able to take the CQL statement and (with very
little modification) use it in the YAML.  While I agree that introducing
the new setting to remove it later is a bit wonky, it is necessary to
support the current CQL statement.  Unless the CQL statement has changed
already.

On Tue, Mar 19, 2024 at 10:52 AM Bowen Song via dev <
dev@cassandra.apache.org> wrote:

> I believe the `foobar_in_kb: 123` format in the cassandra.yaml file is
> deprecated, and the new format is `foobar: 123KiB`. Is there a need to
> introduce new settings entries with the deprecated format only to be
> removed at a later version?
>
>
> On 18/03/2024 14:39, Claude Warren, Jr via dev wrote:
>
> After much work by several people, I have pulled together the changes to
> define the default compression in the cassandra.yaml file and have created
> a pull request [1].
>
> If you are interested this in topic, please take a look at the changes and
> give at least a cursory review.
>
> [1]  https://github.com/apache/cassandra/pull/3168
>
> Thanks,
> Claude
>
>

Default table compression defined in yaml.

[Claude Warren, Jr via dev]

After much work by several people, I have pulled together the changes to
define the default compression in the cassandra.yaml file and have created
a pull request [1].

If you are interested this in topic, please take a look at the changes and
give at least a cursory review.

[1]  https://github.com/apache/cassandra/pull/3168

Thanks,
Claude

Patently invalid Compression parameters in CompressedSequentialWriterTest

[Claude Warren, Jr via dev]

I have been working at cleaning up the Yaml configuration for default table
compression settings and found that the CompressedSequentialWriterTest uses
some parameters that are outside the acceptable limits (like bufferLength
not a power of 2, or maxCompressedLength > bufferLength).

I can understand that these settings were used to set some extreme
condition, but shouldn't they be within the max/min ranges?

Is it also possible that the tests are not testing what we think they are
because of the parameter issue.

I have left the code path in place that allows creation of invalid
parameters but I am concerned.

Claude

[DISCUSS] Update cassandra-stress to use Apache Commons CLI (CASSANDRA-18661)

[Claude Warren, Jr via dev]

I have been working on CASSANDRA-18661 to see if it is possible to migrate
to the Apache commons-cli as noted in the ticket.  It is possible to do so,
and after several pull requests to commons-cli, I have managed to migrate
the settings of the stress tool.  We will have to wait for commons-cli
1.7.0 to be released before we can merge this change into the trunk.
However, I thought it expedient to discuss a few items before that pull
request is made.


   1. The original options were serializable the apache-cli based ones are
   not.  I think this was to support thrift and is no longer necessary.  I
   would like to remove the Serializable interface from the options.  If this
   is not possible, extra work will need to be done to ensure that
   serialization will work.
   2. There has been some discussion on the ticket about legacy scripts
   using the original options making this a breaking change.  It should be
   possible to deploy the same functional stress test with the two different
   command line interfaces.  This will take some extra work.  The net result
   is that the changes are only in the command line and do not impact the
   operation once the options have been set.

Finally, there were few tests that showed what the various options were
doing.  I have written tests for all the new settings implementations.  I
think that we we keep 2 interfaces it would be possible and advisable to
modify the command line tests to support both CLIs.

If you are interested to see what the new CLI options are, I have pasted
the output of a help command into the jira ticket.

Thank you for your time,
Claude

Re: Call for Presentations closing soon: Community over Code EU 2024

[Claude Warren, Jr via dev]

Additionally, if you have a talk about some underlying technology that
could be applicable across multiple projects submit it or a poster based on
it.  We are looking for good cross-project presentations.

Claude
Chair, Community over Code, EU 2024.

On Mon, Jan 8, 2024 at 8:24 PM Paulo Motta  wrote:

> I wanted to remind that the call for speakers for Community Over Code EU
> 2024 (formerly Apachecon EU) will be closing this Friday 2024/01/12
> 23:59:59 GMT.
>
> If you reside in Europe/EMEA and have an interesting talk proposal about
> using, deploying or modifying Apache Cassandra please see details below to
> submit a proposal to this conference.
>
> -- Forwarded message -
> From: Ryan Skraba 
> Date: Mon, Oct 30, 2023 at 1:07 PM
> Subject: Call for Presentations now open: Community over Code EU 2024
> To:
>
>
> (Note: You are receiving this because you are subscribed to the dev@
> list for one or more projects of the Apache Software Foundation.)
>
> It's back *and* it's new!
>
> We're excited to announce that the first edition of Community over
> Code Europe (formerly known as ApacheCon EU) which will be held at the
> Radisson Blu Carlton Hotel in Bratislava, Slovakia from June 03-05,
> 2024! This eagerly anticipated event will be our first live EU
> conference since 2019.
>
> The Call for Presentations (CFP) for Community Over Code EU 2024 is
> now open at https://eu.communityovercode.org/blog/cfp-open/,
> and will close 2024/01/12 23:59:59 GMT.
>
> We welcome submissions on any topic related to the Apache Software
> Foundation, Apache projects, or the communities around those projects.
> We are specifically looking for presentations in the following
> categories:
>
> * API & Microservices
> * Big Data Compute
> * Big Data Storage
> * Cassandra
> * CloudStack
> * Community
> * Data Engineering
> * Fintech
> * Groovy
> * Incubator
> * IoT
> * Performance Engineering
> * Search
> * Tomcat, Httpd and other servers
>
> Additionally, we are thrilled to introduce a new feature this year: a
> poster session. This addition will provide an excellent platform for
> showcasing high-level projects and incubator initiatives in a visually
> engaging manner. We believe this will foster lively discussions and
> facilitate networking opportunities among participants.
>
> All my best, and thanks so much for your participation,
>
> Ryan Skraba (on behalf of the program committee)
>
> [Countdown]:
> https://www.timeanddate.com/countdown/to?iso=20240112T2359=1440
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
>
>

Re: [DISCUSS] Replace Sigar with OSHI (CASSANDRA-16565)

[Claude Warren, Jr via dev]

The pull request is : https://github.com/apache/cassandra/pull/2842

On Mon, Dec 18, 2023 at 10:26 AM Mick Semb Wever  wrote:

>
>
> Can I get an another review/approval for the pull request?
>> https://github.com/apache/cassandra/pull/2842/files
>>
>
>
> It is not clear on the ticket what is being finally proposed, or what is
> to be reviewed, ref: https://github.com/Claudenw/cassandra/pull/6
> Keeping the ticket up to date makes this easier.
>
> btw, I am in the process of reviewing
> https://github.com/apache/cassandra/compare/trunk...instaclustr:cassandra:CASSANDRA-16565-review
>
>

Re: [DISCUSS] Replace Sigar with OSHI (CASSANDRA-16565)

[Claude Warren, Jr via dev]

Can I get an another review/approval for the pull request?
https://github.com/apache/cassandra/pull/2842/files

On Fri, Dec 15, 2023 at 4:04 AM guo Maxwell  wrote:

> +1 too
>
> Mick Semb Wever  于2023年12月15日周五 10:01写道：
>
>>
>>
>>
>>>
>>> Are there objections to making this switch and adding a new dependency?
>>>
>>> [1] https://github.com/apache/cassandra/pull/2842/files
>>> [2] https://issues.apache.org/jira/browse/CASSANDRA-16565
>>>
>>
>>
>>
>> +1 to removing sigar and to add oshi-core
>>
>>

[DISCUSS] Replace Sigar with OSHI (CASSANDRA-16565)

[Claude Warren, Jr via dev]

Greetings,

I have submitted a pull request[1] that replaces the unsupported Sigar
library with the maintained OSHI library.

OSHI is an MIT licensed library that provides information about the
underlying OS much like Sigar did.

The change adds a dependency on oshi-core at the following coordinates:

com.github.oshi
 oshi-core
 6.4.6

In addition to switching to a supported library, this change will reduce
the size of the package as the native Sigar libraries are removed.

Are there objections to making this switch and adding a new dependency?

[1] https://github.com/apache/cassandra/pull/2842/files
[2] https://issues.apache.org/jira/browse/CASSANDRA-16565

Re: Custom FSError and CommitLog Error Handling

[Claude Warren, Jr via dev]

I can see this as a strong improvement in Cassandra management and support
it.

+1 non binding

On Mon, Dec 11, 2023 at 8:28 PM Raymond Huffman 
wrote:

> Hello All,
>
> On our fork of Cassandra, we've implemented some custom behavior for
> handling CommitLog and SSTable Corruption errors. Specifically, if a node
> detects one of those errors, we want the node to stop itself, and if the
> node is restarted, we want initialization to fail. This is useful in
> Kubernetes when you expect nodes to be restarted frequently and makes our
> corruption remediation workflows less error-prone. I think we could make
> this behavior more pluggable by allowing users to provide custom
> implementations of the FSErrorHandler, and the error handler that's
> currently implemented at
> org.apache.cassandra.db.commitlog.CommitLog#handleCommitError via config in
> the same way one can provide custom Partitioners and
> Authenticators/Authorizers.
>
> Would you take as a contribution one of the following?
> 1. user provided implementations of FSErrorHandler and
> CommitLogErrorHandler, set via config; and/or
> 2. new commit failure and disk failure policies that write a poison pill
> file to disk and fail on startup if that file exists
>
> The poison pill implementation is what we currently use - we call this a
> "Non Transient Error" and we want these states to always require manual
> intervention to resolve, including manual action to clear the error. I'd be
> happy to contribute this if other users would find it beneficial. I had
> initially shared this question in Slack, but I'm now sharing it here for
> broader visibility.
>
> -Raymond Huffman
>

Re: [DISCUSS] CASSANDRA-19104: Standardize tablestats formatting and data units

[Claude Warren, Jr via dev]

Why not change the option so that -H will operate as it does now while -Hn
(where n is a digit) will limit the number of decimal places to n.

On Mon, Dec 4, 2023 at 5:11 PM Brad  wrote:

> Thanks, Jacek.  Using three significant digits for disk space is a good
> suggestion.
>
> On Mon, Dec 4, 2023 at 9:58 AM Jacek Lewandowski <
> lewandowski.ja...@gmail.com> wrote:
>
>> This looks great,
>>
>> I'd consider limiting the number of significant digits to 3 in the human
>> readable format. In the above example it would translate to:
>>
>> Space used (live): 1.46 TiB
>> Space used (total): 1.46 TiB
>>
>> Bytes repaired: 0.00 KiB
>> Bytes unrepaired: 4.31 TiB
>> Bytes pending repair: 0.000 KiB
>>
>> I just think with human readable format we just expect to have a grasp
>> view of the stats and 4th significant digit has very little meaning in that
>> case.
>>
>>
>> thanks,
>> Jacek
>>
>>

Re: Immediately Deprecated Code

[Claude Warren, Jr via dev]

My thought was that we have code that is intended to be used for a specific
time frame.  We should clean up the code base when that code is no longer
used.  But we don't have any good way to track that.  This proposal was an
attempt to provide signposts for removing such code.

On Tue, Oct 31, 2023 at 6:56 PM Mick Semb Wever  wrote:

>
> For online upgrades we support skipping majors so long as the majors are
> adjacent.
> That is, any 4.x.z to any 5.x.z
>  ( Is it recommended that you always first patch upgrade the .z to the
> latest before the major upgrade. )
>
> For offline upgrades, we are aiming to maintain all compatibility.
>
> Take care when removing code, there are various (serdes) classes that look
> like they are for other components but are also used in the storage engine.
>
>
>
> On Tue, 31 Oct 2023 at 18:42, Claude Warren, Jr via dev <
> dev@cassandra.apache.org> wrote:
>
>> In your example 5.1 can read 4.x because 4.0 (?) is the earliest version
>> that 5.x supports.  I don't think you can upgrade directly from 3.x to 5.x
>> without an intermediate stop at some version of 4.x can you?  So when we
>> get to 6.x we won't need the 4 -> 5 conversion code because 6 will only
>> support reading from  5.  If I am incorrect and we expect a version to be
>> able to read 2 major versions back then indeed the deprecated since would
>> be 2 major versions ahead of the version when the code was written.
>>
>> On Tue, Oct 31, 2023 at 2:40 PM Andrew Weaver 
>> wrote:
>>
>>> Skipping versions on upgrade is absolutely something that happens in the
>>> real world.  This is particularly highlighted by the discussion around
>>> 5.0/5.1 that's been happening - 5.0 has been described as a potential
>>> "ghost version" which I completely understand.
>>>
>>> Getting rid of some of the old cruft that seems unnecessary (and
>>> strictly speaking is unnecessary) is not without its downsides.  In this
>>> case, that cruft improves the user experience.
>>>
>>> On Tue, Oct 31, 2023 at 5:56 AM Miklosovic, Stefan via dev <
>>> dev@cassandra.apache.org> wrote:
>>>
>>>> Do I understand it correctly that this is basically the case of
>>>> "deprecated on introduction" as we know that it will not be necessary the
>>>> very next version?
>>>>
>>>> I think that not everybody is upgrading from version to version as they
>>>> appear. If somebody upgrades from 4.0 to 5.1 (which we seem to support) (1)
>>>> and you would have introduced the deprecation in 4.0 with intention to
>>>> remove it in 5.0 and somebody jumps to 5.1 straight away, is not that a
>>>> problem? Because they have not made the move via 5.0 where you upgrade
>>>> logic was triggered.
>>>>
>>>> (1)
>>>> https://github.com/apache/cassandra/blob/trunk/test/distributed/org/apache/cassandra/distributed/upgrade/UpgradeTestBase.java#L97-L108
>>>>
>>>> 
>>>> From: Claude Warren, Jr via dev 
>>>> Sent: Tuesday, October 31, 2023 10:57
>>>> To: dev
>>>> Cc: Claude Warren, Jr
>>>> Subject: Immediately Deprecated Code
>>>>
>>>> NetApp Security WARNING: This is an external email. Do not click links
>>>> or open attachments unless you recognize the sender and know the content is
>>>> safe.
>>>>
>>>>
>>>>
>>>> I was thinking about code that is used to migrate from one version to
>>>> another.  For example the code that rewrote the order of the hash values
>>>> used for Bloom filters.  That code was necessary for the version it was
>>>> coded in.  But the next version does not need that code because the next
>>>> version is not going to read the data from 2 versions prior to itself.  So
>>>> the code could be removed for verson+1.
>>>>
>>>> So, would it have made sense to annotate those methods (and variables)
>>>> as deprecated since the version they were written in so the
>>>> methods/variables can be removed in the next version?
>>>>
>>>> If so, what I propose is that all transitional methods and variable be
>>>> marked as deprecated with the version in which they were introduced.
>>>>
>>>>
>>>
>>> --
>>> Andrew Weaver
>>>
>>

Re: Immediately Deprecated Code

[Claude Warren, Jr via dev]

In your example 5.1 can read 4.x because 4.0 (?) is the earliest version
that 5.x supports.  I don't think you can upgrade directly from 3.x to 5.x
without an intermediate stop at some version of 4.x can you?  So when we
get to 6.x we won't need the 4 -> 5 conversion code because 6 will only
support reading from  5.  If I am incorrect and we expect a version to be
able to read 2 major versions back then indeed the deprecated since would
be 2 major versions ahead of the version when the code was written.

On Tue, Oct 31, 2023 at 2:40 PM Andrew Weaver 
wrote:

> Skipping versions on upgrade is absolutely something that happens in the
> real world.  This is particularly highlighted by the discussion around
> 5.0/5.1 that's been happening - 5.0 has been described as a potential
> "ghost version" which I completely understand.
>
> Getting rid of some of the old cruft that seems unnecessary (and strictly
> speaking is unnecessary) is not without its downsides.  In this case, that
> cruft improves the user experience.
>
> On Tue, Oct 31, 2023 at 5:56 AM Miklosovic, Stefan via dev <
> dev@cassandra.apache.org> wrote:
>
>> Do I understand it correctly that this is basically the case of
>> "deprecated on introduction" as we know that it will not be necessary the
>> very next version?
>>
>> I think that not everybody is upgrading from version to version as they
>> appear. If somebody upgrades from 4.0 to 5.1 (which we seem to support) (1)
>> and you would have introduced the deprecation in 4.0 with intention to
>> remove it in 5.0 and somebody jumps to 5.1 straight away, is not that a
>> problem? Because they have not made the move via 5.0 where you upgrade
>> logic was triggered.
>>
>> (1)
>> https://github.com/apache/cassandra/blob/trunk/test/distributed/org/apache/cassandra/distributed/upgrade/UpgradeTestBase.java#L97-L108
>>
>> 
>> From: Claude Warren, Jr via dev 
>> Sent: Tuesday, October 31, 2023 10:57
>> To: dev
>> Cc: Claude Warren, Jr
>> Subject: Immediately Deprecated Code
>>
>> NetApp Security WARNING: This is an external email. Do not click links or
>> open attachments unless you recognize the sender and know the content is
>> safe.
>>
>>
>>
>> I was thinking about code that is used to migrate from one version to
>> another.  For example the code that rewrote the order of the hash values
>> used for Bloom filters.  That code was necessary for the version it was
>> coded in.  But the next version does not need that code because the next
>> version is not going to read the data from 2 versions prior to itself.  So
>> the code could be removed for verson+1.
>>
>> So, would it have made sense to annotate those methods (and variables) as
>> deprecated since the version they were written in so the methods/variables
>> can be removed in the next version?
>>
>> If so, what I propose is that all transitional methods and variable be
>> marked as deprecated with the version in which they were introduced.
>>
>>
>
> --
> Andrew Weaver
>

Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations

[Claude Warren, Jr via dev]

@henrik,  Have you made any progress on this?  I would like to help drive
it forward but I am waiting to see what your code looks like and figure out
what I need to do.  Any update on timeline would be appreciated.

On Mon, Oct 23, 2023 at 9:07 PM Jon Haddad 
wrote:

> I think this is a great more generally useful than the two scenarios
> you've outlined.  I think it could / should be possible to use an object
> store as the primary storage for sstables and rely on local disk as a cache
> for reads.
>
> I don't know the roadmap for TCM, but imo if it allowed for more stable,
> pre-allocated ranges that compaction will always be aware of (plus a bunch
> of plumbing I'm deliberately avoiding the details on), then you could
> bootstrap a new node by copying s3 directories around rather than streaming
> data between nodes.  That's how we get to 20TB / node, easy scale up /
> down, etc, and always-ZCS for non-object store deployments.
>
> Jon
>
> On 2023/09/25 06:48:06 "Claude Warren, Jr via dev" wrote:
> > I have just filed CEP-36 [1] to allow for keyspace/table storage outside
> of
> > the standard storage space.
> >
> > There are two desires  driving this change:
> >
> >1. The ability to temporarily move some keyspaces/tables to storage
> >outside the normal directory tree to other disk so that compaction can
> >occur in situations where there is not enough disk space for
> compaction and
> >the processing to the moved data can not be suspended.
> >2. The ability to store infrequently used data on slower cheaper
> storage
> >layers.
> >
> > I have a working POC implementation [2] though there are some issues
> still
> > to be solved and much logging to be reduced.
> >
> > I look forward to productive discussions,
> > Claude
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-36%3A+A+Configurable+ChannelProxy+to+alias+external+storage+locations
> > [2] https://github.com/Claudenw/cassandra/tree/channel_proxy_factory
> >
>

Re: Immediately Deprecated Code

[Claude Warren, Jr via dev]

Good point.  When I was thinking about this originally I did realize that
the deprecated tag would need a since = v+1 but I neglected to note that in
my original post.
So in your example the code would be maked as deprecated since v5.0 even
though the code base it is being written in is 4.0.  Thus the code would
not be a candidate for removal until 6.0
I think that this make it easier to remember all those bits that can be
deleted later.

On Tue, Oct 31, 2023 at 11:57 AM Miklosovic, Stefan via dev <
dev@cassandra.apache.org> wrote:

> Do I understand it correctly that this is basically the case of
> "deprecated on introduction" as we know that it will not be necessary the
> very next version?
>
> I think that not everybody is upgrading from version to version as they
> appear. If somebody upgrades from 4.0 to 5.1 (which we seem to support) (1)
> and you would have introduced the deprecation in 4.0 with intention to
> remove it in 5.0 and somebody jumps to 5.1 straight away, is not that a
> problem? Because they have not made the move via 5.0 where you upgrade
> logic was triggered.
>
> (1)
> https://github.com/apache/cassandra/blob/trunk/test/distributed/org/apache/cassandra/distributed/upgrade/UpgradeTestBase.java#L97-L108
>
> ________________
> From: Claude Warren, Jr via dev 
> Sent: Tuesday, October 31, 2023 10:57
> To: dev
> Cc: Claude Warren, Jr
> Subject: Immediately Deprecated Code
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> I was thinking about code that is used to migrate from one version to
> another.  For example the code that rewrote the order of the hash values
> used for Bloom filters.  That code was necessary for the version it was
> coded in.  But the next version does not need that code because the next
> version is not going to read the data from 2 versions prior to itself.  So
> the code could be removed for verson+1.
>
> So, would it have made sense to annotate those methods (and variables) as
> deprecated since the version they were written in so the methods/variables
> can be removed in the next version?
>
> If so, what I propose is that all transitional methods and variable be
> marked as deprecated with the version in which they were introduced.
>
>

Immediately Deprecated Code

[Claude Warren, Jr via dev]

I was thinking about code that is used to migrate from one version to
another.  For example the code that rewrote the order of the hash values
used for Bloom filters.  That code was necessary for the version it was
coded in.  But the next version does not need that code because the next
version is not going to read the data from 2 versions prior to itself.  So
the code could be removed for verson+1.

So, would it have made sense to annotate those methods (and variables) as
deprecated since the version they were written in so the methods/variables
can be removed in the next version?

If so, what I propose is that all transitional methods and variable be
marked as deprecated with the version in which they were introduced.

Re: CASSANDRA-18775 (Cassandra supported OSs)

[Claude Warren, Jr via dev]

I closed 18775 as it did not seem reasonable after discussions here.  I
have been working on 16565 and have a pull request [1] and an experimental
suite to show the differences. [2]

[1]  https://github.com/apache/cassandra/pull/2842
[2]  https://github.com/Aiven-Labs/compare_oshi_sigar

On Wed, Oct 25, 2023 at 2:59 PM Josh McKenzie  wrote:

> +1 to drop if we're not using.
>
> On Fri, Oct 20, 2023, at 6:58 PM, Ekaterina Dimitrova wrote:
>
> +1 on removal the whole lib if we are sure we don’t need it. Nothing
> better than some healthy house cleaning
>
>  -1 on partial removals
>
> On Fri, 20 Oct 2023 at 17:34, David Capwell  wrote:
>
> +1 to drop the whole lib…
>
>
> On Oct 20, 2023, at 7:55 AM, Jeremiah Jordan 
> wrote:
>
> Agreed.  -1 on selectively removing any of the libs.  But +1 for removing
> the whole thing if it is no longer used.
>
> -Jeremiah
>
> On Oct 20, 2023 at 9:28:55 AM, Mick Semb Wever  wrote:
>
> Does anyone see any reason _not_ to do this?
>
>
>
> Thanks for bring this to dev@
>
> I see reason not to do it, folk do submit patches for other archs despite
> us not formally maintaining and testing the code for those archs.  Some
> examples are PPC64 Big Endian (CASSANDRA-7476), s390x (CASSANDRA-17723),
> PPC64 Little Endian (CASSANDRA-7381), sparcv9 (CASSANDRA-6628).  Wrote this
> on the ticket too.
>
> +1 for removing sigar altogether (as Brandon points out).
>
>
>

Development Dependencies documentation.

[Claude Warren, Jr via dev]

I just had to change dependencies in Cassandra for the first  time and I
think the documentation [1] is out of date.

First I think most of the file edits are in the ".build" directory.  Adding
jars to the "lib" directory works until calling "ant realclean", so perhaps
the instructions should include regenerating the "lib" folder after making
the edits.

If I am wrong please let me know, otherwise I will open a ticket and update
the documentation.

[1] https://cassandra.apache.org/_/development/dependencies.html

Re: CASSANDRA-16565

[Claude Warren, Jr via dev]

I ended up posting the code at
https://github.com/Aiven-Labs/compare_oshi_sigar if anyone wants to take a
look and see if they get differing results on various systems.

On Tue, Oct 24, 2023 at 4:59 PM Brandon Williams  wrote:

> On Tue, Oct 24, 2023 at 7:48 AM Claude Warren, Jr via dev
>  wrote:
> >
> > Is there someplace I can stash the tgz that others can download it from?
> The file info is : 6269066 okt 24 12:46 compare_oshi_sigar.tgz
>
> Is posting a github branch not sufficient?
>

CASSANDRA-16565

[Claude Warren, Jr via dev]

I am working on https://issues.apache.org/jira/browse/CASSANDRA-16565 and
have a small testing program that executes the sigar and equivalent OSHI
methods to verify that they are the same.  I would like to have this run on
various platforms.

I have tgz with all the libraries and code as well as a run.sh bash script
to compile and execute it.

Is there someplace I can stash the tgz that others can download it from?
The file info is : 6269066 okt 24 12:46 compare_oshi_sigar.tgz

OSHI does not implement all the methods that Sigar does and there is a
difference in the bitness (32/64 bit) of the results.

*Maximum Virtual Memory*

Sigar dates  from the time of 32 bit OS and so when checking for things
like maximum virtual memory it returns -1 (INFINITE) for any value over
0x7fff.

OSHI on the other hand is 64 bit aware and will return long values for the
maximum virtual memory.  Looking at tools like "ulimit" it converts
anything over 0x7fff to the text "infinite" in the return.

To handle this I set the expected Virtual memory to be 0x7FFFl and
accept any value >= that or  -1 as  the acceptable value.

*Maximum Processes*

This appears to be the value of "ulimit -u" which is not supported in
OSHI.  However, on Linux (and I think Mac OS) we can make a call to the
bash interpreter to return "uname -u".  On other systems I log that we
don't have a way to get this value and return the standard default max
processes of 1024.  This will cause the "warnIfRunningInDegradedMode" to
warn about possible degradation.

Claude

CASSANDRA-18775 (Cassandra supported OSs)

[Claude Warren, Jr via dev]

I am looking at https://issues.apache.org/jira/browse/CASSANDRA-18775 and
want to ensure that I do not remove too many libraries.

I think that preserving any sigar library where the file name contains the
word "linux" or "macosx" should be acceptable.  This will preserve:
libsigar-amd64-linux.so
libsigar-ia64-linux.so
libsigar-ppc-linux.so
libsigar-ppc64-aix-5.so
libsigar-ppc64-linux.so
libsigar-ppc64le-linux.so
libsigar-s390x-linux.so
libsigar-universal-macosx.dylib
libsigar-universal64-macosx.dylib
libsigar-x86-linux.so

and remove:

libsigar-amd64-freebsd-6.so
libsigar-amd64-solaris.so
libsigar-ia64-hpux-11.sl
libsigar-pa-hpux-11.sl
libsigar-ppc-aix-5.so
libsigar-sparc-solaris.so
libsigar-sparc64-solaris.so
libsigar-x86-freebsd-5.so
libsigar-x86-freebsd-6.so
libsigar-x86-solaris.so

resulting in a savings of 3,105,384 bytes out of 6,450,526 from the
/lib/sigar-bin directory, a 48% reduction.

Does anyone see any reason _not_ to do this?

Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations

[Claude Warren, Jr via dev]

t;>>> array?
>>>>
>>>> Should the system set the path to the root of the ColumnFamilyStore in
>>>> the ColumnFamilyStore directories instance?
>>>> Should the Directories.getLocationForDisk() do the proxy to the other
>>>> file system?
>>>>
>>>> Where is the proper location to change from the standard internal
>>>> representation to the remote location?
>>>>
>>>>
>>>> On Fri, Sep 29, 2023 at 8:07 AM Claude Warren, Jr <
>>>> claude.war...@aiven.io> wrote:
>>>>
>>>>> Sorry I was out sick and did not respond yesterday.
>>>>>
>>>>> Henrik,  How does your system work?  What is the design strategy?
>>>>> Also is your code available somewhere?
>>>>>
>>>>> After looking at the code some more I think that the best solution is
>>>>> not a FileChannelProxy but to modify the Cassandra File class to get a
>>>>> FileSystem object for a Factory to build the Path that is used within that
>>>>> object.  I think that this makes if very small change that will pick up
>>>>> 90+% of the cases.  We then just need to find the edge cases.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Sep 29, 2023 at 1:14 AM German Eichberger via dev <
>>>>> dev@cassandra.apache.org> wrote:
>>>>>
>>>>>> Super excited about this as well. Happy to help test with Azure and
>>>>>> any other way needed.
>>>>>>
>>>>>> Thanks,
>>>>>> German
>>>>>> --
>>>>>> *From:* guo Maxwell 
>>>>>> *Sent:* Wednesday, September 27, 2023 7:38 PM
>>>>>> *To:* dev@cassandra.apache.org 
>>>>>> *Subject:* [EXTERNAL] Re: [DISCUSS] CEP-36: A Configurable
>>>>>> ChannelProxy to alias external storage locations
>>>>>>
>>>>>> Thanks , So I think a jira can be created now. And I'd be happy to
>>>>>> provide some help with this as well if needed.
>>>>>>
>>>>>> Henrik Ingo  于2023年9月28日周四 00:21写道：
>>>>>>
>>>>>> It seems I was volunteered to rebase the Astra implementation of this
>>>>>> functionality (FileSystemProvider) onto Cassandra trunk. (And publish it,
>>>>>> of course) I'll try to get going today or tomorrow, so that this
>>>>>> discussion can then benefit from having that code available for 
>>>>>> inspection.
>>>>>> And potentially using it as a soluttion to this use case.
>>>>>>
>>>>>> On Tue, Sep 26, 2023 at 8:04 PM Jake Luciani 
>>>>>> wrote:
>>>>>>
>>>>>> We (DataStax) have a FileSystemProvider for Astra we can provide.
>>>>>> Works with S3/GCS/Azure.
>>>>>>
>>>>>> I'll ask someone on our end to make it accessible.
>>>>>>
>>>>>> This would work by having a bucket prefix per node. But there are lots
>>>>>> of details needed to support things like out of bound compaction
>>>>>> (mentioned in CEP).
>>>>>>
>>>>>> Jake
>>>>>>
>>>>>> On Tue, Sep 26, 2023 at 12:56 PM Benedict 
>>>>>> wrote:
>>>>>> >
>>>>>> > I agree with Ariel, the more suitable insertion point is probably
>>>>>> the JDK level FileSystemProvider and FileSystem abstraction.
>>>>>> >
>>>>>> > It might also be that we can reuse existing work here in some cases?
>>>>>> >
>>>>>> > On 26 Sep 2023, at 17:49, Ariel Weisberg  wrote:
>>>>>> >
>>>>>> > 
>>>>>> > Hi,
>>>>>> >
>>>>>> > Support for multiple storage backends including remote storage
>>>>>> backends is a pretty high value piece of functionality. I am happy to see
>>>>>> there is interest in that.
>>>>>> >
>>>>>> > I think that `ChannelProxyFactory` as an integration point is going
>>>>>> to quickly turn into a dead end as we get into really using multiple
>>>>>> storage backends. We need to be able to list files and really the ful

Re: [DISCUSS] CommitLog default disk access mode

[Claude Warren, Jr via dev]

I think introducing the  feature is a good idea.
I also think that it should _NOT_ be enabled by default for all the reasons
stated above.
Finding a cohort of users who are interested in turning it on would provide
a nice testbed to shake out any issues without affecting everyone.

On Tue, Oct 17, 2023 at 3:58 PM C. Scott Andreas 
wrote:

> Let’s please not change the default at the same time the feature is
> introduced.
>
> Making the capability available will allow users to evaluate and quantify
> the benefit of the work, as well as to call out any unintended
> consequences. As users and the project gain confidence in the results, we
> can evaluate changing the default.
>
> – Scott
>
> On Oct 17, 2023, at 4:25 AM, guo Maxwell  wrote:
>
> 
> -1
>
> I still think we should keep it as it is until the  direct io
> for commitlog (read and write) is ready and relatively stable. And then we
> may change the default value to direct io from mmap in a future version,
> such as 5.2, or 6.0.
>
> Pawar, Amit  于2023年10月17日周二 19:03写道：
>
>> [AMD Official Use Only - General]
>>
>> Thank you all for your input. Received total 6 replies and below is the
>> summary.
>>
>>
>>
>> 1. Mmap   : 2/6
>>
>> 2. Direct-I/O : 4/6
>>
>>
>>
>> Default should be changed to Direct-IO then ? please confirm.
>>
>>
>>
>> Thanks,
>>
>> Amit
>>
>>
>>
>> Strongly agree with this point of view that direct IO  can bring great
>> benefits.
>>
>>
>>
>> I have reviewed part of the code, and my preliminary judgment is that it
>> is not very common and limited in some situations, for example, it  works
>> for commitlog's write path only for this patch.So I suggest that the
>> default value should not be modified until the entire function is
>> comprehensive and stable, and then modified in a future version.
>>
>>
>>
>> Sam  于2023年10月17日周二 05:39写道：
>>
>> *Glad you brought up compaction here - I think there would be a
>> significant benefit to moving compaction to direct i/o.*
>>
>>
>>
>> +1. Would love to see this get traction.
>>
>>
>>
>> On Mon, 16 Oct 2023 at 19:38, Jon Haddad 
>> wrote:
>>
>> Glad you brought up compaction here - I think there would be a
>> significant benefit to moving compaction to direct i/o.
>>
>>
>> On 2023/10/16 16:14:28 Benedict wrote:
>> > I have some plans to (eventually) use the commit log as memtable
>> payload storage (ie memtables would reference the commit log entries
>> directly, storing only indexing info), and to back first level of sstables
>> by reference to commit log entries. This will permit us to deliver not only
>> much bigger memtables (cutting compaction throughput requirements by the
>> ratio of size increase - so pretty dramatically), and faster flushing (so
>> better behaviour ling write bursts), but also a fairly cheap and simple way
>> to support MVCC - which will be helpful for transaction throughput.
>> >
>> > There is also a new commit log (“journal”) coming with Accord, that the
>> rest of C* may or may not transition to.
>> >
>> > I only say this because this makes the utility of direct IO for commit
>> log suspect, as we will be reading from the files as a matter of course
>> should this go ahead; and we may end up relying on a different commit log
>> implementation before long anyway.
>> >
>> > This is obviously a big suggestion and is not guaranteed to transpire,
>> and probably won’t within the next year or so, but it should perhaps form
>> some minimal part of any calculus. If the patch is otherwise simple and
>> beneficial I don’t have anything against it, and the use of direct IO could
>> well be of benefit eg in compaction - and also in future if we manage to
>> bring a page management in process. So laying foundations there could be of
>> benefit, even if the commit log eventually does not use it.
>> >
>> > > On 16 Oct 2023, at 17:00, Jon Haddad 
>> wrote:
>> > >
>> > > I haven't looked at the patch, but at a high level, defaulting to
>> direct I/O for commit logs makes a lot of sense to me.
>> > >
>> > >> On 2023/10/16 06:34:05 "Pawar, Amit" wrote:
>> > >> [Public]
>> > >>
>> > >> Hi,
>> > >>
>> > >> CommitLog uses mmap (memory mapped ) segments by default. Direct-IO
>> feature is proposed through new PR[1] to improve the CommitLog IO speed.
>> Enabling this by default could be useful feature to address IO bottleneck
>> seen during peak load.
>> > >>
>> > >> Need your input regarding changing this default. Please suggest.
>> > >>
>> > >> https://issues.apache.org/jira/browse/CASSANDRA-18464
>> > >>
>> > >> thanks,
>> > >> Amit Pawar
>> > >>
>> > >> [1] - https://github.com/apache/cassandra/pull/2777
>> > >>
>> >
>>
>>
>>
>>
>> --
>>
>> you are the apple of my eye !
>>
>
>
> --
> you are the apple of my eye !
>
>

Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations

[Claude Warren, Jr via dev]

;>>> ChannelProxy to alias external storage locations
>>>>
>>>> Thanks , So I think a jira can be created now. And I'd be happy to
>>>> provide some help with this as well if needed.
>>>>
>>>> Henrik Ingo  于2023年9月28日周四 00:21写道：
>>>>
>>>> It seems I was volunteered to rebase the Astra implementation of this
>>>> functionality (FileSystemProvider) onto Cassandra trunk. (And publish it,
>>>> of course) I'll try to get going today or tomorrow, so that this
>>>> discussion can then benefit from having that code available for inspection.
>>>> And potentially using it as a soluttion to this use case.
>>>>
>>>> On Tue, Sep 26, 2023 at 8:04 PM Jake Luciani  wrote:
>>>>
>>>> We (DataStax) have a FileSystemProvider for Astra we can provide.
>>>> Works with S3/GCS/Azure.
>>>>
>>>> I'll ask someone on our end to make it accessible.
>>>>
>>>> This would work by having a bucket prefix per node. But there are lots
>>>> of details needed to support things like out of bound compaction
>>>> (mentioned in CEP).
>>>>
>>>> Jake
>>>>
>>>> On Tue, Sep 26, 2023 at 12:56 PM Benedict  wrote:
>>>> >
>>>> > I agree with Ariel, the more suitable insertion point is probably the
>>>> JDK level FileSystemProvider and FileSystem abstraction.
>>>> >
>>>> > It might also be that we can reuse existing work here in some cases?
>>>> >
>>>> > On 26 Sep 2023, at 17:49, Ariel Weisberg  wrote:
>>>> >
>>>> > 
>>>> > Hi,
>>>> >
>>>> > Support for multiple storage backends including remote storage
>>>> backends is a pretty high value piece of functionality. I am happy to see
>>>> there is interest in that.
>>>> >
>>>> > I think that `ChannelProxyFactory` as an integration point is going
>>>> to quickly turn into a dead end as we get into really using multiple
>>>> storage backends. We need to be able to list files and really the full
>>>> range of filesystem interactions that Java supports should work with any
>>>> backend to make development, testing, and using existing code
>>>> straightforward.
>>>> >
>>>> > It's a little more work to get C* to creates paths for alternate
>>>> backends where appropriate, but that works is probably necessary even with
>>>> `ChanelProxyFactory` and munging UNIX paths (vs supporting multiple
>>>> Fileystems). There will probably also be backend specific behaviors that
>>>> show up above the `ChannelProxy` layer that will depend on the backend.
>>>> >
>>>> > Ideally there would be some config to specify several backend
>>>> filesystems and their individual configuration that can be used, as well as
>>>> configuration and support for a "backend file router" for file creation
>>>> (and opening) that can be used to route files to the backend most
>>>> appropriate.
>>>> >
>>>> > Regards,
>>>> > Ariel
>>>> >
>>>> > On Mon, Sep 25, 2023, at 2:48 AM, Claude Warren, Jr via dev wrote:
>>>> >
>>>> > I have just filed CEP-36 [1] to allow for keyspace/table storage
>>>> outside of the standard storage space.
>>>> >
>>>> > There are two desires  driving this change:
>>>> >
>>>> > The ability to temporarily move some keyspaces/tables to storage
>>>> outside the normal directory tree to other disk so that compaction can
>>>> occur in situations where there is not enough disk space for compaction and
>>>> the processing to the moved data can not be suspended.
>>>> > The ability to store infrequently used data on slower cheaper storage
>>>> layers.
>>>> >
>>>> > I have a working POC implementation [2] though there are some issues
>>>> still to be solved and much logging to be reduced.
>>>> >
>>>> > I look forward to productive discussions,
>>>> > Claude
>>>> >
>>>> > [1]
>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-36%3A+A+Configurable+ChannelProxy+to+alias+external+storage+locations
>>>> > [2] https://github.com/Claudenw/cassandra/tree/channel_proxy_factory
>>>> >
>>>> >
>>>> >
>>>>
>>>>
>>>> --
>>>> http://twitter.com/tjake
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Henrik Ingo
>>>>
>>>> c. +358 40 569 7354
>>>>
>>>> w. www.datastax.com
>>>>
>>>> <https://www.facebook.com/datastax>  <https://twitter.com/datastax>
>>>> <https://www.linkedin.com/company/datastax/>
>>>> <https://github.com/datastax/>
>>>>
>>>>
>>>>
>>>> --
>>>> you are the apple of my eye !
>>>>
>>>

Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations

[Claude Warren, Jr via dev]

ra we can provide.
>>> Works with S3/GCS/Azure.
>>>
>>> I'll ask someone on our end to make it accessible.
>>>
>>> This would work by having a bucket prefix per node. But there are lots
>>> of details needed to support things like out of bound compaction
>>> (mentioned in CEP).
>>>
>>> Jake
>>>
>>> On Tue, Sep 26, 2023 at 12:56 PM Benedict  wrote:
>>> >
>>> > I agree with Ariel, the more suitable insertion point is probably the
>>> JDK level FileSystemProvider and FileSystem abstraction.
>>> >
>>> > It might also be that we can reuse existing work here in some cases?
>>> >
>>> > On 26 Sep 2023, at 17:49, Ariel Weisberg  wrote:
>>> >
>>> > 
>>> > Hi,
>>> >
>>> > Support for multiple storage backends including remote storage
>>> backends is a pretty high value piece of functionality. I am happy to see
>>> there is interest in that.
>>> >
>>> > I think that `ChannelProxyFactory` as an integration point is going to
>>> quickly turn into a dead end as we get into really using multiple storage
>>> backends. We need to be able to list files and really the full range of
>>> filesystem interactions that Java supports should work with any backend to
>>> make development, testing, and using existing code straightforward.
>>> >
>>> > It's a little more work to get C* to creates paths for alternate
>>> backends where appropriate, but that works is probably necessary even with
>>> `ChanelProxyFactory` and munging UNIX paths (vs supporting multiple
>>> Fileystems). There will probably also be backend specific behaviors that
>>> show up above the `ChannelProxy` layer that will depend on the backend.
>>> >
>>> > Ideally there would be some config to specify several backend
>>> filesystems and their individual configuration that can be used, as well as
>>> configuration and support for a "backend file router" for file creation
>>> (and opening) that can be used to route files to the backend most
>>> appropriate.
>>> >
>>> > Regards,
>>> > Ariel
>>> >
>>> > On Mon, Sep 25, 2023, at 2:48 AM, Claude Warren, Jr via dev wrote:
>>> >
>>> > I have just filed CEP-36 [1] to allow for keyspace/table storage
>>> outside of the standard storage space.
>>> >
>>> > There are two desires  driving this change:
>>> >
>>> > The ability to temporarily move some keyspaces/tables to storage
>>> outside the normal directory tree to other disk so that compaction can
>>> occur in situations where there is not enough disk space for compaction and
>>> the processing to the moved data can not be suspended.
>>> > The ability to store infrequently used data on slower cheaper storage
>>> layers.
>>> >
>>> > I have a working POC implementation [2] though there are some issues
>>> still to be solved and much logging to be reduced.
>>> >
>>> > I look forward to productive discussions,
>>> > Claude
>>> >
>>> > [1]
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-36%3A+A+Configurable+ChannelProxy+to+alias+external+storage+locations
>>> > [2] https://github.com/Claudenw/cassandra/tree/channel_proxy_factory
>>> >
>>> >
>>> >
>>>
>>>
>>> --
>>> http://twitter.com/tjake
>>>
>>>
>>>
>>> --
>>>
>>> Henrik Ingo
>>>
>>> c. +358 40 569 7354
>>>
>>> w. www.datastax.com
>>>
>>> <https://www.facebook.com/datastax>  <https://twitter.com/datastax>
>>> <https://www.linkedin.com/company/datastax/>
>>> <https://github.com/datastax/>
>>>
>>>
>>>
>>> --
>>> you are the apple of my eye !
>>>
>>

Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations

[Claude Warren, Jr via dev]

y necessary even with
>> `ChanelProxyFactory` and munging UNIX paths (vs supporting multiple
>> Fileystems). There will probably also be backend specific behaviors that
>> show up above the `ChannelProxy` layer that will depend on the backend.
>> >
>> > Ideally there would be some config to specify several backend
>> filesystems and their individual configuration that can be used, as well as
>> configuration and support for a "backend file router" for file creation
>> (and opening) that can be used to route files to the backend most
>> appropriate.
>> >
>> > Regards,
>> > Ariel
>> >
>> > On Mon, Sep 25, 2023, at 2:48 AM, Claude Warren, Jr via dev wrote:
>> >
>> > I have just filed CEP-36 [1] to allow for keyspace/table storage
>> outside of the standard storage space.
>> >
>> > There are two desires  driving this change:
>> >
>> > The ability to temporarily move some keyspaces/tables to storage
>> outside the normal directory tree to other disk so that compaction can
>> occur in situations where there is not enough disk space for compaction and
>> the processing to the moved data can not be suspended.
>> > The ability to store infrequently used data on slower cheaper storage
>> layers.
>> >
>> > I have a working POC implementation [2] though there are some issues
>> still to be solved and much logging to be reduced.
>> >
>> > I look forward to productive discussions,
>> > Claude
>> >
>> > [1]
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-36%3A+A+Configurable+ChannelProxy+to+alias+external+storage+locations
>> > [2] https://github.com/Claudenw/cassandra/tree/channel_proxy_factory
>> >
>> >
>> >
>>
>>
>> --
>> http://twitter.com/tjake
>>
>>
>>
>> --
>>
>> Henrik Ingo
>>
>> c. +358 40 569 7354
>>
>> w. www.datastax.com
>>
>> <https://www.facebook.com/datastax>  <https://twitter.com/datastax>
>> <https://www.linkedin.com/company/datastax/>
>> <https://github.com/datastax/>
>>
>>
>>
>> --
>> you are the apple of my eye !
>>
>

multiple ParameterizedClass objects?

[Claude Warren, Jr via dev]

I have a case where I would like to be able to specify a
collection of ParameterizedClass objects in the configuration file.  Is
there a standard way to do this?  If not, does anyone have a suggestion for
a clean way to implement it?

Claude

Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations

[Claude Warren, Jr via dev]

Sorry I was out sick and did not respond yesterday.

Henrik,  How does your system work?  What is the design strategy?  Also is
your code available somewhere?

After looking at the code some more I think that the best solution is not a
FileChannelProxy but to modify the Cassandra File class to get a FileSystem
object for a Factory to build the Path that is used within that object.  I
think that this makes if very small change that will pick up 90+% of the
cases.  We then just need to find the edge cases.





On Fri, Sep 29, 2023 at 1:14 AM German Eichberger via dev <
dev@cassandra.apache.org> wrote:

> Super excited about this as well. Happy to help test with Azure and any
> other way needed.
>
> Thanks,
> German
> --
> *From:* guo Maxwell 
> *Sent:* Wednesday, September 27, 2023 7:38 PM
> *To:* dev@cassandra.apache.org 
> *Subject:* [EXTERNAL] Re: [DISCUSS] CEP-36: A Configurable ChannelProxy
> to alias external storage locations
>
> Thanks , So I think a jira can be created now. And I'd be happy to provide
> some help with this as well if needed.
>
> Henrik Ingo  于2023年9月28日周四 00:21写道：
>
> It seems I was volunteered to rebase the Astra implementation of this
> functionality (FileSystemProvider) onto Cassandra trunk. (And publish it,
> of course) I'll try to get going today or tomorrow, so that this
> discussion can then benefit from having that code available for inspection.
> And potentially using it as a soluttion to this use case.
>
> On Tue, Sep 26, 2023 at 8:04 PM Jake Luciani  wrote:
>
> We (DataStax) have a FileSystemProvider for Astra we can provide.
> Works with S3/GCS/Azure.
>
> I'll ask someone on our end to make it accessible.
>
> This would work by having a bucket prefix per node. But there are lots
> of details needed to support things like out of bound compaction
> (mentioned in CEP).
>
> Jake
>
> On Tue, Sep 26, 2023 at 12:56 PM Benedict  wrote:
> >
> > I agree with Ariel, the more suitable insertion point is probably the
> JDK level FileSystemProvider and FileSystem abstraction.
> >
> > It might also be that we can reuse existing work here in some cases?
> >
> > On 26 Sep 2023, at 17:49, Ariel Weisberg  wrote:
> >
> > 
> > Hi,
> >
> > Support for multiple storage backends including remote storage backends
> is a pretty high value piece of functionality. I am happy to see there is
> interest in that.
> >
> > I think that `ChannelProxyFactory` as an integration point is going to
> quickly turn into a dead end as we get into really using multiple storage
> backends. We need to be able to list files and really the full range of
> filesystem interactions that Java supports should work with any backend to
> make development, testing, and using existing code straightforward.
> >
> > It's a little more work to get C* to creates paths for alternate
> backends where appropriate, but that works is probably necessary even with
> `ChanelProxyFactory` and munging UNIX paths (vs supporting multiple
> Fileystems). There will probably also be backend specific behaviors that
> show up above the `ChannelProxy` layer that will depend on the backend.
> >
> > Ideally there would be some config to specify several backend
> filesystems and their individual configuration that can be used, as well as
> configuration and support for a "backend file router" for file creation
> (and opening) that can be used to route files to the backend most
> appropriate.
> >
> > Regards,
> > Ariel
> >
> > On Mon, Sep 25, 2023, at 2:48 AM, Claude Warren, Jr via dev wrote:
> >
> > I have just filed CEP-36 [1] to allow for keyspace/table storage outside
> of the standard storage space.
> >
> > There are two desires  driving this change:
> >
> > The ability to temporarily move some keyspaces/tables to storage outside
> the normal directory tree to other disk so that compaction can occur in
> situations where there is not enough disk space for compaction and the
> processing to the moved data can not be suspended.
> > The ability to store infrequently used data on slower cheaper storage
> layers.
> >
> > I have a working POC implementation [2] though there are some issues
> still to be solved and much logging to be reduced.
> >
> > I look forward to productive discussions,
> > Claude
> >
> > [1]
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-36%3A+A+Configurable+ChannelProxy+to+alias+external+storage+locations
> > [2] https://github.com/Claudenw/cassandra/tree/channel_proxy_factory
> >
> >
> >
>
>
> --
> http://twitter.com/tjake
>
>
>
> --
>
> Henrik Ingo
>
> c. +358 40 569 7354
>
> w. www.datastax.com
>
> <https://www.facebook.com/datastax>  <https://twitter.com/datastax>
> <https://www.linkedin.com/company/datastax/>
> <https://github.com/datastax/>
>
>
>
> --
> you are the apple of my eye !
>

Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations

[Claude Warren, Jr via dev]

I spent a little (very little) time building an S3 implementation using an
Apache licensed S3 filesystem package.  I have not yet tested it but if
anyone is interested it is at
https://github.com/Aiven-Labs/S3-Cassandra-ChannelProxy

In looking at some of the code I think the Cassandra File class needs to be
modified to ask the ChannelProxy for the default file system for the file
in question.  This should resolve some of the issues my original demo has
with some files being created in the data tree.  It may also handle many of
the cases for offline tools as well.


On Tue, Sep 26, 2023 at 7:33 PM Miklosovic, Stefan <
stefan.mikloso...@netapp.com> wrote:

> Would it be possible to make Jimfs integration production-ready then? I
> see we are using it in the tests already.
>
> It might be one of the reference implementations of this CEP. If there is
> a type of workload / type of nodes with plenty of RAM but no disk, some
> kind of compute nodes, it would just hold it all in memory and we might
> "flush" it to a cloud-based storage if rendered to be not necessary anymore
> (whatever that means).
>
> We could then completely bypass the memtables as fetching data from an
> SSTable from memory would be basically roughly same?
>
> On the other hand, that might be achieved by creating a ramdisk so I am
> not sure what exactly we would gain here. However, if it was eventually
> storing these SSTables in a cloud storage, we might "compact" "TWCS tables"
> automatically after so-and-so period by moving them there.
>
> 
> From: Jake Luciani 
> Sent: Tuesday, September 26, 2023 19:03
> To: dev@cassandra.apache.org
> Subject: Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias
> external storage locations
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
>
> We (DataStax) have a FileSystemProvider for Astra we can provide.
> Works with S3/GCS/Azure.
>
> I'll ask someone on our end to make it accessible.
>
> This would work by having a bucket prefix per node. But there are lots
> of details needed to support things like out of bound compaction
> (mentioned in CEP).
>
> Jake
>
> On Tue, Sep 26, 2023 at 12:56 PM Benedict  wrote:
> >
> > I agree with Ariel, the more suitable insertion point is probably the
> JDK level FileSystemProvider and FileSystem abstraction.
> >
> > It might also be that we can reuse existing work here in some cases?
> >
> > On 26 Sep 2023, at 17:49, Ariel Weisberg  wrote:
> >
> > 
> > Hi,
> >
> > Support for multiple storage backends including remote storage backends
> is a pretty high value piece of functionality. I am happy to see there is
> interest in that.
> >
> > I think that `ChannelProxyFactory` as an integration point is going to
> quickly turn into a dead end as we get into really using multiple storage
> backends. We need to be able to list files and really the full range of
> filesystem interactions that Java supports should work with any backend to
> make development, testing, and using existing code straightforward.
> >
> > It's a little more work to get C* to creates paths for alternate
> backends where appropriate, but that works is probably necessary even with
> `ChanelProxyFactory` and munging UNIX paths (vs supporting multiple
> Fileystems). There will probably also be backend specific behaviors that
> show up above the `ChannelProxy` layer that will depend on the backend.
> >
> > Ideally there would be some config to specify several backend
> filesystems and their individual configuration that can be used, as well as
> configuration and support for a "backend file router" for file creation
> (and opening) that can be used to route files to the backend most
> appropriate.
> >
> > Regards,
> > Ariel
> >
> > On Mon, Sep 25, 2023, at 2:48 AM, Claude Warren, Jr via dev wrote:
> >
> > I have just filed CEP-36 [1] to allow for keyspace/table storage outside
> of the standard storage space.
> >
> > There are two desires  driving this change:
> >
> > The ability to temporarily move some keyspaces/tables to storage outside
> the normal directory tree to other disk so that compaction can occur in
> situations where there is not enough disk space for compaction and the
> processing to the moved data can not be suspended.
> > The ability to store infrequently used data on slower cheaper storage
> layers.
> >
> > I have a working POC implementation [2] though there are some issues
> still to be solved and much logging to be reduced.
> >
> > I look forward to productive discussions,
> > Claude
> >
> > [1]
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-36%3A+A+Configurable+ChannelProxy+to+alias+external+storage+locations
> > [2] https://github.com/Claudenw/cassandra/tree/channel_proxy_factory
> >
> >
> >
>
>
> --
> http://twitter.com/tjake
>

Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations

[Claude Warren, Jr via dev]

The intention of the CEP is to lay the groundwork to allow development of
ChannelProxyFactories that are pluggable in Cassandra.  In this way any
storage system can be a candidate for Cassandra storage provided
FileChannels can be created for the system.

As I stated before I think that there may be a need for a
java.nio.FileSystem implementation for  the proxies but I have not had the
time to dig into it yet.

Claude


On Tue, Sep 26, 2023 at 9:01 AM guo Maxwell  wrote:

> In my mind , it may be better to support most cloud storage : aws,
> azure,gcp,aliyun and so on . We may make it a plugable. But in that way, it
> seems there may need a filesystem interface layer for object storage. And
> should we support ,distributed system like hdfs ,or something else. We
> should first discuss what should be done and what should not be done. It
> simply only supports S3, which feels a bit customized for a certain user
> and is not universal enough.Am I right ?
>
> Claude Warren, Jr  于2023年9月26日周二 14:36写道：
>
>> My intention is to develop an S3 storage system using
>> https://github.com/carlspring/s3fs-nio
>>
>> There are several issues yet to be solved:
>>
>>1. There are some internal calls that create files in the table
>>directory that do not use the channel proxy.  I believe that these are
>>making calls on File objects.  I think those File objects are Cassandra
>>File objects not Java I/O File objects, but am unsure.
>>2. Determine if the carlspring s3fs-nio library will be performant
>>enough to work in the long run.  There may be issues with it:
>>   1. Downloading entire files before using them rather than using
>>   views into larger remotely stored files.
>>   2. Requiring a complete file to upload rather than using the
>>   partial upload capability of the S3 interface.
>>
>>
>>
>> On Tue, Sep 26, 2023 at 4:11 AM guo Maxwell  wrote:
>>
>>> "Rather than building this piece by piece, I think it'd be awesome if
>>> someone drew up an end-to-end plan to implement tiered storage, so we can
>>> make sure we're discussing the whole final state, and not an implementation
>>> detail of one part of the final state?"
>>>
>>> Do agree with jeff for this ~~~ If these feature can be supported in oss
>>> cassandra , I think it will be very popular, whether in  a private
>>> deployment environment or a public cloud service (our experience can prove
>>> it). In addition, it is also a cost-cutting option for users too
>>>
>>> Jeff Jirsa  于2023年9月26日周二 00:11写道：
>>>
>>>>
>>>> - I think this is a great step forward.
>>>> - Being able to move sstables around between tiers of storage is a
>>>> feature Cassandra desperately needs, especially if one of those tiers is
>>>> some sort of object storage
>>>> - This looks like it's a foundational piece that enables that. Perhaps
>>>> by a team that's already implemented this end to end?
>>>> - Rather than building this piece by piece, I think it'd be awesome if
>>>> someone drew up an end-to-end plan to implement tiered storage, so we can
>>>> make sure we're discussing the whole final state, and not an implementation
>>>> detail of one part of the final state?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, Sep 24, 2023 at 11:49 PM Claude Warren, Jr via dev <
>>>> dev@cassandra.apache.org> wrote:
>>>>
>>>>> I have just filed CEP-36 [1] to allow for keyspace/table storage
>>>>> outside of the standard storage space.
>>>>>
>>>>> There are two desires  driving this change:
>>>>>
>>>>>1. The ability to temporarily move some keyspaces/tables to
>>>>>storage outside the normal directory tree to other disk so that 
>>>>> compaction
>>>>>can occur in situations where there is not enough disk space for 
>>>>> compaction
>>>>>and the processing to the moved data can not be suspended.
>>>>>2. The ability to store infrequently used data on slower cheaper
>>>>>storage layers.
>>>>>
>>>>> I have a working POC implementation [2] though there are some issues
>>>>> still to be solved and much logging to be reduced.
>>>>>
>>>>> I look forward to productive discussions,
>>>>> Claude
>>>>>
>>>>> [1]
>>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-36%3A+A+Configurable+ChannelProxy+to+alias+external+storage+locations
>>>>> [2] https://github.com/Claudenw/cassandra/tree/channel_proxy_factory
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>> you are the apple of my eye !
>>>
>>
>
> --
> you are the apple of my eye !
>

Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations

[Claude Warren, Jr via dev]

My intention is to develop an S3 storage system using
https://github.com/carlspring/s3fs-nio

There are several issues yet to be solved:

   1. There are some internal calls that create files in the table
   directory that do not use the channel proxy.  I believe that these are
   making calls on File objects.  I think those File objects are Cassandra
   File objects not Java I/O File objects, but am unsure.
   2. Determine if the carlspring s3fs-nio library will be performant
   enough to work in the long run.  There may be issues with it:
  1. Downloading entire files before using them rather than using views
  into larger remotely stored files.
  2. Requiring a complete file to upload rather than using the partial
  upload capability of the S3 interface.



On Tue, Sep 26, 2023 at 4:11 AM guo Maxwell  wrote:

> "Rather than building this piece by piece, I think it'd be awesome if
> someone drew up an end-to-end plan to implement tiered storage, so we can
> make sure we're discussing the whole final state, and not an implementation
> detail of one part of the final state?"
>
> Do agree with jeff for this ~~~ If these feature can be supported in oss
> cassandra , I think it will be very popular, whether in  a private
> deployment environment or a public cloud service (our experience can prove
> it). In addition, it is also a cost-cutting option for users too
>
> Jeff Jirsa  于2023年9月26日周二 00:11写道：
>
>>
>> - I think this is a great step forward.
>> - Being able to move sstables around between tiers of storage is a
>> feature Cassandra desperately needs, especially if one of those tiers is
>> some sort of object storage
>> - This looks like it's a foundational piece that enables that. Perhaps by
>> a team that's already implemented this end to end?
>> - Rather than building this piece by piece, I think it'd be awesome if
>> someone drew up an end-to-end plan to implement tiered storage, so we can
>> make sure we're discussing the whole final state, and not an implementation
>> detail of one part of the final state?
>>
>>
>>
>>
>>
>>
>> On Sun, Sep 24, 2023 at 11:49 PM Claude Warren, Jr via dev <
>> dev@cassandra.apache.org> wrote:
>>
>>> I have just filed CEP-36 [1] to allow for keyspace/table storage outside
>>> of the standard storage space.
>>>
>>> There are two desires  driving this change:
>>>
>>>1. The ability to temporarily move some keyspaces/tables to storage
>>>outside the normal directory tree to other disk so that compaction can
>>>occur in situations where there is not enough disk space for compaction 
>>> and
>>>the processing to the moved data can not be suspended.
>>>2. The ability to store infrequently used data on slower cheaper
>>>storage layers.
>>>
>>> I have a working POC implementation [2] though there are some issues
>>> still to be solved and much logging to be reduced.
>>>
>>> I look forward to productive discussions,
>>> Claude
>>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-36%3A+A+Configurable+ChannelProxy+to+alias+external+storage+locations
>>> [2] https://github.com/Claudenw/cassandra/tree/channel_proxy_factory
>>>
>>>
>>>
>
> --
> you are the apple of my eye !
>

Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations

[Claude Warren, Jr via dev]

external storage can be any storage that you can produce a FileChannel
for.  There is an S3 library that does this so S3 is a definite
possibility for storage in this solution.  My example code only writes to a
different directory on the same system.  And there are a couple of places
where I did not catch the file creation, those have to be found and
redirected to the proxy location.  I think that it may be necessary to have
a java FileSystem object to make the whole thing work.  The S3 library that
I found also has an S3 FileSystem class.

This solution uses the internal file name for for example an sstable name.
The proxyfactory can examine the entire path and make a determination of
where to read/write the file.  So any determination that can be made based
on the information in the file path can be implemented with this approach.
There is no direct inspection of the data being written to determine
routing.  The only routing data are in the file name.

I ran an inhouse demo where I showed that we could reroute a single table
to a different storage while leaving the rest of the tables in the same
keyspace alone.

In discussing this with a colleague we hit upon the term "tiered nodes".
If you can spread your data across the nodes so that some nodes get the
infrequently used data (cold data) and other nodes receive the frequently
used data (hot data) then the cold data nodes can use this process to store
the data on S3 or similar systems.

On Mon, Sep 25, 2023 at 10:45 AM guo Maxwell  wrote:

> Great suggestion,  Can external storage only be local storage media? Or
> can it be stored in any storage medium, such as object storage s3 ?
> We have previously implemented a tiered storage capability, that is, there
> are multiple storage media on one node, SSD, HDD, and data placement based
> on requests. After briefly browsing the proposals, it seems that there are
> some differences. Can you help to do some explain ？ Thanks 。
>
>
> Claude Warren, Jr via dev  于2023年9月25日周一
> 14:49写道：
>
>> I have just filed CEP-36 [1] to allow for keyspace/table storage outside
>> of the standard storage space.
>>
>> There are two desires  driving this change:
>>
>>1. The ability to temporarily move some keyspaces/tables to storage
>>outside the normal directory tree to other disk so that compaction can
>>occur in situations where there is not enough disk space for compaction 
>> and
>>the processing to the moved data can not be suspended.
>>2. The ability to store infrequently used data on slower cheaper
>>storage layers.
>>
>> I have a working POC implementation [2] though there are some issues
>> still to be solved and much logging to be reduced.
>>
>> I look forward to productive discussions,
>> Claude
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-36%3A+A+Configurable+ChannelProxy+to+alias+external+storage+locations
>> [2] https://github.com/Claudenw/cassandra/tree/channel_proxy_factory
>>
>>
>>
>
> --
> you are the apple of my eye !
>

[DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations

[Claude Warren, Jr via dev]

I have just filed CEP-36 [1] to allow for keyspace/table storage outside of
the standard storage space.

There are two desires  driving this change:

   1. The ability to temporarily move some keyspaces/tables to storage
   outside the normal directory tree to other disk so that compaction can
   occur in situations where there is not enough disk space for compaction and
   the processing to the moved data can not be suspended.
   2. The ability to store infrequently used data on slower cheaper storage
   layers.

I have a working POC implementation [2] though there are some issues still
to be solved and much logging to be reduced.

I look forward to productive discussions,
Claude

[1]
https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-36%3A+A+Configurable+ChannelProxy+to+alias+external+storage+locations
[2] https://github.com/Claudenw/cassandra/tree/channel_proxy_factory

Fixes for UDF NPE during restart.

[Claude Warren, Jr via dev]

CASSANDRA-18739 describes a reproducible NPE on restart with some UDFs.
The solution outlined in that ticket was not used and a much simpler
solution provided by Stefan Miklosovic was implemented.  There are 2 pull
requests open for Cassandra 4.0 and 4.1 that have the fairly simple fix as
well as a test case to show that there was a problem and that the fix
works.

Can someone take a look at the issue and pull requests.  Stefan is working
on a solution for v5 that we may want to get in ASAP.

v4.0 - https://github.com/apache/cassandra/pull/2584

v4.1 - https://github.com/apache/cassandra/pull/2585

Thank you,
Claude

Re: [DISCUSSION] Shall we remove ant javadoc task?

[Claude Warren, Jr via dev]

I think that we can get more developers interested if there are available
javadocs.  While many of the core classes are not going to be touched by
someone just starting, being able to understand what the external touch
points are and how they interact with other bits of the system can be
invaluable, particularly when you don't have the entire code base in front
of you.

For example, I just wrote a tool that explores the distribution of keys
across multiple sstables, I needed some of the tools classes but not much
more.  Javadocs would have made that easy if I did not have the source code
in front of me.

I am -1 on removing the javadocs.

On Thu, Aug 3, 2023 at 4:35 AM Josh McKenzie  wrote:

> If anything, the codebase could use a little more package/class/method
> markup in some places
>
> I am impressed with how diplomatic and generous you're being here Derek. :D
>
> On Wed, Aug 2, 2023, at 5:46 PM, Miklosovic, Stefan wrote:
>
> That is a good idea. I would like to have Javadocs valid when going
> through them in IDE. To enforce it, we would have to fix it first. If we
> find a way how to validate Javadocs without actually rendering them, that
> would be cool.
>
> There is a lot of legacy and rewriting of some custom-crafted formatting
> of some comments might be quite a tedious task to do if it is required to
> have them valid. I am in general for valid documentation and even enforcing
> it but what to do with what is already there ...
>
> 
> From: Jacek Lewandowski 
> Sent: Wednesday, August 2, 2023 23:38
> To: dev@cassandra.apache.org
> Subject: Re: [DISCUSSION] Shall we remove ant javadoc task?
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> With or without outputting JavaDoc to HTML, there are some errors which we
> should maybe fix. We want to keep the documentation, but there can be
> syntax errors which may prevent IDE generating a proper preview. So, the
> question is - should we validate the JavaDoc comments as a precommit task?
> Can it be done without actually generating HTML output?
>
> Thanks,
> Jacek
>
> śr., 2 sie 2023, 22:24 użytkownik Derek Chen-Becker  > napisał:
> Oh, whoops, I guess I'm the only one that thinks Javadoc is just the tool
> and/or it's output (not the markup itself) :P If anything, the codebase
> could use a little more package/class/method markup in some places, so I'm
> definitely only in favor of getting rid of the ant task. I should amend my
> statement to be "...I suspect most people are not opening their browsers
> and looking at Javadoc..." :)
>
> Cheers,
>
> Derek
>
>
>
> On Wed, Aug 2, 2023, 1:30 PM Josh McKenzie  jmcken...@apache.org>> wrote:
> most people are not looking at Javadoc when working on the codebase.
> I definitely use it extensively inside the IDE. But never as a compiled
> set of external docs.
>
> Which is to say, I'm +1 on removing the target and I'd ask everyone to
> keep javadoccing your classes and methods where things are non-obvious or
> there's a logical coupling with something else in the system. :)
>
> On Wed, Aug 2, 2023, at 2:08 PM, Derek Chen-Becker wrote:
> +1. If a need comes up for Javadoc we can fix it at that point, but I
> suspect most people are not looking at Javadoc when working on the codebase.
>
> Cheers,
>
> Derek
>
> On Wed, Aug 2, 2023 at 11:11 AM Brandon Williams  dri...@gmail.com>> wrote:
> I don't think even if it works anyone is going to use the output, so
> I'm good with removal.
>
> Kind Regards,
> Brandon
>
> On Wed, Aug 2, 2023 at 11:50 AM Ekaterina Dimitrova
> mailto:e.dimitr...@gmail.com>> wrote:
> >
> > Hi everyone,
> > We were looking into a user report around our ant javadoc task recently.
> > That made us realize it is not run in CI; it finishes successfully even
> if there are hundreds of errors, some potentially breaking doc pages.
> >
> > There was a ticket discussion where a few community members mentioned
> that this task was probably unnecessary. Can we remove it, or shall we fix
> it?
> >
> > Best regards,
> > Ekaterina
>
>
> --
> +---+
> | Derek Chen-Becker |
> | GPG Key available at https://keybase.io/dchenbecker<
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkeybase.io%2Fdchenbecker=05%7C01%7CStefan.Miklosovic%40netapp.com%7C7ca04f0f58764996ab1e08db93a0de2a%7C4b0911a0929b4715944bc03745165b3a%7C0%7C0%7C638266091373361824%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=n%2BrDfikzzoQG%2Fg%2BRvNqEEE6vHP8ZmY1skeosesLK9v0%3D=0>
> and   |
> | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org<
> 

[DISCUSS] Tiered Storage

[Claude Warren, Jr via dev]

I have been thinking about tiered storage wherein infrequently used data
can be moved off to slow (cold) storage (like S3).  I think that CEP-17 in
conjunction with CEP-21 provides an opportunity for an interesting approach.

As I understand it CEP-17 clarified the SSTables interface(s) so that
alternative implementations are possible, most notably CEM-25 (trie format
sstables).  CEP-21 provides a mechanism by which specific primary key
blocks can be assigned to specific servers.

It seems to me that we could implement an SSTable format that reads/writes
S3 storage and then use CEP-21 to direct specific keys to servers that
implement that storage.

I use primary key because I don't think we can reasonably partition the
records onto cold storage using any other method.

I think that records on the cold storage may be deleted, and may be updated
but both operations may take significant time and would require compaction
to be run at some point.  I expect that compaction would be very slow.

I am certain there are issues with this approach and am looking for
feedback before progressing an architecture proposal.

Thanks,
Claude

Re: Bloom filter calculation

[Claude Warren, Jr via dev]

I think we are talking past each other here.  What I was missing was the
size of the filter.  I was assuming that the size of the filter was the
number of bits specified in the BloomFilterCalculations (error on my
part),  what I was missing was the multiplication of the number of bits by
the number of keys.   Is there a fixed number of bits (it looks to be
Integer.MAX_VALUE - 20) or is it calculated somewhere?


On Tue, Jul 11, 2023 at 10:11 AM Benedict  wrote:

> I’m not sure I follow your reasoning. The bloom filter table is false
> positive per sstable given the number of bits *per key*. So for 10 keys you
> would have 200 bits, which yields the same false positive rate as 20 bits
> and 1 key.
>
> It does taper slightly at much larger N, but it’s pretty nominal for
> practical purposes.
>
> I don’t understand what you mean by merging multiple filters together. We
> do lookup multiple bloom filters per query, but only one per sstable, and
> the false positive rate you’re calculating for 10 such lookups would not be
> accurate. This would be 1-(1-0.671)^10 which is still only around a 4%,
> not 100%. You seem to be looking at the false positive rate of a bloom
> filter of 20 bits with 10 entries, which means only 2 bits per entry?
>
> On 11 Jul 2023, at 07:14, Claude Warren, Jr via dev <
> dev@cassandra.apache.org> wrote:
>
> 
> Can someone explain to me how the Bloom filter table in
> BloomFilterCalculations was derived and how it is supposed to work?  As I
> read the table it seems to indicate that with 14 hashes and 20 bits you get
> a fp of 6.71e-05.  But if you plug those numbers into the Bloom filter
> calculator [1],  that is calculated only for 1 item being in the filter.
> If you merge multiple filters together the false positive rate goes up.
> And as [1] shows by 5 merges you are over 50% fp rate and by 10 you are at
> close to 100% fp.  So I have to assume this analysis is wrong.  Can someone
> point me to the correct calculations?
>
> Claude
>
> [1] https://hur.st/bloomfilter/?n==6.71e-05=20=14
>
>

Bloom filter calculation

[Claude Warren, Jr via dev]

Can someone explain to me how the Bloom filter table in
BloomFilterCalculations was derived and how it is supposed to work?  As I
read the table it seems to indicate that with 14 hashes and 20 bits you get
a fp of 6.71e-05.  But if you plug those numbers into the Bloom filter
calculator [1],  that is calculated only for 1 item being in the filter.
If you merge multiple filters together the false positive rate goes up.
And as [1] shows by 5 merges you are over 50% fp rate and by 10 you are at
close to 100% fp.  So I have to assume this analysis is wrong.  Can someone
point me to the correct calculations?

Claude

[1] https://hur.st/bloomfilter/?n==6.71e-05=20=14

Re: [DISCUSS] Bring cassandra-harry in tree as a submodule

[Claude Warren, Jr via dev]

Since the talk was not accepted for Cassandra Summit, would it be possible
to record it as a simple youtube video and publish it so that the detailed
information about how to use Harry is not lost?

On Thu, May 25, 2023 at 7:36 AM Alex Petrov  wrote:

> While we are at it, we may also want to pull the in-jvm dtest API as a
> submodule, and actually move some tests that are common between the
> branches there.
>
> On Thu, May 25, 2023, at 6:03 AM, Caleb Rackliffe wrote:
>
> Isn’t the other reason Accord works well as a submodule that it has no
> dependencies on C* proper? Harry does at the moment, right? (Not that we
> couldn’t address that…just trying to think this through…)
>
> On May 24, 2023, at 6:54 PM, Benedict  wrote:
>
> 
>
> In this case Harry is a testing module - it’s not something we will
> develop in tandem with C* releases, and we will want improvements to be
> applied across all branches.
>
> So it seems a natural fit for submodules to me.
>
>
> On 24 May 2023, at 21:09, Caleb Rackliffe 
> wrote:
>
> 
> > Submodules do have their own overhead and edge cases, so I am mostly in
> favor of using for cases where the code must live outside of tree (such as
> jvm-dtest that lives out of tree as all branches need the same interfaces)
>
> Agreed. Basically where I've ended up on this topic.
>
> > We could go over some interesting examples such as testing 2i (SAI)
>
> +100
>
>
> On Wed, May 24, 2023 at 1:40 PM Alex Petrov  wrote:
>
>
> > I'm about to need to harry test for the paging across tombstone work for
> https://issues.apache.org/jira/browse/CASSANDRA-18424 (that's where my
> own overlapping fuzzing came in). In the process, I'll see if I can't
> distill something really simple along the lines of how React approaches it (
> https://react.dev/learn).
>
> We can pick that up as an example, sure.
>
> On Wed, May 24, 2023, at 4:53 PM, Josh McKenzie wrote:
>
> I have submitted a proposal to Cassandra Summit for a 4-hour Harry
> workshop,
>
> I'm about to need to harry test for the paging across tombstone work for
> https://issues.apache.org/jira/browse/CASSANDRA-18424 (that's where my
> own overlapping fuzzing came in). In the process, I'll see if I can't
> distill something really simple along the lines of how React approaches it (
> https://react.dev/learn).
>
> Ideally we'd be able to get something together that's a high level "In the
> next 15 minutes, you will know and understand A-G and have access to N% of
> the power of harry" kind of offer.
>
> Honestly, there's a *lot* in our ecosystem where we could benefit from
> taking a page from their book in terms of onboarding and getting started
> IMO.
>
> On Wed, May 24, 2023, at 10:31 AM, Alex Petrov wrote:
>
> > I wonder if a mini-onboarding session would be good as a community
> session - go over Harry, how to run it, how to add a test?  Would that be
> the right venue?  I just would like to see how we can not only plug it in
> to regular CI but get everyone that wants to add a test be able to know how
> to get started with it.
>
> I have submitted a proposal to Cassandra Summit for a 4-hour Harry
> workshop, but unfortunately it got declined. Goes without saying, we can
> still do it online, time and resources permitting. But again, I do not
> think it should be barring us from making Harry a part of the codebase, as
> it already is. In fact, we can be iterating on the development quicker
> having it in-tree.
>
> We could go over some interesting examples such as testing 2i (SAI),
> modelling Group By tests, or testing repair. If there is enough appetite
> and collaboration in the community, I will see if we can pull something
> like that together. Input on _what_ you would like to see / hear / tested
> is also appreciated. Harry was developed out of a strong need for
> large-scale testing, which also has informed many of its APIs, but we can
> make it easier to access for interactive testing / unit tests. We have been
> doing a lot of that with Transactional Metadata, too.
>
> > I'll hold off on this until Alex Petrov chimes in. @Alex -> got any
> thoughts here?
>
> Yes, sorry for not responding on this thread earlier. I can not understate
> how excited I am about this, and how important I think this is. Time
> constraints are somehow hard to overcome, but I hope the results brought by
> TCM will make it all worth it.
>
> On Wed, May 24, 2023, at 4:23 PM, Alex Petrov wrote:
>
> I think pulling Harry into the tree will make adoption easier for the
> folks. I have been a bit swamped with Transactional Metadata work, but I
> wanted to make some of the things we were using for testing TCM available
> outside of TCM branch. This includes a bunch of helper methods to perform
> operations on the clusters, data generation, and more useful stuff. Of
> course, the question always remains about how much time I want to spend
> porting it all to Gossip, but I think we can find a reasonable compromise.
>
> I would not set this improvement as a 

[DISCUSS] Standalone downgrader

[Claude Warren, Jr via dev]

I have an open pull request [1] to merge in a standalone downgrader.

The problem has been that between v3 and v4 there was a breaking change in
the system local table where the columns "broadcast_port", "listen_port",
and "rpc_port" were added.   The code (in the current pull request)
provides functionality to remove those columns when the older table is
written.  The code also allows for other transformations of the columns,
though none are implemented.

In order for the downgrade to work the following steps are taken (not all
are in this code, some are in a script I have for testing the process)


   1. Execute the standalone downgrade on the desired table(s).
   2. Delete the system_schema tables.
   3. Delete the *-Filter.db, *-Index.db, *-TOC.txt, *-Digest.*, and
   *-Summary.db for the modified table(s)
   4. Delete the original files (e.g. nb-*)
   5. Start the earlier version of the software.

I tested the current code by starting 4.1 to create the tables.  Downgraded
all the tables in the database to "ma" version, followed the above steps
and started 3.11   According to the logs 3.11.14 started.

The current pull request code is not as clean as I would like it, but it
does work correctly.

I would like some comments on the general approach for removing columns
where they are filtered out of the row definition during writing.

Your assistance is appreciated,
Claude

[1] https://github.com/apache/cassandra/pull/2045

[COMPRESSION PARAMETERS] Question

[Claude Warren, Jr via dev]

Currently the compression parameters has an option called enable.  When
enable=false all the other options have to be removed.  But it seems to me
that we should support enabled=false without removing all the other
parameters so that users can disable the compression for testing or problem
resolution without losing an of the other parameter settings.  So to be
clear:

The following is valid:
hints_compression:
- class_name: foo
  parameters:
   - chunk_length_in_kb : 16 ;

But this is not:
hints_compression:
- class_name: foo
  parameters:
   - chunk_length_in_kb : 16 ;
  enabled : false ;

Currently when enabled is set false is constructs a default
CompressionParam object with the class name set to null.

Is there a reason to keep this or should we accept the parameters and
construct a CompressionParam with the parameters while continuing to set
the class name to null?

Claude

[DISCUSS] Initial implementation of cassandra-conf with nodetool example

[Claude Warren, Jr via dev]

The pull request [1] is a proposed fix for CASSANDRA-17773.  I am looking
for comments and a decision as to whether to move forward or not with this
change.

The goal is to remove much of the boiler-plate code from scripts without
changing their functionality or arguments and to add the ability to debug
the configuration settings on a running system.

Please take a look and comment,
Claude

[1] https://github.com/apache/cassandra/pull/1950

[DISCUSS] Moving standard boiler plate script blocks.

[Claude Warren, Jr via dev]

I would like to get some more eyes on
https://github.com/apache/cassandra/pull/1950/files wich arises from
CASSANDRA-17773

The basic idea is to:

   - Move the boiler plate script code to a single sourced file.
   - Add code to make debugging scripts easier, this in response to
   CASSANDRA-17773

The pull is a WIP as it shows the new sourced file and how it would be
included in the production scripts.  Two examples are provided.

If it is agreed that this is the approach we would like to take I will
complete the updates of the remaining files.

Your input is appreciated,
Claude

Re: Downgradability

[Claude Warren, Jr via dev]

You also need to remove the system.local.broadcast_port column as that does
not exist in the earlier version and when the earlier version attempts to
read it the code throws an UnknownColumnException.

On Thu, Feb 23, 2023 at 11:27 AM Jacek Lewandowski <
lewandowski.ja...@gmail.com> wrote:

> Running upgrade tests backwards is great idea which does not require extra
> work.
>
> For stats metadata it already supports writing in previous serialization
> version
>
> We need a small fix in compression metadata and that's it.
>
> A flag with the write format version is probably LHF.
>
> Maybe let's try,  we still have time to fix it before 5.0
>
>
> czw., 23 lut 2023, 10:57 użytkownik Benedict 
> napisał:
>
>> Forget downgradeability for a moment: we should not be breaking format
>> compatibility without good reason. Bumping a major version isn’t enough of
>> a reason.
>>
>> Can somebody explain to me why this is being fought tooth and nail, when
>> the work involved is absolutely minimal?
>>
>> Regarding tests: what more do you want, than running our upgrade suite
>> backwards?
>>
>>
>> On 23 Feb 2023, at 09:45, Benjamin Lerer  wrote:
>>
>> 
>>
>>> Can somebody explain to me what is so burdensome, that we seem to be
>>> spending longer debating it than it would take to implement the necessary
>>> changes?
>>
>>
>> I believe that we all agree on the outcome. Everybody wants
>> downgradability. The issue is on the path to get there.
>>
>> As far as I am concerned, I would like to see a proper solution and as
>> Jeff suggested the equivalent of the upgrade tests as gatekeepers. Having
>> everybody trying to enforce it on his own way will only lead to a poor
>> result in my opinion with some addoc code that might not really guarantee
>> real downgradability in the end.
>> We have rushed in the past to get feature outs and pay the price for it.
>> I simply prefer that we take the time to do things right.
>>
>> Thanks to Scott and you, downgradability got a much better visibility so
>> no matter what approach we pick, I am convinced that we will get there.
>>
>> Le jeu. 23 févr. 2023 à 09:49, Claude Warren, Jr via dev <
>> dev@cassandra.apache.org> a écrit :
>>
>>> Broken downgrading can be fixed (I think) by modifying the
>>> SearializationHeader.toHeader() method where it currently throws an
>>> UnknownColumnException.  If we can, instead of throwing the exception,
>>> create a dropped column for the unexpected column then I think the code
>>> will work.
>>>
>>> I realise that to do this in the wild is not possible as it is a
>>> change to released code, but we could handle it going forward.
>>>
>>> On Wed, Feb 22, 2023 at 11:21 PM Henrik Ingo 
>>> wrote:
>>>
>>>> ... ok apparently shift+enter  sends messages now?
>>>>
>>>> I was just saying if at least the file format AND system/tables -
>>>> anything written to disk - can be protected with a switch, then it allows
>>>> for quick downgrade by shutting down the entire cluster and restarting with
>>>> the downgraded binary. It's a start.
>>>>
>>>> To be able to do that live in a distributed system needs to consider
>>>> much more: gossip, streaming, drivers, and ultimately all features, because
>>>> we don't' want an application developer to use a shiny new thing that a)
>>>> may not be available on all nodes, or b) may disappear if the cluster has
>>>> to be downgraded later.
>>>>
>>>> henrik
>>>>
>>>> On Thu, Feb 23, 2023 at 1:14 AM Henrik Ingo 
>>>> wrote:
>>>>
>>>>> Just this once I'm going to be really brief :-)
>>>>>
>>>>> Just wanted to share for reference how Mongodb implemented
>>>>> downgradeability around their 4.4 version:
>>>>> https://www.mongodb.com/docs/manual/release-notes/6.0-downgrade-sharded-cluster/
>>>>>
>>>>> Jeff you're right. Ultimately this is about more than file formats.
>>>>> However, ideally if at least the
>>>>>
>>>>> On Mon, Feb 20, 2023 at 10:02 PM Jeff Jirsa  wrote:
>>>>>
>>>>>> I'm not even convinced even 8110 addresses this - just writing
>>>>>> sstables in old versions won't help if we ever add things like new types 
>>>>>> or
>>>>>> new types of collections without other control abilities. 

[DISCUSS] Single boilerplate script

[Claude Warren, Jr via dev]

Pull request https://github.com/apache/cassandra/pull/1950/files is an
attempt to move the boilerplate coding from the script files into a single
maintainable file.

This change does 4 things:

   1. Moves the standard boiler plate from the standard scripts into a
   single maintainable script to be sourced in the original scripts with
   minimal changes.
   2. Creates a debug logging so that problem determination (as it
   CASSANDRA-17773 )
   is easier.
   3. Has code to preserve environment variables when needed (c.f. nodetool
   script).
   4. Provides a verification method that will verify that all standard
   environment variables are set.

In practice this is a simple 2 line replacement for most boilerplate
blocks.  Examples of the simple case (sstableloader) and a more complex
case (nodetool) are included in the current pull request.

If there is consensus, I will update the other scripts in the bin directory
to utilize the sourced boilerplate and request a full review of the pull
request.

Re: Downgradability

[Claude Warren, Jr via dev]

Broken downgrading can be fixed (I think) by modifying the
SearializationHeader.toHeader() method where it currently throws an
UnknownColumnException.  If we can, instead of throwing the exception,
create a dropped column for the unexpected column then I think the code
will work.

I realise that to do this in the wild is not possible as it is a change to
released code, but we could handle it going forward.

On Wed, Feb 22, 2023 at 11:21 PM Henrik Ingo 
wrote:

> ... ok apparently shift+enter  sends messages now?
>
> I was just saying if at least the file format AND system/tables - anything
> written to disk - can be protected with a switch, then it allows for quick
> downgrade by shutting down the entire cluster and restarting with the
> downgraded binary. It's a start.
>
> To be able to do that live in a distributed system needs to consider much
> more: gossip, streaming, drivers, and ultimately all features, because we
> don't' want an application developer to use a shiny new thing that a) may
> not be available on all nodes, or b) may disappear if the cluster has to be
> downgraded later.
>
> henrik
>
> On Thu, Feb 23, 2023 at 1:14 AM Henrik Ingo 
> wrote:
>
>> Just this once I'm going to be really brief :-)
>>
>> Just wanted to share for reference how Mongodb implemented
>> downgradeability around their 4.4 version:
>> https://www.mongodb.com/docs/manual/release-notes/6.0-downgrade-sharded-cluster/
>>
>> Jeff you're right. Ultimately this is about more than file formats.
>> However, ideally if at least the
>>
>> On Mon, Feb 20, 2023 at 10:02 PM Jeff Jirsa  wrote:
>>
>>> I'm not even convinced even 8110 addresses this - just writing sstables
>>> in old versions won't help if we ever add things like new types or new
>>> types of collections without other control abilities. Claude's other email
>>> in another thread a few hours ago talks about some of these surprises -
>>> "Specifically during the 3.1 -> 4.0 changes a column broadcast_port was
>>> added to system/local.  This means that 3.1 system can not read the table
>>> as it has no definition for it.  I tried marking the column for deletion in
>>> the metadata and in the serialization header.  The later got past the
>>> column not found problem, but I suspect that it just means that data
>>> columns after broadcast_port shifted and so incorrectly read." - this is a
>>> harder problem to solve than just versioning sstables and network
>>> protocols.
>>>
>>> Stepping back a bit, we have downgrade ability listed as a goal, but
>>> it's not (as far as I can tell) universally enforced, nor is it clear at
>>> which point we will be able to concretely say "this release can be
>>> downgraded to X".   Until we actually define and agree that this is a
>>> real goal with a concrete version where downgrade-ability becomes real, it
>>> feels like things are somewhat arbitrarily enforced, which is probably very
>>> frustrating for people trying to commit work/tickets.
>>>
>>> - Jeff
>>>
>>>
>>>
>>> On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi  wrote:
>>>
 I’m a big fan of maintaining backward compatibility. Downgradability
 implies that we could potentially roll back an upgrade at any time. While I
 don’t think we need to retain the ability to downgrade in perpetuity it
 would be a good objective to maintain strict backward compatibility and
 therefore downgradability until a certain point. This would imply
 versioning metadata and extending it in such a way that prior version(s)
 could continue functioning. This can certainly be expensive to implement
 and might bloat on-disk storage. However, we could always offer an option
 for the operator to optimize the on-disk structures for the current version
 then we can rewrite them in the latest version. This optimizes the storage
 and opens up new functionality. This means new features that can work with
 old on-disk structures will be available while others that strictly require
 new versions of the data structures will be unavailable until the operator
 migrates to the new version. This migration IMO should be irreversible.
 Beyond this point the operator will lose the ability to downgrade which is
 ok.

 Dinesh

 On Feb 20, 2023, at 10:40 AM, Jake Luciani  wrote:

 
 There has been progress on

 https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928

 Which is similar to what datastax does for DSE. Would this be an
 acceptable solution?

 Jake

 On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell 
 wrote:

> It seems “An alternative solution is to implement/complete
> CASSANDRA-8110 ”
> can give us more options if it is finished
>
> Branimir Lambov 于2023年2月20日 周一下午11:03写道：
>
>> Hi everyone,
>>
>> There has been a discussion lately about changes to the sstable
>> format in 

Re: Removing columns from sstables

[Claude Warren, Jr via dev]

Close.  It is still in the table so the v3.x code that reads system.local
will detect it and fail on an unknown column as that code appears to be
looking at the actual on-disk format.

It sounds like the short answer is that there is no way to physically
remove the column from the on-disk format once it is added.

On Wed, Feb 22, 2023 at 11:28 AM Erick Ramirez 
wrote:

> When a column is dropped from a table, it is added to the
> system.dropped_columns table so it doesn't get returned in the results. Is
> that what you mean? 
>
>>

Removing columns from sstables

[Claude Warren, Jr via dev]

Greetings,

I have been looking through the code and I can't find any place where
columns are removed from an sstable.   I have found that rows can be
deleted.  Columns can be marked as deleted.  But I have found no place
where the deleted cell is removed from the row.  Is there the concept of
completely removing all traces of the column from the table?

The specific case I am working on is downgrading v4.x system.local table to
v3.1 format.  This involves the removal of the broadcast_port column so
that the hardcoded definition of the v3.1 table can read the sstable from
disk.

Any assistance or pointers would be appreciated,
Claude

Re: Downgradability

[Claude Warren, Jr via dev]

My goal in implementing CASSANDRA-8928

was
to be able to take the current version 4.x and write it as the earliest 3.x
version possible.  The reasoning being that if that was possible then
whatever 3.x version was executed would be able to automatically read the
early 3.x version.  My thought was that each release version would have the
ability to downgrade to the earliest previous version.  In this way if
users need to they could string together a number of downgrader versions to
move from 5.x to 3.x.

My testing has been pretty straightforward, I created 4 docker containers
using the standard published Cassandra docker images for 3.1 and 4.0 with
data mounted on an external drive.  two of the containers (one of each
version) did not automatically start Cassandra.  My process was then:

   1. start and stop Cassandra 4.0 to create the default data files
   2. start the Cassandra 4.0 container that does not automatically run
   Cassandra and execute the new downgrade functionality.
   3. start the Cassandra 3.1 container and dump the logs.  If the system
   started then I knew that I at least had a proof of concept.  So far no-go.



On Tue, Feb 21, 2023 at 8:57 AM Branimir Lambov <
branimir.lam...@datastax.com> wrote:

> It appears to me that the first thing we need to start this feature off is
> a definition of a suite of tests together with a set of rules to keep the
> suite up to date with new features as they are introduced. The moment that
> suite is in place, we can start having some confidence that we can enforce
> downgradability.
>
> Something like this will definitely catch incompatibilities in SSTable
> formats (such as the one in CASSANDRA-17698 that I managed to miss during
> review), but will also be able to identify incompatible system schema
> changes among others, and at the same time rightfully ignore non-breaking
> changes such as modifications to the key cache serialization formats.
>
> Is downgradability in scope for 5.0? It is a feature like any other, and I
> don't see any difficulty adding it (with support for downgrade to 4.x) a
> little later in the 5.x timeline.
>
> Regards,
> Branimir
>
> On Tue, Feb 21, 2023 at 9:40 AM Jacek Lewandowski <
> lewandowski.ja...@gmail.com> wrote:
>
>> I'd like to mention CASSANDRA-17056 (CEP-17) here as it aims to introduce
>> multiple sstable formats support. It allows for providing an implementation
>> of SSTableFormat along with SSTableReader and SSTableWriter. That could be
>> extended easily to support different implementations for certain version
>> ranges, like one impl for ma-nz, other for oa+, etc. without having a
>> confusing implementation with a lot of conditional blocks. Old formats in
>> such case could be maintained separately from the main code and easily
>> switched any time.
>>
>> thanks
>> - - -- --- -  -
>> Jacek Lewandowski
>>
>>
>> wt., 21 lut 2023 o 01:46 Yuki Morishita  napisał(a):
>>
>>> Hi,
>>>
>>> What I wanted to address in my comment in CASSANDRA-8110(
>>> https://issues.apache.org/jira/browse/CASSANDRA-8110?focusedCommentId=17641705=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17641705)
>>> is to focus on better upgrade experience.
>>>
>>> Upgrading the cluster can be painful for some orgs with mission critical
>>> Cassandra cluster, where they cannot tolerate less availability because of
>>> the inability to replace the downed node.
>>> They also need to plan rolling back to the previous state when something
>>> happens along the way.
>>> The change I proposed in CASSANDRA-8110 is to achieve the goal of at
>>> least enabling SSTable streaming during the upgrade by not upgrading the
>>> SSTable version. This can make the cluster to easily rollback to the
>>> previous version.
>>> Downgrading SSTable is not the primary focus (though Cassandra needs to
>>> implement the way to write SSTable in older versions, so it is somewhat
>>> related.)
>>>
>>> I'm preparing the design doc for the change.
>>> Also, if I should create a separate ticket from CASSANDRA-8110 for the
>>> clarity of the goal of the change, please let me know.
>>>
>>>
>>> On Tue, Feb 21, 2023 at 5:31 AM Benedict  wrote:
>>>
 FWIW I think 8110 is the right approach, even if it isn’t a panacea. We
 will have to eventually also tackle system schema changes (probably not
 hard), and may have to think a little carefully about other things, eg with
 TTLs the format change is only the contract about what values can be
 present, so we have to make sure the data validity checks are consistent
 with the format we write. It isn’t as simple as writing an earlier version
 in this case (unless we permit truncating the TTL, perhaps)

 On 20 Feb 2023, at 20:24, Benedict  wrote:


 
 In a self-organising community, things that aren’t self-policed
 naturally end up policed in an adhoc 

downgrade sstables

[Claude Warren, Jr via dev]

I have been working on downgrading sstables for awhile now.  I have the
downgrader mostly working.  The only issue is when downgrading
system tables.

Specifically during the 3.1 -> 4.0 changes a column broadcast_port was
added to system/local.  This means that 3.1 system can not read the table
as it has no definition for it.  I tried marking the column for deletion in
the metadata and in the serialization header.  The later got past the
column not found problem, but I suspect that it just means that data
columns after broadcast_port shifted and so incorrectly read.

I tried to run scrub to see if it would clean up the issues.  No luck.  I
suspect I need to rewrite data columns and remove the broadcast_port
altogether.

Does anyone have a suggestion for how to approach this problem?  Is
rewriting the sstable the only solution?  Can anyone point me to
example code for forcing the rewrite in an offline system?

Thanks,
Claude

Re: Cassandra CI Status 2023-01-07

[Claude Warren, Jr via dev]

New Failures from Build Lead Week 5

*** CASSANDRA-18198 - "AttributeError: module 'py' has no attribute 'io'"
reported in multiple tests
- reported in 4.1, 3.11, and 3.0
- identified as a possible class loader issue associated with
CASSANDRA-18150

*** CASSANDRA-18191 - Native Transport SSL tests failing
- TestNativeTransportSSL.test_connect_to_ssl and
TestNativeTransportSSL.test_connect_to_ssl (novnode)
- TestNativeTransportSSL.test_connect_to_ssl_optional and
TestNativeTransportSSL.test_connect_to_ssl_optional (nvnode)


On Mon, Jan 23, 2023 at 10:10 PM Caleb Rackliffe 
wrote:

> New failures from Build Lead Week 4:
>
> *** CASSANDRA-18188 - Test failure in
> upgrade_tests.cql_tests.cls.test_limit_ranges
> - trunk
> - AttributeError: module 'py' has no attribute 'io'
>
> *** CASSANDRA-18189 - Test failure in
> cqlsh_tests.test_cqlsh_copy.TestCqlshCopy.test_bulk_round_trip_with_timeouts
> - 4.0
> - assert 10 == 94764
> - other failures currently open in this test class, but at least
> superficially, different errors (see CASSANDRA-17322, CASSANDRA-18162)
>
> Timeouts continue to manifest in many places.
>
> On Sun, Jan 15, 2023 at 6:02 AM Mick Semb Wever  wrote:
>
>> *** The Butler (Build Lead)
>>>
>>> The introduction of Butler and the Build Lead was a wonderful
>>> improvement to our CI efforts.  It has brought a lot of hygiene in
>>> listing out flakies as they happened.  Noted that this has in-turn
>>> increased the burden in getting our major releases out, but that's to
>>> be seen as a one-off cost.
>>>
>>
>>
>> New Failures from Build Lead Week 3.
>>
>>
>> *** CASSANDRA-18156
>> – 
>> repair_tests.deprecated_repair_test.TestDeprecatedRepairNotifications.test_deprecated_repair_error_notification
>>  - AssertionError: Node logs don't have an error message for the failed
>> repair
>>  - hard regression
>>  - 3.0, 3.11,
>>
>> *** CASSANDRA-18164 – CASTest Message serializedSize(12) does not match
>> what was written with serialize(out, 12) for verb
>> PAXOS2_COMMIT_AND_PREPARE_RSP
>>  - serializer class org.apache.cassandra.net.Message$Serializer;
>> expected 1077, actual 1079
>>  - 4.1, trunk
>>
>> *** CASSANDRA-18158
>> – 
>> org.apache.cassandra.distributed.upgrade.MixedModeReadTest.mixedModeReadColumnSubsetDigestCheck
>>  - Cannot achieve consistency level ALL
>>  - 3.11, trunk
>>
>> *** CASSANDRA-18159 – repair_tests.repair_test.TestRepair.test_*dc_repair
>>   - AssertionError: null
>> in MemtablePool$SubPool.released(MemtablePool.java:193)
>>  - 3.11, 4.0, 4.1, trunk
>>
>> *** CASSANDRA-18160
>> – 
>> cdc_test.TestCDC.test_insertion_and_commitlog_behavior_after_reaching_cdc_total_space
>>  - Found orphaned index file in after CDC state not in former
>>  - 4.1, trunk
>>
>> *** CASSANDRA-18161 –
>>  
>> org.apache.cassandra.transport.CQLConnectionTest.handleCorruptionOfLargeMessageFrame
>>  - AssertionFailedError in
>> CQLConnectionTest.testFrameCorruption(CQLConnectionTest.java:491)
>>  - 4.0, 4.1, trunk
>>
>> *** CASSANDRA-18162 –
>> cqlsh_tests.test_cqlsh_copy.TestCqlshCopy.test_bulk_round_trip_non_prepared_statements
>> - Inet address 127.0.0.3:7000 is not available: [Errno 98] Address
>> already in use
>> - 3.0, 3.11, 4.0, 4.1, trunk
>>
>> *** CASSANDRA-18163 –
>>  
>> transient_replication_test.TestTransientReplicationRepairLegacyStreaming.test_speculative_write_repair_cycle
>>  - AssertionError Incoming stream entireSSTable
>>  - 4.0, 4.1, trunk
>>
>>
>> While writing these up, some thoughts…
>>  - While Butler reports failures against multiple branches, there's no
>> feedback/sync that the ticket needs its fixVersions updated when failures
>> happen in other branches after the ticket is created.
>>  - In 4.0 onwards, a majority of the failures are timeouts (>900s),
>> reinforcing that the current main problem we are facing in ci-cassandra.a.o
>> is saturation/infra
>>
>>
>>
>>
>>

Re: [DISCUSSION] Framework for Internal Collection Exposure and Monitoring API Alignment

[Claude Warren, Jr via dev]

Actually, Maxim's proposal does not depend on JMX being present or not.
What the proposal does is make it easier to create/sync multiple
presentations of the same internal data:  Virtual Tables, JMX, Metrics,
next year's  greatest data presentation strategy.  Removing JMX from the
mix just reduces the number of implementations, it does not in any way
invalidate or change the proposal.

On Mon, Jan 30, 2023 at 11:27 AM Ekaterina Dimitrova 
wrote:

> +1 on starting that discussion, thank you for volunteering Benjamin!
>
> On Mon, 30 Jan 2023 at 4:59, Benjamin Lerer  wrote:
>
>> It seems to me that the question that we need to answer, before Maxim
>> puts more effort into this work, is: what is the future of JMX?
>> I agree that we have never been clear about that decision up to now. At
>> least now we have a reason to clarify that point.
>>
>> I can start a discussion about that if people agree?
>>
>> Le dim. 29 janv. 2023 à 10:14, Dinesh Joshi  a écrit :
>>
>>> I’m also very interested in this area. I quickly skimmed the proposal
>>> and IIUC it doesn’t call for moving away from JMX. Instead I think it’s
>>> making it easier to expose metrics over various interfaces. Maxim please
>>> correct me if I’m wrong in my understanding.
>>>
>>> I also second Josh’s point on JMX usage. I know it’s disabled in some
>>> deployments but it is not common practice in most places.
>>>
>>> Dinesh
>>>
>>> On Jan 28, 2023, at 4:10 PM, Josh McKenzie  wrote:
>>>
>>> 
>>> First off - thanks so much for putting in this effort Maxim! This is
>>> excellent work.
>>>
>>> Some thoughts on the CEP and responses in thread:
>>>
>>> *Considering that JMX is usually not used and disabled in production
>>> environments for various performance and security reasons, the operator may
>>> not see the same picture from various of Dropwizard's metrics exporters and
>>> integrations as Cassandra's JMX metrics provide [1][2].*
>>>
>>> I don't think this assertion is true. Cassandra is running in a *lot*
>>> of places in the world, and JMX has been in this ecosystem for a long time;
>>> we need data that is basically impossible to get to claim "JMX is usually
>>> not used in C* environments in prod".
>>>
>>> I also wonder about if we should care about JMX?  I know many wish to
>>> migrate (its going to be a very long time) away from JMX, so do we need a
>>> wrapper to make JMX and vtables consistent?
>>>
>>> If we can move away from a bespoke vtable or JMX based implementation
>>> and instead have a templatized solution each of these is generated from,
>>> that to me is the superior option. There's little harm in adding new JMX
>>> endpoints (or hell, other metrics framework integration?) as a byproduct of
>>> adding new vtable exposed metrics; we have the same maintenance obligation
>>> to them as we have to the vtables and if it generates from the same base
>>> data, we shouldn't have any further maintenance burden due to its presence
>>> right?
>>>
>>> we wish to move away from JMX
>>>
>>> I do, and you do, and many people do, but I don't believe *all* people
>>> on the project do. The last time this came up in slack the conclusion was
>>> "Josh should go draft a CEP to chart out a path to moving off JMX while
>>> maintaining backwards-compat w/existing JMX metrics for environments that
>>> are using them" (so I'm excited to see this CEP pop up before I got to it!
>>> ;)). Moving to a system that gives us a 0-cost way to keep JMX and vtable
>>> in sync over time on new metrics seems like a nice compromise for folks
>>> that have built out JMX-based maintenance infra on top of C*. Plus removing
>>> the boilerplate toil on vtables. win-win.
>>>
>>> If we add a column to the end of the JMX row did we just break users?
>>>
>>> I *think* this is arguably true for a vtable / CQL-based solution as
>>> well from the "you don't know how people are using your API" perspective.
>>> Unless we have clear guidelines about discretely selecting the columns you
>>> want from a vtable and trust users to follow them, if people have brittle
>>> greedy parsers pulling in all data from vtables we could very well break
>>> them as well by adding a new column right? Could be wrong here; I haven't
>>> written anything that consumes vtable metric data and maybe the obvious
>>> idiom in the face of that is robust in the presence of column addition.
>>> /shrug
>>>
>>> It's certainly more flexible and simpler to write to w/out detonating
>>> compared to JMX, but it's still an API we'd be revving.
>>>
>>> On Sat, Jan 28, 2023, at 4:24 PM, Ekaterina Dimitrova wrote:
>>>
>>> Overall I have similar thoughts and questions as David.
>>>
>>> I just wanted to add a reminder about this thread from last summer[1].
>>> We already have issues with the alignment of JMX and Settings Virtual
>>> Table. I guess this is how Maxim got inspired to suggest this framework
>>> proposal which I want to thank him for! (I noticed he assigned
>>> CASSANDRA-15254)
>>>
>>> Not to open the Pandora 

Re: [DISCUSSION] Cassandra's code style and source code analysis

[Claude Warren, Jr via dev]

Turn it on at warning (or lower) level now, so people have some idea of the
size of change to their current code.

On Wed, Jan 25, 2023 at 12:05 PM Miklosovic, Stefan <
stefan.mikloso...@netapp.com> wrote:

> Thank you Maxim for doing this.
>
> It is nice to see this effort materialized in a PR.
>
> I would wait until bigger chunks of work are committed to trunk (like
> CEP-15) to not collide too much. I would say we can postpone doing this
> until the actual 5.0 release, last weeks before it so we would not clash
> with any work people would like to include in 5.0. This can go in anytime,
> basically.
>
> Are people on the same page?
>
> Regards
>
> 
> From: Maxim Muzafarov 
> Sent: Monday, January 23, 2023 19:46
> To: dev@cassandra.apache.org
> Subject: Re: [DISCUSSION] Cassandra's code style and source code analysis
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
>
> Hello everyone,
>
> You can find the changes here:
> https://issues.apache.org/jira/browse/CASSANDRA-17925
>
> While preparing the code style configuration for the Eclipse IDE, I
> discovered that there was no easy way to have complex grouping options
> for the set of packages. So we need to add extra blank lines between
> each group of packages so that all the configurations for Eclipse,
> NetBeans, IntelliJ IDEA and checkstyle are aligned. I should have
> checked this earlier for sure, but I only did it for static imports
> and some groups, my bad. The resultant configuration looks like this:
>
> java.*
> [blank line]
> javax.*
> [blank line]
> com.*
> [blank line]
> net.*
> [blank line]
> org.*
> [blank line]
> org.apache.cassandra.*
> [blank line]
> all other imports
> [blank line]
> static all other imports
>
> The pull request is here:
> https://github.com/apache/cassandra/pull/2108
>
> The configuration-related changes are placed in a dedicated commit, so
> it should be easy to make a review:
>
> https://github.com/apache/cassandra/pull/2108/commits/84e292ddc9671a0be76ceb9304b2b9a051c2d52a
>
> 
>
> Another important thing to mention is that the total amount of changes
> for organising imports is really big (more than 2000 files!), so we
> need to decide the right time to merge this PR. Although rebasing or
> merging changes to development branches should become much easier
> ("Accept local" + "Organize imports"), we still need to pay extra
> attention here to minimise the impact on major patches for the next
> release.
>
> On Mon, 16 Jan 2023 at 13:16, Maxim Muzafarov  wrote:
> >
> > Stefan,
> >
> > Thank you for bringing this topic up. I'll prepare the PR shortly with
> > option 4, so everyone can take a look at the amount of changes. This
> > does not force us to go exactly this path, but it may shed light on
> > changes in general.
> >
> > What exactly we're planning to do in the PR:
> >
> > 1. Checkstyle AvoidStarImport rule, so no star imports will be allowed.
> > 2. Checkstyle ImportOrder rule, for controlling the order.
> > 3. The IDE code style configuration for Intellij IDEA, NetBeans, and
> > Eclipse (it doesn't exist for Eclipse yet).
> > 4. The import order according to option 4:
> >
> > ```
> > java.*
> > javax.*
> > [blank line]
> > com.*
> > net.*
> > org.*
> > [blank line]
> > org.apache.cassandra.*
> > [blank line]
> > all other imports
> > [blank line]
> > static all other imports
> > ```
> >
> >
> >
> > On Mon, 16 Jan 2023 at 12:39, Miklosovic, Stefan
> >  wrote:
> > >
> > > Based on the voting we should go with option 4?
> > >
> > > Two weeks passed without anybody joining so I guess folks are all
> happy with that or this just went unnoticed?
> > >
> > > Let's give it time until the end of this week (Friday 12:00 UTC).
> > >
> > > Regards
> > >
> > > 
> > > From: Maxim Muzafarov 
> > > Sent: Tuesday, January 3, 2023 14:31
> > > To: dev@cassandra.apache.org
> > > Subject: Re: [DISCUSSION] Cassandra's code style and source code
> analysis
> > >
> > > NetApp Security WARNING: This is an external email. Do not click links
> or open attachments unless you recognize the sender and know the content is
> safe.
> > >
> > >
> > >
> > >
> > > Folks,
> > >
> > > Let me update the voting status and put together everything we have so
> > > far. We definitely need more votes to have a solid foundation for this
> > > change, so I encourage everyone to consider the options above and
> > > share them in this thread.
> > >
> > >
> > > Total for each applicable option:
> > >
> > > 4-th option -- 4 votes
> > > 3-rd option -- 3 votes
> > > 5-th option -- 1 vote
> > > 1-st option -- 0 votes
> > > 2-nd option -- 0 votes
> > >
> > > On Thu, 22 Dec 2022 at 22:06, Mick Semb Wever  wrote:
> > > >>
> > > >>
> > > >> 3. Total 5 groups, 2968 files to change
> > > >>
> > > >> ```
> > > >> org.apache.cassandra.*
> > > 

Upgrading sstables and default partitioner.

[Claude Warren, Jr via dev]

Greetings,

I am working on porting a fix for table upgrade order into V3.0 and have
come across the following issue:

ERROR 10:23:31 Cannot open
/home/claude/apache/cassandra/build/test/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/me-89-big;
partitioner org.apache.cassandra.dht.ByteOrderedPartitioner does not match
system partitioner org.apache.cassandra.dht.Murmur3Partitioner.  Note that
the default partitioner starting with Cassandra 1.2 is Murmur3Partitioner,
so you will need to edit that to match your old partitioner if upgrading.

Now I know that this can be corrected by setting the default partitioner in
the test code for later versions but in 3.0 we are simply calling the
bin/sstableupgrade script.  This got me wondering.


   1. Should the upgrade fail if the partitioner is different?  I think
   that the partitioner should simply upgrade the format and leave the
   specified partitioner as it is.  If we need to change the partitioner  then
   we need a way to do it with a command line/environment option for the
   sstableupgrade to function.
   2. At what point did the system move from being ByteOrderd to Murmur3?
   Wouldn't the upgradetables script have failed at that point?
   3. #2 leads me to ask, who uses the upgradetables script?  Since later
   Cassandra versions can read earlier versions it must only be used when
   skipping entire versions.

Discussion/solutions for these questions/problems is greatly appreciated,
Claude

Re: [DISCUSS] Clear rules about sstable versioning and downgrade support

[Claude Warren, Jr via dev]

What does this mean for the Trie sstable format?

Would it perhaps make sense to version the sstable upgrader (and future
downgrader) based on the highest version they understand?  for example
sstableupgrader version N will handle the n? versions so it can upgrade
from m? while sstabledowngrader version N can downgrade from m? to
n, where  is the lower limit that downgrader can write.  In
this way users trying to upgrade from out of support major releases can use
multiple upgraders and downgraders to move forward and/or recover from a
failed upgrade.

In this case the sstablesupgrader and sstablesdowngrader should report any
tables that it can not handle and not execute the upgrade/downgrade.


On Fri, Jan 13, 2023 at 1:17 PM Jacek Lewandowski <
lewandowski.ja...@gmail.com> wrote:

> Hi,
>
> I'd like to bring that topic to your attention. I think that we should
> think about allowing users to downgrade under certain conditions. For
> example, always allow for downgrading to any previous minor release.
>
> Clear rules should make users feel safer when upgrading and perhaps
> encourage trying Cassandra at all.
>
> One of the things related to that is sstable format version. It consists
> of major and minor components and is incremented independently from
> Cassandra releases. One rule here is that a Cassandra release producing
> sstables at version XY should be able to read any sstable with version
> (X-1)* and X* (which means that all the minor future versions X. Perhaps we
> could make some commitment to change major sstable format only with new
> major release?
>
> What do you think?
>
> Thanks
> - - -- --- -  -
> Jacek Lewandowski
>

Re: upgrade sstable selection

[Claude Warren, Jr via dev]

Actually since the Directories.SSTableLister stores the Components in a
HashMap indexed by the Descriptor.  Since the upgrade/downgrade code
retrieves the list in hash order there is no guarantee that they will be in
order.  I suspect that this is a bug.

On Tue, Jan 10, 2023 at 12:34 PM Brandon Williams  wrote:

> > I think this means that the Directories.SSTableLister on occasion
> returns files in the incorrect order during a call to
> lister.list().entrySet()
>
> This seems easy enough to verify by looping it and examining the results.
>
> Kind Regards,
> Brandon
>
> On Tue, Jan 10, 2023 at 4:44 AM Claude Warren, Jr via dev
>  wrote:
> >
> > Greetings,
> >
> > I am working on the downgradesstables code and seem to have a problem
> with ordering of the downgrade or perhaps the Directories.SSTableLister
> >
> > I lifted the code from upgradesstables to select the files to
> downgrade.  The only difference in the code that selects the files to
> downgrade is the actual selection of the file.  There is no change to the
> ordering of the files that are evaluated for inclusion.  Yet I think the
> downgrade ordering is incorrect.
> >
> > My process is to start 3.1 version to create the tables and then use the
> 4.0 code base to run the standaloneupgrader and then the
> standalonedowngrader
> >
> > When running the standaloneupgrader on system local I see the following
> > {{noformat}}
> > Found 3 sstables that need upgrading.
> > Upgrading
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-1-big-Data.db')
> > Upgrade of
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-1-big-Data.db')
> complete.
> > Upgrading
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-2-big-Data.db')
> > Upgrade of
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-2-big-Data.db')
> complete.
> > Upgrading
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-3-big-Data.db')
> > Upgrade of
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-3-big-Data.db')
> complete.
> > {{noformat}}
> >
> > when running the standalonedowngrader is see
> > {{noformat}}
> > Found 3 sstables that need downgrading.
> > Downgrading
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-6-big-Data.db')
> > Downgrade of
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-6-big-Data.db')
> complete.
> > Downgrading
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big-Data.db')
> > Downgrade of
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big-Data.db')
> complete.
> > Downgrading
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big-Data.db')
> > Downgrade of
> BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big-Data.db')
> complete.
> > {{noformat}}}
> >
> > Note the order of the generations in the downgrader (I have seen similar
> out of order issues with the upgrader, but infrequently)
> >
> > The difference between the upgrader and downgrader code in the
> questionable section (
> https://github.com/Claudenw/cassandra/blob/CASSANDRA-8928/src/java/org/apache/cassandra/tools/StandaloneDowngrader.java#:~:text=new%20ArrayList%3C%3E()%3B-,//%20Downgrade%20sstables,%7D,-int%20numSSTables%20%3D)
> is on line 101 where the files are selected and put into a list.  I think
> this means that the Directories.SSTableLister on occasion returns files in
> the incorrect order during a call to lister.list().entrySet()
> >
> > I believe that the files are processed in the order specified and that
> the generations get switched around.  This is evidenced by the file size of
> the Data file associated with the generations as it moves through the
> process.  In this case we expect the nb-6 to become ma-7 as per the output
> from the run.  In actuality we want nb-6 to be nb-9.
> >
> > {{noformat}}
> >   ma   nb  ma
> > 1  212 4  2128  212
> > 2   64 5   709   70
> > 3 4876 6 48837 4883
> > {{noformat}}
> >
> > So now my question, has anyone seen the behaviour before?
> >
> > Oh, to make things more interesting I am using Docker images of 3.1 and
> a modified 4.0 that turns off the execution so I can just run the upgrade
> and downgrade.
> >
> > Any help would be appreciated,
> > Claude
>

upgrade sstable selection

[Claude Warren, Jr via dev]

Greetings,

I am working on the downgradesstables code and seem to have a problem with
ordering of the downgrade or perhaps the Directories.SSTableLister

I lifted the code from upgradesstables to select the files to downgrade.
The only difference in the code that selects the files to downgrade is the
actual selection of the file.  There is no change to the ordering of the
files that are evaluated for inclusion.  Yet I think the downgrade ordering
is incorrect.

My process is to start 3.1 version to create the tables and then use the
4.0 code base to run the standaloneupgrader and then the
standalonedowngrader

When running the standaloneupgrader on system local I see the following
{{noformat}}
Found 3 sstables that need upgrading.
Upgrading
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-1-big-Data.db')
Upgrade of
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-1-big-Data.db')
complete.
Upgrading
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-2-big-Data.db')
Upgrade of
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-2-big-Data.db')
complete.
Upgrading
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-3-big-Data.db')
Upgrade of
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/ma-3-big-Data.db')
complete.
{{noformat}}

when running the standalonedowngrader is see
{{noformat}}
Found 3 sstables that need downgrading.
Downgrading
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-6-big-Data.db')
Downgrade of
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-6-big-Data.db')
complete.
Downgrading
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big-Data.db')
Downgrade of
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big-Data.db')
complete.
Downgrading
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big-Data.db')
Downgrade of
BigTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big-Data.db')
complete.
{{noformat}}}

Note the order of the generations in the downgrader (I have seen similar
out of order issues with the upgrader, but infrequently)

The difference between the upgrader and downgrader code in the questionable
section (
https://github.com/Claudenw/cassandra/blob/CASSANDRA-8928/src/java/org/apache/cassandra/tools/StandaloneDowngrader.java#:~:text=new%20ArrayList%3C%3E()%3B-,//%20Downgrade%20sstables,%7D,-int%20numSSTables%20%3D)
is on line 101 where the files are selected and put into a list.  I think
this means that the Directories.SSTableLister on occasion returns files in
the incorrect order during a call to lister.list().entrySet()

I believe that the files are processed in the order specified and that the
generations get switched around.  This is evidenced by the file size of the
Data file associated with the generations as it moves through the process.
In this case we expect the nb-6 to become ma-7 as per the output from the
run.  In actuality we want nb-6 to be nb-9.

{{noformat}}
  ma   nb  ma
1  212 4  2128  212
2   64 5   709   70
3 4876 6 48837 4883
{{noformat}}

So now my question, has anyone seen the behaviour before?

Oh, to make things more interesting I am using Docker images of 3.1 and a
modified 4.0 that turns off the execution so I can just run the upgrade and
downgrade.

Any help would be appreciated,
Claude

Weird results

[Claude Warren, Jr via dev]

I am working on a StandaloneDowngrader.java based on StandaloneUpgrader.java

While working on the tests I had a problem with 2 test (testFlagArgs and
testDefaultCall) that failed with:

ERROR [main] 2022-12-14 10:35:20,051 SSTableReader.java:496 - Cannot open
/home/claude/apache/cassandra/build/test/cassandra/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/nb-41-big;
partitioner org.apache.cassandra.dht.ByteOrderedPartitioner does not match
system partitioner org.apache.cassandra.dht.Murmur3Partitioner.  Note that
the default partitioner starting with Cassandra 1.2 is Murmur3Partitioner,
so you will need to edit that to match your old partitioner if upgrading.

The same tests failed in the StandaloneUpgraderTests on which the
StandaloneDowngraderTests are based.

After chatting with Jake I added code to set the partitioner using
DatabaseDescriptor.setPartitionerUsafe() and a try catch  block to make
sure it got reset in one test.  BOTH tests worked.

I then removed the just added code and both tests continued to work.

I restarted the IDE and both tests continued to work.

So I am not sure how adding and then removing code (including the include
statements) can make the tests work.  But I wanted to post this here so
that if there are other weird cases perhaps we can figure out what is
happening.

Re: Aggregate functions on collections, collection functions and MAXWRITETIME

[Claude Warren, Jr via dev]

at support it. In any
>>> case, the fact that we look for inspiration in other databases to minimize
>>> surprise etc. doesn't mean that we have to do exactly the same. After all,
>>> CQL is not SQL and our collections aren't SQL arrays.
>>>
>>> Note that the collection functions added by CASSANDRA-8877 don't operate
>>> on single values either. That idea was proposed by Yifan on CASSANDRA-18078
>>> and it looked good to Francisco and me. The patch is on CASSANDRA-18085,
>>> already reviewed and blocked waiting on the outcome of this discussion.
>>>
>>> The new collection functions can do the same as the new MAXWRITE
>>> function, but not only for getting max timestamps, but also min timestamps
>>> and min/max ttls. The missing part is that MAXWRITE can accept both
>>> collections and single elements, so callers don't need to know the type of
>>> the column. That's the motivation behind the idea of doing the same with
>>> the collection functions, so they can entirely replace MAXWRITE.
>>>
>>> However I wouldn't be against leaving the collection functions working
>>> only on collections, as originally designed, and as they currently are on
>>> trunk. The question is what we do with MAXWRITETIME. That function is also
>>> only on trunk, and it might be repetitive given the more generic collection
>>> functions. It's also a bit odd that there isn't, for example, a similar
>>> MINTTL function. Maybe we should start a separate discussion thread about
>>> that new function?
>>>
>>>
>>>
>>> On Thu, 8 Dec 2022 at 14:21, Benedict  wrote:
>>>
>>>> 1) Do they offer ARRAY_SUM or ARRAY_AVG?
>>>> 2) Do they define ARRAY_COUNT or ARRAY_LENGTH?
>>>> 3) A map is a collection in C* parlance, but I gather from below you
>>>> expect these methods not to operate on them?
>>>>
>>>> Does ARRAY_MAX operate on single values? If we are to base our
>>>> decisions on norms elsewhere, we should be consistent about it.
>>>>
>>>> It’s worth noting that ARRAY is an ISO SQL concept, as is MULTISET.
>>>> Some databases also have Set or Map types, such as MySQL’s Set and
>>>> Postgres’ hstore. These databases only support ARRAY_ functions, seemingly,
>>>> plus special MULTISET operators defined by the SQL standard where that data
>>>> type is supported.
>>>>
>>>>
>>>>
>>>> On 8 Dec 2022, at 12:11, Andrés de la Peña 
>>>> wrote:
>>>>
>>>> 
>>>> "ARRAY_MAX" and "ARRAY_MIN" functions to get the max/min element in a
>>>> list are not an uncommon practice. You can find them in SparkSQL, Amazon
>>>> Timestream, Teradata, etc. Since we have what we call collections instead
>>>> or arrays, it makes sense to call the analogous functions "COLLECTION_MAX",
>>>> "COLLECTION_MIN", etc.
>>>>
>>>> As for maps, CASSANDRA-8877 also introduced "MAP_KEYS" and "MAP_VALUES"
>>>> functions to get the keys or the values of a map, so one can feed them to
>>>> "MAX", "COLLECTION_MAX", etc. That isn't anything too original either, you
>>>> can find identical functions on SparkSQL for example.
>>>>
>>>> I find simple utility functions easier to use than subqueries. But we
>>>> don't have to chose. We can also have subqueries if someone finds the time
>>>> to work on them.
>>>>
>>>> On Thu, 8 Dec 2022 at 12:04, Claude Warren, Jr via dev <
>>>> dev@cassandra.apache.org> wrote:
>>>>
>>>>>  I think the semantics of the situation is important here.
>>>>>
>>>>>
>>>>> Let’s take MAX as our example aggregate function..
>>>>>
>>>>>
>>>>> We all expect that in a DB context MAX(column) will return the value
>>>>> of the column with the maximum value. That is the expected semantics of 
>>>>> MAX.
>>>>>
>>>>>
>>>>> The question here is that there are data types that are multi-valued
>>>>> and there is a desire to apply MAX to the values within the column. I 
>>>>> would
>>>>> expect that this would return the maximum value of the column every row in
>>>>> the DB.
>>>>>
>>>>>
>>>>> So if there were a keyword that operated like t

Review requested: Add downgradesstables

[Claude Warren, Jr via dev]

https://github.com/apache/cassandra/pull/2045

https://issues.apache.org/jira/browse/CASSANDRA-8928

This is a work in progress and I am looking for some feedback.

This fix appears to work correctly. But I think the placement of the v3
directory is probably not the best and perhaps should be moved under the
db/compaction directory where the Downgrader code is.

Suggestions appreciated.
The changes are:

*added*

   - db/compaction/Downgrader
   - tools/StandAloneDowngrader
   - getNextsstableID to ColumnFamilyStore
   - DOWNGRADE_SSTABLES to OperationType
   - hasMaxCompressedLength to CompressionMetadata constructor and
   associated calls.
   - V3 to SSTableFormat.Type
   - added io/sstable/format/big/v3 directory containing BigFormatV3,
   BigTableReaderV3, BitTableScannerV3, BitTableWriterV3

*modified*

   - CompressionMetadata to skip output of maxCompressedLength if not
   supported

*notes*

   - io/sstable/format/big/v3 classes are the V3 classes modified as
   necessary to run within the V4 environment.

Re: [DISCUSSION] Cassandra's code style and source code analysis

[Claude Warren, Jr via dev]

+1 avoid star rule

On Wed, Dec 7, 2022 at 5:30 PM Maxim Muzafarov  wrote:

> Dear community,
>
>
> I have created the epic with code-style activities to track the progress:
> https://issues.apache.org/jira/browse/CASSANDRA-18090
>
> In my understanding, there is no need to format whole the code base at
> once according to the code style described on the page [1], and the
> best strategy here is to go forward with small evolutionary changes.
> Thus eventually we will come up with a set of rules convenient for all
> members of the community. In my mind, having one commit per an added
> code style rule should be easy to look at for a reviewer, the git
> commits history as well as rebasing/merging other pull requests that
> may be affected by the new rules.
>
>
> I want to raise one more question related to class imports and the
> classses import order for a wider discussion. The import order is well
> described on the code style page [1], but using wildcard imports is
> not mentioned at all. The wildcard imports with their drawbacks has
> has already been raised in the JIRA issue [2] and didn't get enough
> attention.
>
> The checkstyle has the rules we are interested in for import control
> and they must be considered together. We can implement them in a
> single pull request or one by one, or use only the last one:
> - AvoidStarImport
> - CustomImportOrder
>
> But still, I think that wildcard imports have more disadvantages
> (class names conflicts e.g. java.util.*, java.sql.* or a new version
> of a library has name clashes) than advantages and such problems will
> be found in later CI cycles.
> Currently, I've implemented the AvoidStarImport checkstyle rule in a
> dedicated pull request [3][4], so you will be able to see all amount
> of the changes with removing wildcard imports. The changes are made
> for the checkstyle configuration as well as for code style
> configurations for different IDEs we supported.
>
> So, the open questions here are:
>
> - Should the source code obey the AvoidStarImport rule [3]? (I think yes);
> - Should we implement AvoidStarImport and CustomImportOrder in a
> single pull request or do it one by one?
>
>
> Anyway, I will fix the result of the agreement over the
> AvoidStarImport rule on the documentation page [1].
>
>
>
> [1] https://cassandra.apache.org/_/development/code_style.html
> [2] https://issues.apache.org/jira/browse/CASSANDRA-17925
> [3] https://issues.apache.org/jira/browse/CASSANDRA-18089
> [4] https://github.com/apache/cassandra/pull/2041
>
> On Thu, 1 Dec 2022 at 11:55, Claude Warren, Jr via dev
>  wrote:
> >
> > The last time I worked on a project that tried to implement a coding
> style across the project it was "an education".  The short story is that
> trying to "mitigate" the code base, with respect to style, is either a
> massive change or a long slow process.
> >
> > Arguments here have stated that earlier attempts to have the tooling
> reformat the code did not go well.  What we ended up doing was turned on
> the style checker and looked at the number of issues across the project.
> When new code was accepted the number of issues could not rise.  Eventually
> most of the code was clean, with a few well coded legacy bits still not up
> to standard.  We could do something similar here.  Much like code coverage,
> you can't perform a merge unless the number of style errors remains the
> same or decreases.
> >
> > As with all software rules, this is a strong recommendation as I am
> certain that there are edge/corner case exceptions to be found.
> >
> >
> >
> >
> > On Wed, Nov 30, 2022 at 3:30 PM Patrick McFadin 
> wrote:
> >>
> >> Why are we still debating build tooling? I think you’re wrong, but I’ve
> conceded - on the assumption that we can get enough volunteers willing to
> adopt responsibility for the new world order.
> >>
> >> Not debating. I am just throwing in my support since I have been in the
> Camp of Ant.
> >>
> >> On Wed, Nov 30, 2022 at 1:29 AM Benedict  wrote:
> >>>
> >>> Why are we still debating build tooling? I think you’re wrong, but
> I’ve conceded - on the assumption that we can get enough volunteers willing
> to adopt responsibility for the new world order.
> >>>
> >>> I suggest five long term contributors nominate themselves as the build
> file maintainers, and collectively manage a safe and painless migration for
> the rest of us - and agree to maintain and develop the new build file going
> forwards, and support the community as they adopt it.
> >>>
> >>> On the topic of over-exuberant 

Re: Aggregate functions on collections, collection functions and MAXWRITETIME

[Claude Warren, Jr via dev]

 I think the semantics of the situation is important here.


Let’s take MAX as our example aggregate function..


We all expect that in a DB context MAX(column) will return the value of the
column with the maximum value. That is the expected semantics of MAX.


The question here is that there are data types that are multi-valued and
there is a desire to apply MAX to the values within the column. I would
expect that this would return the maximum value of the column every row in
the DB.


So if there were a keyword that operated like the Java BiFunction class
where the Function would apply a second function to the column data. For
purposes of this discussion let’s call this Function APPLY.


So APPLY( MAX, column ) would return the maximum value from the column for
each row in the DB.


MAX(APPLY(MAX,column)) would get the maximum value from the column across
all the rows.


Similarly APPLY could be used with other functions MAX(APPLY(MIN,column))
the largest minimum value from the column across all rows.


These statements make clear semantically what is being asked for.

On Thu, Dec 8, 2022 at 10:57 AM Benedict  wrote:

> I meant unnest, not unwrap.
>
> On 8 Dec 2022, at 10:34, Benedict  wrote:
>
> 
> 
>
> I do not think we should have functions that aggregate across rows and
> functions that operate within a row use the same name.
>
>
> I’m sympathetic to that view for sure. I wouldn’t be too disappointed by
> that outcome, and SQL engines seem to take a similar approach, however they
> mostly rely on sub-queries to get around this problem, and the SQL standard
> introduces UNWRAP for operating on arrays (by translating them into a
> table), permitting subqueries to aggregate them. It seems to me we have
> four options:
>
> 1) introduce functionality similar to UNWRAP and subqueries
> 2) introduce new syntax to permit operating on collections with the same
> functions
> 3) permit the same functions to operate on both, with a precedence order,
> and introduce syntax to permit breaking the precedence order
> 4) introduce new functions
>
> (1) might look like SELECT (SELECT MAX(item) FROM UNWRAP(list)) AS
> max_item FROM table
>
> (2) and (3) might look something like:
>
> SELECT MAX(list AS COLLECTION) or
> SELECT MAX(list AS ROWS)
>
> (4) might look something like we have already, but perhaps with different
> names
>
> The comparator for collections is the lexicographical compare on the
> collection items
>
>
> This is a fair point, I mistakenly thought it sorted first on size. Even
> this definition is a little funkier for Map types, where the values of a
> key may cause something to sort earlier than a map whose next key sorts
> first. There are multiple potential lexicographical sorts for Maps (i.e.,
> by keys first, then values, or by (key, value) pairs), so this is
> particularly poorly defined IMO.
>
> The maximum of a blob type is pretty well defined I think, as are boolean,
> inetaddress etc. However, even for List or Set collections there’s multiple
> reasonable functions one could define for maximum, so it would make more
> sense to me to permit the user to define the comparison as part of the MAX
> function if we are to offer it. However, with the lexicographical
> definition we have I am somewhat less concerned for Set and List. Map seems
> like a real problem though, if we support these operators (which perhaps we
> do not).
>
>
> On 7 Dec 2022, at 12:13, Andrés de la Peña  wrote:
>
> 
> The comparator for collections is the lexicographical compare on the
> collection items. That might nor be the more useful thing but it's not
> impossible to imagine cases where that ordering can be useful. To make a
> random example, you can use a list column to store the name and surnames of
> a person, considering that some persons can have multiple surnames. You can
> then sort rows based on that list column, to get the names in alphabetical
> order, or to get the first or last person according to that order. I'm sure
> we can think on more cases were the lexicographical order of a list can be
> useful, although I agree that's is not the greatest feature ever.
>
> It's worth mentioning that collections are not the only data types where
> the MIN/MAX functions are of dubious utility. For example, blob columns can
> also be used with MIN/MAX. Same as with collections, the min/max blobs are
> selected according to the comparator for the data type. That comparator is
> the lexicographic compare on the unsigned values of the byte contents.
>
> The utility of MIN/MAX on inet and boolean columns isn't very clear
> either, although one can always imagine use cases. Fox example, MAX of a
> boolean column can be used as a logical disjunction.
>
> If we were to special-case MIN/MAX functions to reject collections, we
> should also reject other data types such as, at least, blobs. That would
> require a deprecation plan.
>
> Also, it's not that the comparator used by MIN/MAX is an internal obscure
> thing. The 

Re: [DISCUSSION] Cassandra's code style and source code analysis

[Claude Warren, Jr via dev]

its to switch even if it "just works" now?
>>>
>>> 
>>> From: Benedict 
>>> Sent: Friday, November 25, 2022 11:07
>>> To: dev@cassandra.apache.org
>>> Subject: Re: [DISCUSSION] Cassandra's code style and source code analysis
>>>
>>> NetApp Security WARNING: This is an external email. Do not click links
>>> or open attachments unless you recognize the sender and know the content is
>>> safe.
>>>
>>>
>>>
>>>
>>> There’s always a handful of people asking for it, but notably few if any
>>> of the full time contributors doing the majority of the core development of
>>> Cassandra. It strikes me as something very appealing to others, but less so
>>> to those wanting to get on with development.
>>>
>>> I never really see a good argument articulated for the migration,
>>> besides general hand waving that ant is old, and people like newer build
>>> systems. Ant is working fine, so there isn’t a strong technical reason to
>>> replace it, and there are good organisational reasons not to.
>>>
>>> Why do you consider a migration inevitable?
>>>
>>>
>>>
>>> > On 25 Nov 2022, at 09:58, Miklosovic, Stefan <
>>> stefan.mikloso...@netapp.com> wrote:
>>> >
>>> > Interesting take on Ant / no-Ant, Benedict. I am very curious how
>>> this unfolds. My long-term perception is that changing it to something else
>>> is more or less inevitable but if there is a broader consensus to not do
>>> that  well.
>>> >
>>> > 
>>> > From: Benedict 
>>> > Sent: Friday, November 25, 2022 10:52
>>> > To: dev@cassandra.apache.org
>>> > Subject: Re: [DISCUSSION] Cassandra's code style and source code
>>> analysis
>>> >
>>> > NetApp Security WARNING: This is an external email. Do not click links
>>> or open attachments unless you recognize the sender and know the content is
>>> safe.
>>> >
>>> >
>>> >
>>> >
>>> > I was in a bit of a rush last night. I should say that I’m of course
>>> +1 a general endeavour to clean this up, and to expand our use of linters,
>>> and I appreciate your volunteering to help out in this way Maxim.
>>> >
>>> > However, responding to Stefan, I’m pretty -1 migrating from ant to
>>> another build system without really good reason. Migration has a real cost
>>> to productivity for all existing contributors, and the phantom of
>>> increasing new contributions has never paid off historically. I’m all for
>>> easing people into participation, but not at penalty to the existing
>>> contributor base.
>>> >
>>> > If the only reason is to make it easier to open in a different IDE, we
>>> can perhaps have some basic build files outlining code structure for
>>> importing, that are compatible with our canonical ant build? We could
>>> perhaps even generate them.
>>> >
>>> >
>>> >> On 25 Nov 2022, at 09:35, Miklosovic, Stefan <
>>> stefan.mikloso...@netapp.com> wrote:
>>> >>
>>> >> For the record, I was testing that same combo Claude mentioned and
>>> it did not work out of the box but it is definitely possible to set up
>>> successfully. I do not remember the details.
>>> >>
>>> >> To replay to Maxim, it all seems good to me, roughly, but I humbly
>>> think it all boils down to Maven/Gradle refactoring and on top of that we
>>> can do all else.
>>> >>
>>> >> For example, there is (1) where the solution, besides fixing the
>>> tests, is to introduce an Ant task which would check this on build. That
>>> being said, how is that going to look like when we change Ant for something
>>> else? That stuff suddenly becomes obsolete.
>>> >>
>>> >> This case maybe applies to other problems we want to solve as well. I
>>> do not want to do something tailored for one build system just to rewrite
>>> it all or to spend significant amount of time on that again when we switch
>>> the build system.
>>> >>
>>> >> For that reason I think changing Ant for something else should be top
>>> priority (as I understand that it the hot topic for community for very long
>>> time) and then everything else should

Re: [DISCUSSION] Cassandra's code style and source code analysis

[Claude Warren, Jr via dev]

+1 for the concept as a whole.  I am certain I could find nits to pick if I
looked deeply.

@mck -- I did have a problem with Cassandra + Eclipse + Java11
(Classpath).  I gave up and am spending time trying to learn IntelliJ.  I
also mentioned it in one of the discussion areas.

Claude

On Thu, Nov 24, 2022 at 8:55 PM Mick Semb Wever  wrote:

>  Thank you for a solid write up Maxim. And welcome to Cassandra, it's
> very positive to see you here.
>
> I whole-heartedly agree with nearly everything you write. Some input
> and questions inline.
>
>
> >
> > As you may know, the infrastructure team has disabled public sign-up
> > to ASF JIRA (the GitHub issues are recommended instead).
> >
>
>
> I suspect (based on chatter in general, but not on dev@ AFAIK) is to
> avoid GH issues and stick with jira. The sign-up hurdle we will
> document on the website: CASSANDRA-18064
>
>
>
> > == 1. Make the checkstyle config a single point of truth for the
> > source code style. ==
> >
> > The checkstyle is already used and included in the Cassandra project
> > build lifecycle (ant command line, Jenkins, CircleCI). There is no
> > need to maintain code style configurations for different types of IDEs
> > (e.g. IntelliJ inspections configuration) since the checkstyle.xml
> > file can be directly imported to IDE used by a developer. This is fair
> > for Intellij Idea, NetBeans, and Eclipse.
>
>
> Big +1
>
>
> > So, I propose to focus on the checks themselves and checking pull
> > requests with automation scripts, rather than maintaining these
> > integrations. The benefits here are avoiding all issues with
> > maintaining configurations for different IDEs. Another good advantage
> > of this approach would be the ability to add new checkstyle rules
> > without touching IDE configuration - and such tickets will be LFH and
> > easy to commit.
> >
> > The actions points here are:
> >
> > - create an umbrella JIRA ticket for all checkstyle issues e.g. [8]
> > (or label checkstyle);
> > - add checkstyle to GitHub pull requests using GitHub actions (execute
> > ant command);
>
>
> Instead of custom GHA scripting, please use our existing
> cassandra-artifact.sh (which should already include all such checks).
>
> Something like
> https://github.com/apache/cassandra/compare/cassandra-3.11...thelastpickle:cassandra:mck/github-actions/3.11
>
>
>
> > == 3. Enable pushing backwards build results for both Jenkins and
> > CircleCI to GitHub pull requests. ==
> >
> > The goal here is to have a "green checkbox" for those GitHub pull
> > requests that have corresponding Jenkins/CircleCI runs. According to
> > the following links, it is completely possible to have those.
> >
> > https://github.com/jenkinsci/github-branch-source-plugin
> > https://circleci.com/docs/enable-checks/
> >
> > Moreover, the GitHub Branch Source Plugin is already enabled for the
> > Cassandra project [16]. The same seems should work the same way for
> > CirleCI, but I have faced the infrastructure team comment [17] that
> > describes admin permissions are required for that, so the question is
> > still open here. I could dig a bit more once we've agreed on it.
> >
> > The actions points here are:
> > - enable Jenkins integration for GitHub pull requests;
> > - enable CircleCI integration for GitHub pull requests;
>
>
> Some folk use CircleCI, some use ci-cassandra. The green checkbox idea
> is great, so long as it's optional. We don't want PRs triggering the
> runs either (there are other mechanisms for triggering for now).
>
>
> > The actions points here are:
> >
> > - initiate a wide survey for user and dev lists, to get to know about
> > the usages;
> > - remove those configurations that are not used anymore;
> > - force migration from Ant to Gradle/Maven;
>
>
> Let's leave this out for now. There's too many unknowns here. If
> there's an IDE configuration that's broken, no one has reported it for
> ages, and no one is around to fix it, then I say we should raise the
> discussion to remove it.
>
> The Gradle/Maven migration is a hot one, contribute to that discussion
> but let's not tangle this work up with it, IMHO.
>
> Totally agree that IDE project files should be as light weight as possible.
>
>
> > Summarizing everything proposed above I think it is possible to
> > simplify adding small contributions easier to the codebase as well as
> > save a bunch of committer's time.
> >
> > So,
> > WDYT about the things described above?
> > Should I create a CEP for this?
>
>
> I see no need for a CEP here. An epic and tickets will work.
> Again, thanks for the input Maxim!
>

Re: A proposal for refactoring the CircleCI config

[Claude Warren, Jr via dev]

I have been working on
https://issues.apache.org/jira/projects/CASSANDRA/issues/CASSANDRA-18012
which modifies the generate.sh script for the circleci configurations.
Perhaps all of this should be rolled into one change?

On Fri, Nov 11, 2022 at 3:47 AM Ekaterina Dimitrova 
wrote:

> Hey Derek,
> Thanks for looking into this.
> As we spoke in Slack, probably an easy way to show people how things will
> look like is to have a prototype with some minimal config. Could be even
> not Cassandra one but something that will show how things will look like
> and improve the current model.
> Thanks,
> Ekaterina
>
> On Wed, 2 Nov 2022 at 17:08, David Capwell  wrote:
>
>> Here is the ticket I was talking about
>> https://issues.apache.org/jira/browse/CASSANDRA-17600
>>
>>
>> On Nov 2, 2022, at 1:29 PM, Derek Chen-Becker 
>> wrote:
>>
>> For the parallel param logic, sounds fine to me.  Not sure if this would
>> also work for resource_type, but I still argue that xlarge isn’t needed in
>> 90% of the cases its used… so fixing this may be better than param there….
>> So yes, I would be cool with this change if it basically removes the
>> patching logic… I had another JIRA to have a python script rewrite the
>> YAML, but this method may solve in a cleaner way.
>>
>>
>> Almost any part of a CircleCI definition can be replaced with a
>> parameter, so basically we want config-2_1.yml to be a template, and
>> we plug different values in as desired. Would you mind sending a link
>> to that JIRA so I can understand that use case?
>>
>> About matrix jobs; I don’t know them in circle but have used in other
>> places, this sounds good to me.  I would also enhance to argue that JVM is
>> just 1 config and we sadly have many more:
>>
>> JVM: [8, 11, 17]
>> VNODE: [true, false]
>> CDC: [true, false]
>> COMPRESSION: [true, false]
>> MEMTABLE: [skiplist, shardedskiplist, trie]
>>
>>
>> My understanding is that we could parameterize all of these such that
>> we could use a matrix as long as all combinations are valid. Let me
>> get parameterization of basic configuration reviewed first, and then
>> we can take a look at how to matricize things.
>>
>> Cheers,
>>
>> Derek
>>
>> --
>> +---+
>> | Derek Chen-Becker |
>> | GPG Key available at https://keybase.io/dchenbecker and   |
>> | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
>> | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
>> +---+
>>
>>
>>

Re: [DISSCUSS] Access to JDK internals only after dev mailing list consensus?

[Claude Warren, Jr via dev]

This change looks good to me.  It is clear and concise.

On Fri, Nov 4, 2022 at 9:50 PM Ekaterina Dimitrova 
wrote:

> 
>
> I finally got the chance to put down a proposal for a section at the end
> of the Cassandra Code Style document.
> Please help a fellow non-native speaker and definitely not a writer with
> some constructive criticism. :-)
> My proposal is in this commit -
>
> https://github.com/ekaterinadimitrova2/cassandra-website/commit/4a9edc7e88fd9fc2c6aa8a84290b75b02cac03bf
>
> I noticed the Dependencies section suggested in the past by Benedict was
> missing, even if we had a consensus around that. I added it back from the
> original doc.
>
> Best regards ,
> Ekaterina
>
> On Fri, 9 Sep 2022 at 13:34, Ekaterina Dimitrova 
> wrote:
>
>> Hi everyone,
>> Seems to me that people will be fine with heads up on the mailing list
>> and details on tickets?
>> Plus update of the Code Style to add a point around that, as suggested.
>>
>> I will leave this thread open a few more days and if there are no
>> objections I will continue with documenting it.
>>
>> Have a great weekend everyone!
>>
>> On Fri, 2 Sep 2022 at 14:01, Ekaterina Dimitrova 
>> wrote:
>>
>>> Git and jira , nothing specific
>>>
>>> On Fri, 2 Sep 2022 at 12:51, Derek Chen-Becker 
>>> wrote:
>>>
 I think it's fine to state it explicitly rather than making it an
 assumption. Are we tracking any usage of internals in the codebase
 currently?

 Cheers,

 Derek

 On Fri, Sep 2, 2022 at 6:30 AM Ekaterina Dimitrova <
 e.dimitr...@gmail.com> wrote:

>
>
> “ A quick heads up to the dev list with the jira would be sufficient
> for anybody interested in discussing it further to comment on the jira.”
>
> Agreed, I did’t mean voting but more or less we have the lazy
> consensus or sharing concerns. Discussing them on a ticket should be 
> enough
> but it needs to happen. Also, it shouldn’t be  more often than adding
> dependencies I guess.
>
> JDK team is only closing more and more internals and warning us about
> potential breakages. I want to prevent us from urgent fixing in patch
> releases and to ease the maintenance.
>
> I think ensuring that it is clearly documented why an exception is
> acceptable and what options were considered will be of benefit for
> maintenance. We can revise in time what has changed.
>
> “ . Unless absolutely needed we should avoid accessing the internals.
> Folks on this project should understand why. We can make the dangers of
> this explicit in our contributor documentation. ”
> +1
>
> On Fri, 2 Sep 2022 at 1:26, Dinesh Joshi  wrote:
>
>> Personally not opposed to this. However, this is something that
>> should be vetted closely by the reviewers. Unless absolutely needed we
>> should avoid accessing the internals. Folks on this project should
>> understand why. We can make the dangers of this explicit in our 
>> contributor
>> documentation. However, requiring an entire dev list discussion around it
>> seems unnecessary. A quick heads up to the dev list with the jira would 
>> be
>> sufficient for anybody interested in discussing it further to comment on
>> the jira. WDYT?
>>
>> Dinesh
>>
>> On Sep 1, 2022, at 8:31 AM, Ekaterina Dimitrova <
>> e.dimitr...@gmail.com> wrote:
>>
>> Hi everyone,
>>
>>
>> Some time ago we added a note to the project Cassandra Code Style:
>> “New dependencies should not be included without community consensus
>> first being obtained via a [DISCUSS] thread on the
>> dev@cassandra.apache.org mailing list”
>>
>> I would like to suggest also to add a point around accessing JDK
>> internals. Any  patch that suggests accessing internals and/or adding 
>> even
>> more add-opens/add-exports to be approved prior commit on the mailing 
>> list.
>>
>> It seems to me the project can only benefit of this visibility. If
>> something is accepted as an exception, we need to have the right
>> understanding and visibility of why; in some cases maybe to see for
>> alternatives, to have follow up tickets opened, ownership taken. In my
>> opinion this will be very helpful for maintaining the codebase.
>>
>> If others agree with that I can add a sentence to the Code Style.
>> Please let me know what you think.
>>
>> Best regards,
>> Ekaterina
>>
>>
>>

 --
 +---+
 | Derek Chen-Becker |
 | GPG Key available at https://keybase.io/dchenbecker and   |
 | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
 | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
 +---+

[DISCUSS] WIP pull request for CASSANDRA-17773

[Claude Warren, Jr via dev]

Greetings,

I have created a pull request [1] as a work in progress for CASSANDRA-17773.

The idea here is to move the boiler plate include of cassandra.in.sh and
cassandra-env.sh into one sourced script that also provides a function
(verify_env) to verify that required variables are set.

verify_env  will also display the value of the required and other requested
vars if the ENV_DEBUG environment var is set.

Documentation can be found in the cassandra-conf.sh file.
Example usage is provided in the modified nodetool script.

Please let me know if this seems like a good response to the issues from
CASSANDRA-17773

If this seems appropriate then my plan is to modify the other scripts in
the /bin directory to utilize the cassandra-conf.sh script.

Claude

[1] https://github.com/apache/cassandra/pull/1950

Re: CEP-23: Enhancement for Sparse Data Serialization

[Claude Warren, Jr via dev]

I see that there is one proposal that was discarded.  I wonder how that got
there.

On Tue, Oct 25, 2022 at 2:52 PM Josh McKenzie  wrote:

> ... I don't know that we've navigate that question before. My immediate
> reaction is as the proposer you should be able to close it down unless it's
> gone to a vote and/or a vote has passed.
>
> If someone else wants to pick it up later that's fine.
>
> On Tue, Oct 25, 2022, at 7:35 AM, Claude Warren, Jr via dev wrote:
>
> I would like to discard CEP-23.  As I am the proposer, is a vote required?
>
> What is the process?
>
> Claude
>
>
>

CEP-23: Enhancement for Sparse Data Serialization

[Claude Warren, Jr via dev]

I would like to discard CEP-23.  As I am the proposer, is a vote required?

What is the process?

Claude

Re: CEP-21 and complete cluster replacement.

[Claude Warren, Jr via dev]

My understanding of our process is (assuming we have 3 nodes A,B,C):

   - Add A' to the cluster with the same keyspace as A.
   - Remove A from the cluster.
   - Add B' to the cluster
   - Remove B from the cluster
   - Add C' to the cluster
   - Remove C from the cluster.

Currently these operations have to be performed in sequence.  My
understanding is that you can't add more than one node at a time.  What we
would like to do is do this is 3 steps:

   - Add A', B', C' to the cluster.
   - Wait for all 3 to be accepted and functioning.
   - Remove A, B, C from the cluster.

Does CEP-21 make this possible?

On Thu, Oct 20, 2022 at 1:43 PM Sam Tunnicliffe  wrote:

> I'm not sure I 100% understand the question, but the things covered in
> CEP-21 won't enable you to as an operator to bootstrap all your new nodes
> without fully joining, then perform an atomic CAS to replace the existing
> members. CEP-21 alone also won't solve all cross-version streaming issues,
> which is one reason performing topology-modifying operations like bootstrap
> & decommission during an upgrade are not generally considered a good idea.
>
> Transactional metadata will make the bootstrapping (and decommissioning)
> experience a whole lot more stable and predictable, so in the short term I
> would expect the recommended rolling approach to upgrades would improve
> significantly.
>
>
> > On 20 Oct 2022, at 12:24, Claude Warren, Jr via dev <
> dev@cassandra.apache.org> wrote:
> >
> > After CEP-21 would it be possible to take a cluster of 6 nodes, spin up
> 6 new nodes to duplicate the 6 existing nodes and then spin down the
> original 6 nodes.  Basically, I am thinking of the case where a cluster is
> running version x.y.z and want to run x.y.z+1, can they spin up an equal
> number of x.y.z+1 systems and replace the old ones without shutting down
> the cluster?
> >
> > We currently try something like this where we spin up 1 system and then
> drop 1 system until all the old nodes are replaced.  This process
> frequently runs into streaming failures while bootstrapping.
> >
> > Any insights would be appreciated.
> >
> > Claude
>
>

CEP-21 and complete cluster replacement.

[Claude Warren, Jr via dev]

After CEP-21 would it be possible to take a cluster of 6 nodes, spin up 6
new nodes to duplicate the 6 existing nodes and then spin down the original
6 nodes.  Basically, I am thinking of the case where a cluster is running
version x.y.z and want to run x.y.z+1, can they spin up an equal number of
x.y.z+1 systems and replace the old ones without shutting down the cluster?

We currently try something like this where we spin up 1 system and then
drop 1 system until all the old nodes are replaced.  This process
frequently runs into streaming failures while bootstrapping.

Any insights would be appreciated.

Claude

Re: [Discuss] CEP-24 Password validation and generation

[Claude Warren, Jr via dev]

Just to clarify, I have no objections to the current plan.

On Thu, Oct 13, 2022 at 2:56 PM Claude Warren, Jr 
wrote:

> I am not familiar with the Diagnostics framework but it sounds like it
> would satisfy the need.  Thanks for pointing it out.  I will dive into it
> to get an understanding of how it works.
>
> On Thu, Oct 13, 2022 at 1:52 PM Miklosovic, Stefan <
> stefan.mikloso...@netapp.com> wrote:
>
>> Hi Claude,
>>
>> we may also integrate with Diagnostics framework Cassandra already ships.
>> I would say this better suits to your requirements for observability. I am
>> not sure to what degree you are familiar with Diagnostics though. To give
>> you a better picture, events are fired and external observers (in the
>> framework called "subscribers") would be notified about the internal
>> accordingly. As of now, observers / subscribers are meant to integrate with
>> JMX through which these events flow.
>>
>> Do you think Diagnostics events would satisfy your needs?
>>
>> Regards
>>
>> 
>> From: Claude Warren, Jr via dev 
>> Sent: Thursday, October 13, 2022 14:43
>> To: dev@cassandra.apache.org
>> Subject: Re: [Discuss] CEP-24 Password validation and generation
>>
>> NetApp Security WARNING: This is an external email. Do not click links or
>> open attachments unless you recognize the sender and know the content is
>> safe.
>>
>>
>>
>> The only difference I see is that I see observability (observer) as being
>> a way to retrieve (or be notified about) data used within a process.
>> Logging on the other hand, is a preservation of a state discovered in an
>> observable object.  Observability can drive logging but it can also drive
>> aggregate statistics in grafana, and things like that.
>>
>> My reading of the CEP-3 is that it is intended to provide system-wide
>> soft and hard limits, it is not an observability framework.  It makes sense
>> for the validator to implement CEP-3 but I think that an observability
>> interface is required as well.
>>
>> On Thu, Oct 13, 2022 at 12:36 PM Miklosovic, Stefan <
>> stefan.mikloso...@netapp.com<mailto:stefan.mikloso...@netapp.com>> wrote:
>> Hi Claude,
>>
>> all you say makes sense to me. I do not see any discrepancies. It will be
>> logged as discussed already.
>>
>> The complexity of password validation is partly covered by the library we
>> want to use (Passay). It will inform you in a very detailed manner when it
>> comes to what violations of a policy there are. We are not going to invent
>> a wheel here, fortunately.
>>
>> Terminology you used - "observer" - is Guardrail itself (CEP-3). It will
>> be the one doing reporting e.g by logging and returning warnings / errors,
>> if any, back to user who executed that query.
>>
>> The approach we took indeed can also be extended in such a way that it
>> would be possible to know what was the last time a password was changed for
>> some user. This is the direct consequence of us having a table of previous
>> password for checking that a user is not reusing them. There is a timestamp
>> column specified here (1) if you check the schema of that table closely so
>> to answer "when was the password changed lastly" is rather easy to know -
>> "select created from system_auth.previous passwords where role = 'stefan'
>> limit 1"
>>
>> To your requirements:
>> A simple implementation of the validator that performs series of
>> configurable tests against the password would probably be sufficient for
>> the validation
>>
>> Sure, this is configurable, by either implementing a custom validator if
>> you find the default one insufficient or configuring the default one
>> accordingly.
>>
>> "A simple implementation of the observer that logs the messages Jeff
>> suggested would probably be sufficient."
>>
>> Yes, no problem with logging from Guardrail directly.
>>
>> (1)
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-24%3A+Password+validation+and+generation#CEP24:Passwordvalidationandgeneration-Validationofanewpasswordagainstpreviouspasswords
>>
>> Regards
>>
>> 
>> From: Claude Warren, Jr > claude.war...@aiven.io>>
>> Sent: Thursday, October 13, 2022 12:50
>> To: Miklosovic, Stefan
>> Cc: dev@cassandra.apache.org<mailto:dev@cassandra.apache.org>
>> Subject: Re: [Discuss] CEP-24 Password validation and generation

Re: [Discuss] CEP-24 Password validation and generation

[Claude Warren, Jr via dev]

I am not familiar with the Diagnostics framework but it sounds like it
would satisfy the need.  Thanks for pointing it out.  I will dive into it
to get an understanding of how it works.

On Thu, Oct 13, 2022 at 1:52 PM Miklosovic, Stefan <
stefan.mikloso...@netapp.com> wrote:

> Hi Claude,
>
> we may also integrate with Diagnostics framework Cassandra already ships.
> I would say this better suits to your requirements for observability. I am
> not sure to what degree you are familiar with Diagnostics though. To give
> you a better picture, events are fired and external observers (in the
> framework called "subscribers") would be notified about the internal
> accordingly. As of now, observers / subscribers are meant to integrate with
> JMX through which these events flow.
>
> Do you think Diagnostics events would satisfy your needs?
>
> Regards
>
> ________________
> From: Claude Warren, Jr via dev 
> Sent: Thursday, October 13, 2022 14:43
> To: dev@cassandra.apache.org
> Subject: Re: [Discuss] CEP-24 Password validation and generation
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> The only difference I see is that I see observability (observer) as being
> a way to retrieve (or be notified about) data used within a process.
> Logging on the other hand, is a preservation of a state discovered in an
> observable object.  Observability can drive logging but it can also drive
> aggregate statistics in grafana, and things like that.
>
> My reading of the CEP-3 is that it is intended to provide system-wide soft
> and hard limits, it is not an observability framework.  It makes sense for
> the validator to implement CEP-3 but I think that an observability
> interface is required as well.
>
> On Thu, Oct 13, 2022 at 12:36 PM Miklosovic, Stefan <
> stefan.mikloso...@netapp.com<mailto:stefan.mikloso...@netapp.com>> wrote:
> Hi Claude,
>
> all you say makes sense to me. I do not see any discrepancies. It will be
> logged as discussed already.
>
> The complexity of password validation is partly covered by the library we
> want to use (Passay). It will inform you in a very detailed manner when it
> comes to what violations of a policy there are. We are not going to invent
> a wheel here, fortunately.
>
> Terminology you used - "observer" - is Guardrail itself (CEP-3). It will
> be the one doing reporting e.g by logging and returning warnings / errors,
> if any, back to user who executed that query.
>
> The approach we took indeed can also be extended in such a way that it
> would be possible to know what was the last time a password was changed for
> some user. This is the direct consequence of us having a table of previous
> password for checking that a user is not reusing them. There is a timestamp
> column specified here (1) if you check the schema of that table closely so
> to answer "when was the password changed lastly" is rather easy to know -
> "select created from system_auth.previous passwords where role = 'stefan'
> limit 1"
>
> To your requirements:
> A simple implementation of the validator that performs series of
> configurable tests against the password would probably be sufficient for
> the validation
>
> Sure, this is configurable, by either implementing a custom validator if
> you find the default one insufficient or configuring the default one
> accordingly.
>
> "A simple implementation of the observer that logs the messages Jeff
> suggested would probably be sufficient."
>
> Yes, no problem with logging from Guardrail directly.
>
> (1)
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-24%3A+Password+validation+and+generation#CEP24:Passwordvalidationandgeneration-Validationofanewpasswordagainstpreviouspasswords
>
> Regards
>
> 
> From: Claude Warren, Jr  claude.war...@aiven.io>>
> Sent: Thursday, October 13, 2022 12:50
> To: Miklosovic, Stefan
> Cc: dev@cassandra.apache.org<mailto:dev@cassandra.apache.org>
> Subject: Re: [Discuss] CEP-24 Password validation and generation
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> I think we might be in violent agreement here.
>
> The point I was trying to make is that the rules for valid passwords are
> many and varied.  I have worked at places where they wanted to know the
> time since the last password change, this was used to prevent the rapid
> change of password to  

Re: [Discuss] CEP-24 Password validation and generation

[Claude Warren, Jr via dev]

 with
> existing role in the db why he failed to log in, when it comes to this CEP
> (is not it actually already in place? CQLSH says your username / password
> combo is invalid on login already) This CEP has nothing to do with it.
>
> What we have in mind, I think, it is more about informing him about the
> details when the password he tries to set (upon role creation) or change
> (via role alteration), is not valid, based on the policy.
>
> I reckon that what Jeff simply wants to see is a log if such change was
> successful or not. Lets repeat here what Jeff would like to see:
>
> "Password changed for user X, complying with policies (reuse, complexity,
> entropy)"
> "ERROR: Password rejected due to policy violation (reuse)"
> "ERROR: Password rejected due to policy violation (complexity)"
> "ERROR: Password rejected due to policy violation (entropy)"
>
> This is a generalized version of what we already have in place in CEP, we
> have there information like:
>
> Password must be 10 or more characters in length. Password must contain 2
> or more uppercase characters. Password matches 3 of 4 character rules, but
> 4 are required.
> Password matches one of 5 previous passwords.
> Password must be 12 or more characters in length
>
> Now, I have to admit that the information we provide above, in contrast of
> what Jeff mentioned, is quite verbose. It is questionable whether we should
> be so specific or whether more generalized version is enough.
>
> Maybe two versions of the logs would be the most appropriate - ours (more
> detailed) would be returned to a user in cqlsh as a warning / error after
> unsuccessful query execution but the messages Jeff mentioned would be
> written in system logs via slf4j. So we would be detailed for a user but
> general for auditing purposes.
>
> Do you think this makes sense to you all? I think this is want you said,
> more or less, in your middle paragraph, just formulated differently.
>
> I agree with Jackson with the password meter e-mail. After all, if
> somebody really wants that to happen, since our solution is pluggable,
> people can implement their own password-meter-based solution if they find
> it necessary.
>
> To fail a password when it is reused (or found among previous n). I am on
> the edge here. I understand what Josh is telling, that we can go just so
> far when it comes to prevent people from doing wrong things, maybe
> increasing the password history to 20 last passwords would be enough.
> Anyway, I plan to make this historical password verification optional so it
> might be turned on / off on demand.
>
> Finally, when it comes to password dictionaries. This might be included in
> the CEP but I would keep it out for the very first implementation and it
> can be finished afterwards in some other commit. I do not find it
> absolutely necessary to do it right now.
>
> Regards,
>
> Stefan
>
> 
> From: Claude Warren, Jr via dev  dev@cassandra.apache.org>>
> Sent: Thursday, October 13, 2022 9:44
> To: dev@cassandra.apache.org<mailto:dev@cassandra.apache.org>
> Subject: Fwd: [Discuss] CEP-24 Password validation and generation
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
>
> I managed not to send this to the mailaing list...
>
>
> I don't know the govt spec. but there is a US govt security level where
> you are not allowed to inform the user why the login failed.
>
>
> It seems to me that there are 2 intertwined components being discussed.
>
> 1) A component to perform a user password change capability
>
> 2) A plugable validation component.
>
> 3) A pluggable observability component.
>
> Without a validation component all passwords are valid and provides user
> messages for failures.  Validation receives the new password and some
> list of old passwords as arguments.  Validation returns a structure
> comprising the success/failure, the user message, internal result,
> internal result message.
>
> The observability implementations could log the results, send counts to
> Grafana, etc.  If there is no observer then no results are presented.
>
>
> Alternatively the validation could accept the observability component as
> an argument and pass the internal result and internal result message
> directly to the observability component.
>
>
>

Re: [Discuss] CEP-24 Password validation and generation

[Claude Warren, Jr via dev]

I think we might be in violent agreement here.

The point I was trying to make is that the rules for valid passwords are
many and varied.  I have worked at places where they wanted to know the
time since the last password change, this was used to prevent the rapid
change of password to  get back to the original one (I think 5 was the
example earlier).  Anyway, the point was, identify the information
necessary from the system to fulfill the rules we think of (so far this is
the new password, a list of old passwords, and the time of the last
password change) and call a validator plugin passing it the new password,
list of passwords, date of last change, and an observer instance.

The validator implementation will verify the instance and report any issues
to the observer and return true/false and potentially a user message.

Any logging is attached to the observer, any reporting to grafana or
similar reporting is attached to the observer.

A simple implementation of the validator that performs series of
configurable tests against the password would probably be sufficient for
the validation
A simple implementation of the observer that logs the messages Jeff
suggested would probably be sufficient.

Both would allow much more complex validation and/or reporting as necessary.

On Thu, Oct 13, 2022 at 9:26 AM Miklosovic, Stefan <
stefan.mikloso...@netapp.com> wrote:

> Hi Claude,
>
> you said: "I don't know the govt spec. but there is a US govt security
> level where you are not allowed to inform the user why the login failed."
>
> I do not think this is the case. Nobody is going to inform a user with
> existing role in the db why he failed to log in, when it comes to this CEP
> (is not it actually already in place? CQLSH says your username / password
> combo is invalid on login already) This CEP has nothing to do with it.
>
> What we have in mind, I think, it is more about informing him about the
> details when the password he tries to set (upon role creation) or change
> (via role alteration), is not valid, based on the policy.
>
> I reckon that what Jeff simply wants to see is a log if such change was
> successful or not. Lets repeat here what Jeff would like to see:
>
> "Password changed for user X, complying with policies (reuse, complexity,
> entropy)"
> "ERROR: Password rejected due to policy violation (reuse)"
> "ERROR: Password rejected due to policy violation (complexity)"
> "ERROR: Password rejected due to policy violation (entropy)"
>
> This is a generalized version of what we already have in place in CEP, we
> have there information like:
>
> Password must be 10 or more characters in length. Password must contain 2
> or more uppercase characters. Password matches 3 of 4 character rules, but
> 4 are required.
> Password matches one of 5 previous passwords.
> Password must be 12 or more characters in length
>
> Now, I have to admit that the information we provide above, in contrast of
> what Jeff mentioned, is quite verbose. It is questionable whether we should
> be so specific or whether more generalized version is enough.
>
> Maybe two versions of the logs would be the most appropriate - ours (more
> detailed) would be returned to a user in cqlsh as a warning / error after
> unsuccessful query execution but the messages Jeff mentioned would be
> written in system logs via slf4j. So we would be detailed for a user but
> general for auditing purposes.
>
> Do you think this makes sense to you all? I think this is want you said,
> more or less, in your middle paragraph, just formulated differently.
>
> I agree with Jackson with the password meter e-mail. After all, if
> somebody really wants that to happen, since our solution is pluggable,
> people can implement their own password-meter-based solution if they find
> it necessary.
>
> To fail a password when it is reused (or found among previous n). I am on
> the edge here. I understand what Josh is telling, that we can go just so
> far when it comes to prevent people from doing wrong things, maybe
> increasing the password history to 20 last passwords would be enough.
> Anyway, I plan to make this historical password verification optional so it
> might be turned on / off on demand.
>
> Finally, when it comes to password dictionaries. This might be included in
> the CEP but I would keep it out for the very first implementation and it
> can be finished afterwards in some other commit. I do not find it
> absolutely necessary to do it right now.
>
> Regards,
>
> Stefan
>
> 
> From: Claude Warren, Jr via dev 
> Sent: Thursday, October 13, 2022 9:44
> To: dev@cassandra.apache.org
> Subject: Fwd: [Discuss] CEP-24 Password validation and generation
>
>

Fwd: [Discuss] CEP-24 Password validation and generation

[Claude Warren, Jr via dev]

I managed not to send this to the mailaing list...


I don't know the govt spec. but there is a US govt security level where
you are not allowed to inform the user why the login failed.


It seems to me that there are 2 intertwined components being discussed.

1) A component to perform a user password change capability

2) A plugable validation component.

3) A pluggable observability component.

Without a validation component all passwords are valid and provides user
messages for failures.  Validation receives the new password and some
list of old passwords as arguments.  Validation returns a structure
comprising the success/failure, the user message, internal result,
internal result message.

The observability implementations could log the results, send counts to
Grafana, etc.  If there is no observer then no results are presented.


Alternatively the validation could accept the observability component as
an argument and pass the internal result and internal result message
directly to the observability component.

Re: [DISCUSS] CEP-21: Transactional Cluster Metadata

[Claude Warren, Jr via dev]

Should

(**) It may seem counterintuitive, that A is being written to even after
> we've stopped reading from it. This is done in order to guarantee that by
> the time we stop writing to the node giving up the range, there is no
> coordinator that may attempt reading from it without learning about *at
> least* the epoch where it is not a part of a read set. In other words, we
> have to keep writing until there's any chance there might be a reader.


instead read:

(**) It may seem counterintuitive, that A is being written to even after
we've stopped reading from it. This is done in order to guarantee that by
the time we stop writing to the node giving up the range, there is no
coordinator that may attempt reading from it without learning about *at
least* the epoch where it is not a part of a read set. In other words, we
have to keep writing *while* there's any chance there might be a reader.

On Tue, Aug 23, 2022 at 7:13 PM Mick Semb Wever  wrote:

>
>
> I just want to say I’m really excited about this work. It’s one of the
>> last remaining major inadequacies of the project that makes it hard for
>> people to deploy, and hard for us to develop.
>>
>>
>
> Second this. And what a solid write up Sam - it's a real joy reading this
> CEP.
>

Re: [DISCUSS] CEP-20: Dynamic Data Masking

[Claude Warren, Jr via dev]

This change appears to be looking at two aspects:

   1. Add metadata to columns
   2. Add functionality based on the metadata.

If the system had a generic user defined metadata and the ability to define
filter functions at the point where data are being returned to the client
it would be possible for users implement this filter, or any other filter
on the data.

The concept of user defined metadata and filters could be applied to
other parts of the system as well.  For example, if the metadata were
accessible from UDFs the metadata could be used in low level filters to
remove rows from queries before they were returned.




On Wed, Aug 24, 2022 at 9:29 AM Claude Warren, Jr 
wrote:

> The PCI DSS Standard v4_0
> 
>  requires
> that credit card numbers stored on the system must be "rendered
> unreadable", thus this proposal is _NOT_ a good way to protect credit card
> numbers.  In fact, for any critically sensitive data this is not an
> appropriate solution.  However, there seems to be agreement that it is
> appropriate for obfuscating some data in some queries by some users.
>
>
>
> On Wed, Aug 24, 2022 at 9:02 AM Benjamin Lerer  wrote:
>
>> Is it typical for a masking feature to make no effort to prevent
>>> unmasking? I’m just struggling to see the value of this without such
>>> mechanisms. Otherwise it’s just a default formatter, and we should consider
>>> renaming the feature IMO
>>
>>
>> The security that Dynamic Data Masking is bringing is related to how you
>> make use of the feature. It is somehow the same with passwords. If you use
>> a weak password it does not bring much security.
>> Masking a field like people's gender is useless because you will be able
>> to determine its value in one query. On the other hand masking credit card
>> numbers makes a lot of sense as it will complicate the life of the person
>> trying to have access to it and the queries needed to reach the information
>> will leave some clear traces in the audit log.
>>
>> Dynamic Data Masking is not a magic bullet. Nevertheless, it is a good
>> way to protect sensitive data like credit card numbers or passwords.
>>
>>
>> Le mer. 24 août 2022 à 09:40, Benedict  a écrit :
>>
>>> Is it typical for a masking feature to make no effort to prevent
>>> unmasking? I’m just struggling to see the value of this without such
>>> mechanisms. Otherwise it’s just a default formatter, and we should consider
>>> renaming the feature IMO
>>>
>>> On 23 Aug 2022, at 21:27, Andrés de la Peña 
>>> wrote:
>>>
>>> 
>>> As mentioned in the CEP document, dynamic data masking doesn't try to
>>> prevent malicious users with SELECT permissions to indirectly guess the
>>> real value of the masked value. This can easily be done by just trying
>>> values on the WHERE clause of SELECT queries. DDM would not be a
>>> replacement for proper column-level permissions.
>>>
>>> The data served by the database is usually consumed by applications that
>>> present this data to end users. These end users are not necessarily the
>>> users directly connecting to the database. With DDM, it would be easy for
>>> applications to mask sensitive data that is going to be consumed by the end
>>> users. However, the users directly connecting to the database should be
>>> trusted, provided that they have the right SELECT permissions.
>>>
>>> In other words, DDM doesn't directly protect the data, but it eases the
>>> production of protected data.
>>>
>>> Said that, we could later go one step ahead and add a way to prevent
>>> untrusted users from inferring the masked data. That could be done adding a
>>> new permission required to use certain columns on WHERE clauses, different
>>> to the current SELECT permission. That would play especially well with
>>> column-level permissions, which is something that we still have pending.
>>>
>>> On Tue, 23 Aug 2022 at 19:13, Aaron Ploetz 
>>> wrote:
>>>
 Applying this should prevent querying on a field, else you could leak
> its contents, surely?
>

 In theory, yes.  Although I could see folks doing something like this:

 SELECT COUNT(*) FROM patients
 WHERE year_of_birth = 2002
 AND date_of_birth >= '2002-04-01'
 AND date_of_birth < '2002-11-01';

 In this case, the rows containing the masked key column(s) could be
 filtered on without revealing the actual data.  But again, that's probably
 better for a "phase 2" of the implementation.

 Agreed on not being a queryable field. That would also preclude
> secondary indexing, right?


 Yes, that's my thought as well.

 On Tue, Aug 23, 2022 at 12:42 PM Derek Chen-Becker <
 de...@chen-becker.org> wrote:

> Agreed on not being a queryable field. That would also preclude
> secondary indexing, right?
>
> On Tue, Aug 23, 2022 at 11:20 AM Benedict  wrote:
>
>> Applying this should prevent querying on a 

Re: [DISCUSS] CEP-20: Dynamic Data Masking

[Claude Warren, Jr via dev]

The PCI DSS Standard v4_0

requires
that credit card numbers stored on the system must be "rendered
unreadable", thus this proposal is _NOT_ a good way to protect credit card
numbers.  In fact, for any critically sensitive data this is not an
appropriate solution.  However, there seems to be agreement that it is
appropriate for obfuscating some data in some queries by some users.



On Wed, Aug 24, 2022 at 9:02 AM Benjamin Lerer  wrote:

> Is it typical for a masking feature to make no effort to prevent
>> unmasking? I’m just struggling to see the value of this without such
>> mechanisms. Otherwise it’s just a default formatter, and we should consider
>> renaming the feature IMO
>
>
> The security that Dynamic Data Masking is bringing is related to how you
> make use of the feature. It is somehow the same with passwords. If you use
> a weak password it does not bring much security.
> Masking a field like people's gender is useless because you will be able
> to determine its value in one query. On the other hand masking credit card
> numbers makes a lot of sense as it will complicate the life of the person
> trying to have access to it and the queries needed to reach the information
> will leave some clear traces in the audit log.
>
> Dynamic Data Masking is not a magic bullet. Nevertheless, it is a good way
> to protect sensitive data like credit card numbers or passwords.
>
>
> Le mer. 24 août 2022 à 09:40, Benedict  a écrit :
>
>> Is it typical for a masking feature to make no effort to prevent
>> unmasking? I’m just struggling to see the value of this without such
>> mechanisms. Otherwise it’s just a default formatter, and we should consider
>> renaming the feature IMO
>>
>> On 23 Aug 2022, at 21:27, Andrés de la Peña  wrote:
>>
>> 
>> As mentioned in the CEP document, dynamic data masking doesn't try to
>> prevent malicious users with SELECT permissions to indirectly guess the
>> real value of the masked value. This can easily be done by just trying
>> values on the WHERE clause of SELECT queries. DDM would not be a
>> replacement for proper column-level permissions.
>>
>> The data served by the database is usually consumed by applications that
>> present this data to end users. These end users are not necessarily the
>> users directly connecting to the database. With DDM, it would be easy for
>> applications to mask sensitive data that is going to be consumed by the end
>> users. However, the users directly connecting to the database should be
>> trusted, provided that they have the right SELECT permissions.
>>
>> In other words, DDM doesn't directly protect the data, but it eases the
>> production of protected data.
>>
>> Said that, we could later go one step ahead and add a way to prevent
>> untrusted users from inferring the masked data. That could be done adding a
>> new permission required to use certain columns on WHERE clauses, different
>> to the current SELECT permission. That would play especially well with
>> column-level permissions, which is something that we still have pending.
>>
>> On Tue, 23 Aug 2022 at 19:13, Aaron Ploetz  wrote:
>>
>>> Applying this should prevent querying on a field, else you could leak
 its contents, surely?

>>>
>>> In theory, yes.  Although I could see folks doing something like this:
>>>
>>> SELECT COUNT(*) FROM patients
>>> WHERE year_of_birth = 2002
>>> AND date_of_birth >= '2002-04-01'
>>> AND date_of_birth < '2002-11-01';
>>>
>>> In this case, the rows containing the masked key column(s) could be
>>> filtered on without revealing the actual data.  But again, that's probably
>>> better for a "phase 2" of the implementation.
>>>
>>> Agreed on not being a queryable field. That would also preclude
 secondary indexing, right?
>>>
>>>
>>> Yes, that's my thought as well.
>>>
>>> On Tue, Aug 23, 2022 at 12:42 PM Derek Chen-Becker <
>>> de...@chen-becker.org> wrote:
>>>
 Agreed on not being a queryable field. That would also preclude
 secondary indexing, right?

 On Tue, Aug 23, 2022 at 11:20 AM Benedict  wrote:

> Applying this should prevent querying on a field, else you could leak
> its contents, surely? This pretty much prohibits using it in a clustering
> key, and a partition key with the ordered partitioner - but probably also 
> a
> hashed partitioner since we do not use a cryptographic hash and the hash
> function is well defined.
>
> We probably also need to ensure that any ALLOW FILTERING queries on
> such a field are disabled.
>
> Plausibly the data could be cryptographically jumbled before using it
> in a primary key component (or permitting filtering), but it is probably
> easier and safer to exclude for now…
>
> On 23 Aug 2022, at 18:13, Aaron Ploetz  wrote:
>
> 
> Some thoughts on this one:
>
> In a prior job, we'd give app teams access to a single keyspace, 

Re: [DISCUSS] CEP-20: Dynamic Data Masking

[Claude Warren, Jr via dev]

This seems to me to be a client display filter, applied at the last moment
as data are streaming back to the client.  It has no impact on any keys,
queries or secondary internal index or materialized view.  It simply
prevents the display from showing the complete value.  It does not preclude
determining what some values are by building carefully crafted queries.





On Wed, Aug 24, 2022 at 8:40 AM Benedict  wrote:

> Is it typical for a masking feature to make no effort to prevent
> unmasking? I’m just struggling to see the value of this without such
> mechanisms. Otherwise it’s just a default formatter, and we should consider
> renaming the feature IMO
>
> On 23 Aug 2022, at 21:27, Andrés de la Peña  wrote:
>
> 
> As mentioned in the CEP document, dynamic data masking doesn't try to
> prevent malicious users with SELECT permissions to indirectly guess the
> real value of the masked value. This can easily be done by just trying
> values on the WHERE clause of SELECT queries. DDM would not be a
> replacement for proper column-level permissions.
>
> The data served by the database is usually consumed by applications that
> present this data to end users. These end users are not necessarily the
> users directly connecting to the database. With DDM, it would be easy for
> applications to mask sensitive data that is going to be consumed by the end
> users. However, the users directly connecting to the database should be
> trusted, provided that they have the right SELECT permissions.
>
> In other words, DDM doesn't directly protect the data, but it eases the
> production of protected data.
>
> Said that, we could later go one step ahead and add a way to prevent
> untrusted users from inferring the masked data. That could be done adding a
> new permission required to use certain columns on WHERE clauses, different
> to the current SELECT permission. That would play especially well with
> column-level permissions, which is something that we still have pending.
>
> On Tue, 23 Aug 2022 at 19:13, Aaron Ploetz  wrote:
>
>> Applying this should prevent querying on a field, else you could leak its
>>> contents, surely?
>>>
>>
>> In theory, yes.  Although I could see folks doing something like this:
>>
>> SELECT COUNT(*) FROM patients
>> WHERE year_of_birth = 2002
>> AND date_of_birth >= '2002-04-01'
>> AND date_of_birth < '2002-11-01';
>>
>> In this case, the rows containing the masked key column(s) could be
>> filtered on without revealing the actual data.  But again, that's probably
>> better for a "phase 2" of the implementation.
>>
>> Agreed on not being a queryable field. That would also preclude secondary
>>> indexing, right?
>>
>>
>> Yes, that's my thought as well.
>>
>> On Tue, Aug 23, 2022 at 12:42 PM Derek Chen-Becker 
>> wrote:
>>
>>> Agreed on not being a queryable field. That would also preclude
>>> secondary indexing, right?
>>>
>>> On Tue, Aug 23, 2022 at 11:20 AM Benedict  wrote:
>>>
 Applying this should prevent querying on a field, else you could leak
 its contents, surely? This pretty much prohibits using it in a clustering
 key, and a partition key with the ordered partitioner - but probably also a
 hashed partitioner since we do not use a cryptographic hash and the hash
 function is well defined.

 We probably also need to ensure that any ALLOW FILTERING queries on
 such a field are disabled.

 Plausibly the data could be cryptographically jumbled before using it
 in a primary key component (or permitting filtering), but it is probably
 easier and safer to exclude for now…

 On 23 Aug 2022, at 18:13, Aaron Ploetz  wrote:

 
 Some thoughts on this one:

 In a prior job, we'd give app teams access to a single keyspace, and
 two roles: a read-write role and a read-only role.  In some cases, a
 "privileged" application role was also requested.  Depending on the
 requirements, I could see the UNMASK permission being applied to the RW or
 privileged roles.  But if there's a problem on the table and the operators
 go in to investigate, they will likely use a SUPERUSER account, and they'll
 see that data.

 How hard would it be for SUPERUSERs to *not* automatically get the
 UNMASK permission?

 I'll also echo the concerns around masking primary key components.
 It's highly likely that certain personal data properties would be used as a
 partition or clustering key (ex: range query for people born within a
 certain timeframe).  In addition to the "breaks existing" concern, I'm
 curious about the challenges around getting that to work with the current
 primary key implementation.

 Does this first implementation only apply to payload (non-key)
 columns?  The examples in the CEP currently do not show primary key
 components being masked.

 Thanks,

 Aaron


 On Tue, Aug 23, 2022 at 6:44 AM Henrik Ingo 
 wrote:

> 

Re: [DISCUSS] CEP-20: Dynamic Data Masking

[Claude Warren, Jr via dev]

I am more interested in the motivation where it is stated:

Many users have the need of masking sensitive data, such as contact info,
> age, gender, credit card numbers, etc. Dynamic data masking (DDM) allows to
> obscure sensitive information while still allowing access to the masked
> columns, and without changing the stored data.


There is an unspoken assumption that the stored data format can not be
changed.  It feels like this solution is starting from a false premise.
Throughout the document there are guard statements about how this does not
replace encryption.  Isn't there an assumption here that encryption can not
be used?  Would we not be better served to build in an encryption strategy
that keeps the data encrypted until the user shows permissions to decrypt,
like the unmask property?  An encryption strategy that can work within the
Cassandra internals?

I think that issue is that there are some data fields that should not be
discoverable by unauthorized users/systems, and I think this solution masks
that issue.  I fear that this capability will be seized upon by pointy
haired managers as a cheaper alternative to encryption, regardless of the
warnings otherwise, and that as a whole will harm the Cassandra ecosystem.

Yes, encryption is more difficult to implement and will take longer, but
this feels like a sticking plaster that distracts from that underlying
issue.

my 0.02

On Mon, Aug 22, 2022 at 12:30 AM Andrés de la Peña 
wrote:

> > If the column names are the same for masked and unmasked data, it would
>> impact existing applications. I am curious what the transition plan look
>> like for applications that expect unmasked data?
>
> For example, let’s say you store SSNs and Birth dates. Upon enabling this
>> feature, let’s say the app user is not given the UNMASK permission. Now the
>> app is receiving masked values for these columns. This is fine for most
>> read only applications. However, a lot of times these columns may be used
>> as primary keys or part of primary keys in other tables. This would break
>> existing applications.
>> How would this work in mixed mode when  ew nodes in the cluster are
>> masking data and others aren’t? How would it impact the driver?
>> How would the application learn that the column values are masked? This
>> is important in case a user has UNMASK permission and then later taken
>> away. Again this would break a lot of applications.
>
>
> Changing the masking of a column is a schema change, and as such it can be
> risky for existing applications. However, differently to deleting a column
> or revoking a SELECT permission, suddenly activating masking might pass
> undetected for existing applications.
>
> Applications developed after the introduction of this feature can check
> the table schema to know if a column is masked or not. We can even add a
> specific system view to ease this, if we think it's worth it. However,
> administrators should not activate masking when there could be applications
> that are not aware of the feature. We should be clear about this in the
> documentation.
>
> This is the way data masking seems to work in the databases I've checked.
> I also though that we could just change the name of the column when it's
> masked to something as "masked(column_name)", as it is discussed in the CEP
> document. This would make it impossible to miss that a column is masked.
> However, applications should be prepared to use different column names when
> reading result sets, depending on whether the data is masked for them or
> not. None of the databases mentioned on the "other databases" section of
> the CEP does this kind of column renaming, so it might be a kind of exotic
> behaviour. wdyt?
>
> On Fri, 19 Aug 2022 at 19:17, Andrés de la Peña 
> wrote:
>
>> > This type of feature is very useful, but it may be easier to analyze
>>> this proposal if it’s compared with other DDM implementations from other
>>> databases? Would it be reasonable to add a table to the proposal comparing
>>> syntax and output from eg Azure SQL vs Cassandra vs whatever ?
>>
>>
>> Good idea. I have added a section at the end of the document briefly
>> describing how some other databases deal with data masking, and with links
>> to their documentation for the topic. I am not an expert in none of those
>> databases, so please take my comments there with a grain of salt.
>>
>> On Fri, 19 Aug 2022 at 17:30, Jeff Jirsa  wrote:
>>
>>> This type of feature is very useful, but it may be easier to analyze
>>> this proposal if it’s compared with other DDM implementations from other
>>> databases? Would it be reasonable to add a table to the proposal comparing
>>> syntax and output from eg Azure SQL vs Cassandra vs whatever ?
>>>
>>>
>>> On Aug 19, 2022, at 4:50 AM, Andrés de la Peña 
>>> wrote:
>>>
>>> 
>>> Hi everyone,
>>>
>>> I'd like to start a discussion about this proposal for dynamic data
>>> masking:
>>> 

Re: Is this an MV bug?

[Claude Warren, Jr via dev]

Perhaps my diagram was not clear.  I am starting with mutations on the base
table.  I assume they are not bundled together so from separate CQL
statements.

On Fri, Aug 19, 2022 at 11:11 AM Claude Warren, Jr 
wrote:

> If each mutation comes from a separate CQL they would be separate, no?
>
>
> On Fri, Aug 19, 2022 at 10:17 AM Benedict  wrote:
>
>> If M1 and M2 both operate over the same partition key they won’t be
>> separate mutations, they should be combined into a single mutation before
>> submission to SP.mutate
>>
>> > On 19 Aug 2022, at 10:05, Claude Warren, Jr via dev <
>> dev@cassandra.apache.org> wrote:
>> >
>> > 
>> >
>> > # Table definitions
>> >
>> > Table [ Primary key ] other data
>> > base  [ A B C ] D E
>> > MV[ D C ] A B E
>> >
>> >
>> > # Initial  data
>> > base   -> MV
>> > [ a b c ] d e  -> [d c] a b e
>> > [ a' b c ] d e -> [d c] a' b e
>> >
>> >
>> > ## Mutations -> expected outcome
>> >
>> > M1: base [ a b c ] d e'  -> MV [ d c ] a b e'
>> > M2: base [ a b c ] d' e -> MV [ d' c ] a b e
>> >
>> > ## processing bug
>> > Assume lock can not be obtained during processing of M1.
>> >
>> > The mutation M1 sleeps to wait for lock. (Trunk Keyspace.java : 601 )
>> >
>> > Assume M2 obtains the lock and executes.
>> >
>> > MV is now
>> > [ d' c ] a b e
>> >
>> > M1 then obtains the lock and executes
>> >
>> > MV is now
>> > [ d c ] a b e'
>> > [ d' c] a b e
>> >
>> > base is
>> > [ a b c ] d e'
>> >
>> > MV entry "[ d' c ] a b e" is orphaned
>> >
>> >
>>
>>

Re: Is this an MV bug?

[Claude Warren, Jr via dev]

If each mutation comes from a separate CQL they would be separate, no?


On Fri, Aug 19, 2022 at 10:17 AM Benedict  wrote:

> If M1 and M2 both operate over the same partition key they won’t be
> separate mutations, they should be combined into a single mutation before
> submission to SP.mutate
>
> > On 19 Aug 2022, at 10:05, Claude Warren, Jr via dev <
> dev@cassandra.apache.org> wrote:
> >
> > 
> >
> > # Table definitions
> >
> > Table [ Primary key ] other data
> > base  [ A B C ] D E
> > MV[ D C ] A B E
> >
> >
> > # Initial  data
> > base   -> MV
> > [ a b c ] d e  -> [d c] a b e
> > [ a' b c ] d e -> [d c] a' b e
> >
> >
> > ## Mutations -> expected outcome
> >
> > M1: base [ a b c ] d e'  -> MV [ d c ] a b e'
> > M2: base [ a b c ] d' e -> MV [ d' c ] a b e
> >
> > ## processing bug
> > Assume lock can not be obtained during processing of M1.
> >
> > The mutation M1 sleeps to wait for lock. (Trunk Keyspace.java : 601 )
> >
> > Assume M2 obtains the lock and executes.
> >
> > MV is now
> > [ d' c ] a b e
> >
> > M1 then obtains the lock and executes
> >
> > MV is now
> > [ d c ] a b e'
> > [ d' c] a b e
> >
> > base is
> > [ a b c ] d e'
> >
> > MV entry "[ d' c ] a b e" is orphaned
> >
> >
>
>

Is this an MV bug?

[Claude Warren, Jr via dev]

# Table definitions

Table [ Primary key ] other data
base  [ A B C ] D E
MV[ D C ] A B E


# Initial  data
base   -> MV
[ a b c ] d e  -> [d c] a b e
[ a' b c ] d e -> [d c] a' b e


## Mutations -> expected outcome

M1: base [ a b c ] d e'  -> MV [ d c ] a b e'
M2: base [ a b c ] d' e -> MV [ d' c ] a b e

## processing bug
Assume lock can not be obtained during processing of M1.

The mutation M1 sleeps to wait for lock. (Trunk Keyspace.java : 601 )

Assume M2 obtains the lock and executes.

MV is now
[ d' c ] a b e

M1 then obtains the lock and executes

MV is now
[ d c ] a b e'
[ d' c] a b e

base is
[ a b c ] d e'

MV entry "[ d' c ] a b e" is orphaned

Re: [Proposal] add pull request template

[Claude Warren, Jr via dev]

Since there seems to be agreement, I opened a ticket (CASSANDRA-17837) and
a pull request (https://github.com/apache/cassandra/pull/1799) in so that
the final text can be hashed out and accepted.

I also used the proposed pull request in the text of the pull so that it
can be seen in all its glory 

On Thu, Aug 18, 2022 at 9:10 PM Josh McKenzie  wrote:

> I have never seen this
> kind of git merging strategy elsewhere, I am not sure if I am not
> experienced enough or we are truly unique the way we do things.
>
> I am very fond of this project and this community. THAT SAID ;) you could
> replace "kind of git merging strategy" with a lot of different things and
> have it equally apply on this project.
>
> Perils of being a mature long-lived project I suspect. I'm all for us
> doing the hard work of introspecting on how we do things and changing them
> to improve or match industry standards where applicable.
>
> On Thu, Aug 18, 2022, at 3:33 PM, Stefan Miklosovic wrote:
>
> Interesting, thanks for explicitly writing that down. I humbly think
> the CI and the convenience of the GitHub workflow is ultimately
> secondary when it comes to the code-base as such. Indeed, nice to
> have, but if it turns out to be uncomfortable in other ways, I guess
> we just have to live with what we have. TBH I have never seen this
> kind of git merging strategy elsewhere, I am not sure if I am not
> experienced enough or we are truly unique the way we do things.
> However, it does make sense.
>
> On Thu, 18 Aug 2022 at 21:28, Benedict 
> wrote:
> >
> > The benefits being extolled involve people setting up GitHub bots to
> integrate with PRs to run CI etc, which will require some non-trivial
> investment by somebody to put together
> >
> > The alternative merge strategy being discussed is not to merge, but to
> instead cherry-pick or rebase. This means we can produce separate PRs for
> each branch, that can be merged independently via the GitHub API. The
> downside of this is that there are no merge commits, while one upside of
> this is that there are no merge commits.
> >
> > On 18 Aug 2022, at 20:20, Stefan Miklosovic <
> stefan.mikloso...@instaclustr.com> wrote:
> >
> > No chicken-egg to me. All it takes is ctrl+c & ctrl+v on your merging
> > commits. How would new merging strategy actually look like? I am all
> > ears. This seems to be quite nice as is if we stick to be more verbose
> > what we did.
> >
> > On Thu, 18 Aug 2022 at 20:27, Benedict  wrote:
> >
> >
> > Was it?
> >
> >
> > I mean, we’ve all (or most) I think worked on projects with those
> things, so we all know what the benefits are?
> >
> >
> > It’s fair to point out that we don’t have it even running for any branch
> yet. However there’s perhaps a chicken-and-egg situation, where I’m unsure
> the investment to develop can be justified by those who are able, if
> there’s a chance it will be discarded? I can’t see us maintaining a
> bifurcated process, where some patches go through automation and others
> don’t, so if we don’t change the merge strategy that work would presumably
> end up wasted.
> >
> >
> > On 18 Aug 2022, at 18:53, Mick Semb Wever  wrote:
> >
> >
> > 
> >
> >
> > That debatable benefit aside, not doing merge commits would also open up
> options for us to use PR's for merges and integrate running CI, and
> blocking on clean CI, pre-merge. Which has some other pretty big benefits.
> :)
> >
> >
> >
> >
> > The past agreement IIRC was to start doing those things on trunk-only so
> we can evaluate them for real.
>
>
>

Re: [Proposal] add pull request template

[Claude Warren, Jr via dev]

I am all for simplification.  How about

- start of text 

Issue resolved:  CASSANDRA-



 - [ ] Jira ticket contains a description of: what is fixed, why it is
needed, and what branches to apply it to.

 - [ ] Commits have been squashed to remove intermediate development
commit messages.

 - [ ] Key commit messages start with the issue number (CASSANDRA-)


either - [ ] this is a trivial documentation change. (e.g. fixes a typo)

or:
 - [ ] Tests are included.
 - [ ] Documentation changes and/or updates are included.


By submitting this pull request, I acknowledge that I am making a
contribution to the Apache Software Foundation under the terms and
conditions of the [Contributor's
Agreement](https://www.apache.org/licenses/contributor-agreements.html).


See the [Apache Cassandra "Contributing to Cassandra"
guide](https://cassandra.apache.org/_/development/index.html) and/or
the [Apache Cassandra "Working on Documentation"
guide](https://cassandra.apache.org/_/development/documentation.html)



 end of text 

On Tue, Aug 16, 2022 at 8:42 AM Erick Ramirez 
wrote:

> +1 this is a great idea. But personally, I'm not too fussed about the
> level of detail that is in the template -- what is important is that
> contributors are reminded that there needs to be a ticket associated with
> contributions. Without being too prescriptive, aspiring contributors should
> really familiarise themselves with how to contribute[1] so they would know
> to search existing tickets first to avoid things like duplication.
>
>  Additionally, I personally prefer details about a contribution to be
> documented in a ticket rather than a PR because information stored in
> tickets are more persistent. Having said that, it doesn't hurt to have the
> details included in the PR as long as it is in the ticket too. Cheers!
>
>>

Re: [Proposal] add pull request template

[Claude Warren, Jr via dev]

If there is consensus that the PR template is a good idea, I'll create a
branch and we can wrangle the words.

I think the pull request generally is against a specific branch but having
the branch listed in the template is not a bad idea.  I do want to avoid
having too many questions.  We could change the text "Pull request
Description:" to be something more descriptive like "Describe what the pull
request fixes, why it is needed, and what branch(s) you expect it to be
applied to."

Or perhaps the branch question should be separate. Opinions?



On Mon, Aug 15, 2022 at 10:00 AM Stefan Miklosovic <
stefan.mikloso...@instaclustr.com> wrote:

> I like auto linking, definitely a handy feature.
>
> I am not sure about the content of the pull request description. I
> would include what that PR is actually for / why it is necessary to
> merge it and into what branches a contributor expects that PR to be
> merged in. However, this might be omitted if all this information is
> in a JIRA ticket already, I find the correct auto linking to be the
> most crucial here.
>
> There might be a bullet point for adding relevant CI builds (Jenkins or
> Circle).
>
> I am not sure we are going to enforce a commit message to start with
> the issue number. The issue number is already mentioned in the commit
> message. I feel like this kind of stuff is not crucial for a PR to be
> opened, a committer who is actually going to merge it will take extra
> time and care when it comes to these formalities anyway. The reason
> why a PR should be merged should be the priority.
>
> On Mon, 15 Aug 2022 at 10:41, Claude Warren, Jr via dev
>  wrote:
> >
> > Github provides the ability to add a pull request template [1].  I think
> that such a template could assist in making the pull requests better.
> Something like the text below, along with verifying that CASSANDRA-### will
> link to Jira [2], should provide the information needed and remind
> submitters of what is desired.
> >
> > If there is agreement here, I'll open a pull request to add the
> documentation and ask Apache devops to verify that the CASSANDRA-### will
> link to Jira.
> >
> > Claude
> >
> > [1]
> https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/creating-a-pull-request-template-for-your-repository
> for more information.
> >  [2]
> https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/managing-repository-settings/configuring-autolinks-to-reference-external-resources
> >
> > - start of text
> >
> > Issue resolved #  CASSANDRA-
> >
> > Pull request Description:
> >
> >
> >
> > 
> >
> > - [ ] Commits have been squashed to remove intermediate development
> commit messages.
> >  - [ ] Key commit messages start with the issue number (CASSANDRA-)
> >
> >
> > either
> >  - [ ] this is a trivial documentation change. (e.g. fixes a typo)
> >
> > or:
> >  - [ ] Tests are included.
> >  - [ ] Documentation changes and/or updates are included.
> >
> >
> > By submitting this pull request, I acknowledge that I am making a
> contribution to the Apache Software Foundation under the terms and
> conditions of the [Contributor's Agreement](
> https://www.apache.org/licenses/contributor-agreements.html).
> >
> > 
> >
> > See the [Apache Cassandra "Contributing to Cassandra" guide](
> https://cassandra.apache.org/_/development/index.html) and/or the [Apache
> Cassandra "Working on Documentation" guide](
> https://cassandra.apache.org/_/development/documentation.html)
>

[Proposal] add pull request template

[Claude Warren, Jr via dev]

Github provides the ability to add a pull request template [1].  I think
that such a template could assist in making the pull requests better.
Something like the text below, along with verifying that CASSANDRA-### will
link to Jira [2], should provide the information needed and remind
submitters of what is desired.

If there is agreement here, I'll open a pull request to add the
documentation and ask Apache devops to verify that the CASSANDRA-### will
link to Jira.

Claude

[1]
https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/creating-a-pull-request-template-for-your-repository
for more information.
 [2]
https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/managing-repository-settings/configuring-autolinks-to-reference-external-resources

- start of text

Issue resolved #  CASSANDRA-

Pull request Description:





- [ ] Commits have been squashed to remove intermediate development
commit messages.
 - [ ] Key commit messages start with the issue number (CASSANDRA-)


either - [ ] this is a trivial documentation change. (e.g. fixes a typo)

or:
 - [ ] Tests are included.
 - [ ] Documentation changes and/or updates are included.


By submitting this pull request, I acknowledge that I am making a
contribution to the Apache Software Foundation under the terms and
conditions of the [Contributor's
Agreement](https://www.apache.org/licenses/contributor-agreements.html).



See the [Apache Cassandra "Contributing to Cassandra"
guide](https://cassandra.apache.org/_/development/index.html) and/or
the [Apache Cassandra "Working on Documentation"
guide](https://cassandra.apache.org/_/development/documentation.html)

Re: [DISCUSS] Remove Dead Pull Requests

[Claude Warren, Jr via dev]

I agree the amount of work is somewhat overwhelming for the proposed
change, but I was referring to the lack of a Jira ticket blocking the pull
request.  At least that is how it looks to the new observer.  Perhaps we
should add a "trivial change" label for requests that do not have a ticket
and are trivial.

How many branches do the changes currently need to be applied to?  I assume
this goes up by 1 after the next release.

On Thu, Aug 11, 2022 at 9:36 AM Benjamin Lerer  wrote:

> Is there an objection to accepting "typo" corrections without a ticket?
>>
>
> One problem to be aware of is that those pull requests need to be
> converted in patches and merged manually up to trunk if they were done on
> older branches. So it might not look like it at first but it can be quite
> time consuming.
>
> Le jeu. 11 août 2022 à 10:07, Benedict  a écrit :
>
>> Those all seem like good suggestions to me
>>
>> On 11 Aug 2022, at 08:44, Claude Warren, Jr via dev <
>> dev@cassandra.apache.org> wrote:
>>
>> 
>> My original goal was to reduce the number of pull requests in the backlog
>> as it appears, from the outside, that the project does not really care for
>> outside contributions when there are over 200 pull requests pending and
>> many of them multiple years old.  I guess that is an optics issue.  Upon
>> looking at the older backlog, there were a few that I felt could be closed
>> because they didn't have tickets, or were trivial (i.e. typo correction),
>> or for which the original repository no longer exists.  However, from the
>> conversation here, it seems like the older pull requests are being used as
>> a long term storage for ideas that have not come to fruition and for which
>> the original developer may no longer be active.
>>
>> Looking through the pull request backlog there are a number of requests
>> that are not associated with a ticket.  Perhaps we should add pull
>> request template to github to request the associated ticket number when the
>> pull request is made.  The template can also request any other information
>> we this appropriate to speeding acceptance of the request.  I would add a
>> "This is a trivial change" checkbox for things like typo changes.  Is
>> there any documentation on the pull request process?  I think I saw
>> something that said patches were requested, but I can't find it now.  We
>> could add a link to any such documentation in the template as well.
>>
>> Is there an objection to accepting "typo" corrections without a ticket?
>>
>>
>>
>> Claude
>>
>> On Wed, Aug 10, 2022 at 5:08 PM Josh McKenzie 
>> wrote:
>>
>>> I think of this from a discoverability and workflow perspective at least
>>> on the JIRA side, though many of the same traits apply to PR's. Some
>>> questions that come to mind:
>>>
>>> 1. Are people grepping through the backlog of open items for things to
>>> work on they'd otherwise miss if they were closed out?
>>> 2. Are people grepping via specific text phrases in the summary, with or
>>> without "resolution = unresolved",  to try and find things on a particular
>>> topic to work on?
>>> 3. Relying on labels? Components? Something else?
>>>
>>> My .02: folks that are new to the project probably need more guidance on
>>> what to look for to get engaged with which is served by the LHF +
>>> unresolved + status emails + @cassandra_mentors. Mid to long-timers are
>>> probably more likely to search for specific topics, but may search for open
>>> tickets with patches attached or Patch Available things (seems unlikely as
>>> most of us have areas we're focused on but is possible?)
>>>
>>> The status quo today (leave things open if work has been done on it
>>> and/or it's an idea that clearly still has some relevance) seems to satisfy
>>> the most use-cases and retain the most flexibility, so I'd advocate for us
>>> not making a change just to make a change. While we could add a tag or
>>> resolution that indicates something closed out due to it being stale, my
>>> intuition is that people will just tack on "resolution = unresolved OR
>>> labels = closed_stale" in the JIRA case or sift through all things not
>>> merged in the PR case to effectively end up with the same body of results
>>> they're getting today.
>>>
>>> Given the ability of JQL to sort and slice based on updated times as
>>> well, it's relatively trivial to exclude stale tickets in your queries if
>>> you'

Re: [DISCUSS] Remove Dead Pull Requests

[Claude Warren, Jr via dev]

osed but are still
> relevant, but I think it looks bad for the project to have a large amount
> of stale unacted PRs.
>
> Em qua., 10 de ago. de 2022 às 11:08, C. Scott Andreas <
> sc...@paradoxica.net> escreveu:
>
> Claude, can you say more about the goal or purpose that closing tickets
> advances?
>
> There are quite a lot of tickets with patches attached that the project
> has either not been able to act on at the time; or which the original
> contributor started but was unable to complete. We’ve picked up many of
> these after a couple years and carried them to completion. Byte-comparable
> types come to mind. There are many, many more.
>
> Closing these tickets would be a very terminal action. If the goal is to
> distinguish what’s active from tickets that have gone quiet, adding a
> “dormant” label might work.
>
> - Scott
>
> On Aug 10, 2022, at 1:00 AM, Claude Warren, Jr via dev <
> dev@cassandra.apache.org> wrote:
>
> 
> At the moment we have 222 open pull requests.  Some dating back 4 years.
> For some the repository from which they were pulled from has been deleted.
> For many there are branch conflicts.
>
> Now, I am new here so please excuse any misstatements and attribute to
> ignorance not malice any offence.
>
> I would like to propose the  following:
>
>
>1. We accept simple typo corrections without a ticket.
>2. Add a "Propose Close" label
>3. We "Propose Close" any pull request for which the originating
>repository has been deleted.
>4. We "Propose Close" any ticket, other than simple typo corrections,
>that has been labeled missing-ticket for more than 30 days.
>5. We Close any pull request that has been in the "Propose Close"
>state for more than 30 days.
>
> I don't have access to make any of these changes.  If granted access I
> would be willing to manage the process.
>
> Claude
>
>
>

Re: Cassandra project status update 2022-08-03

[Claude Warren, Jr via dev]

Perhaps flaky tests need to be handled differently.  Is there a way to
build a statistical model of the current flakiness of the test that we can
then use during testing to accept the failures?  So if an acceptable level
of flakiness is developed then if the test fails, it needs to be run again
or multiple times to get a sample and ensure that the failure is not
statistically significant.



On Wed, Aug 10, 2022 at 8:51 AM Benedict Elliott Smith 
wrote:

> 
> > We can start by putting the bar at a lower level and raise the level
> over time
>
> +1
>
> > One simple approach that has been mentioned several time is to run the
> new tests added by a given patch in a loop using one of the CircleCI tasks
>
> I think if we want to do this, it should be extremely easy - by which I
> mean automatic, really. This shouldn’t be too tricky I think? We just need
> to produce a diff of new test classes and methods within existing classes.
> If there doesn’t already exist tooling to do this, I can probably help out
> by putting together something to output @Test annotated methods within a
> source tree, if others are able to turn this into a part of the CircleCI
> pre-commit task (i.e. to pick the common ancestor with trunk, 4.1 etc, and
> run this task for each of the outputs). We might want to start
> standardising branch naming structures to support picking the upstream
> branch.
>
> > We should also probably revert newly committed patch if we detect that
> they introduced flakies.
>
> There should be a strict time limit for reverting a patch for this reason,
> as environments change and what is flaky now was not necessarily before.
>
> On 9 Aug 2022, at 12:57, Benjamin Lerer  wrote:
>
> At this point it is clear that we will probably never be able to remove
> some level of flakiness from our tests. For me the questions are: 1) Where
> do we draw the line for a release ? and 2) How do we maintain that line
> over time?
>
> In my opinion, not all flakies are equals. Some fails every 10 runs, some
> fails 1 in a 1000 runs. I would personally draw the line based on that
> metric. With the circleci tasks that Andres has added we can easily get
> that information for a given test.
> We can start by putting the bar at a lower level and raise the level over
> time when most of the flakies that we hit are above that level.
>
> At the same time we should make sure that we do not introduce new flakies.
> One simple approach that has been mentioned several time is to run the new
> tests added by a given patch in a loop using one of the CircleCI tasks.
> That would allow us to minimize the risk of introducing flaky tests. We
> should also probably revert newly committed patch if we detect that they
> introduced flakies.
>
> What do you think?
>
>
>
>
>
> Le dim. 7 août 2022 à 12:24, Mick Semb Wever  a écrit :
>
>>
>>
>> With that said, I guess we can just revise on a regular basis what
>>> exactly are the last flakes and not numbers which also change quickly up
>>> and down with the first change in the Infra.
>>>
>>
>>
>> +1, I am in favour of taking a pragmatic approach.
>>
>> If flakies are identified and triaged enough that, with correlation from
>> both CI systems, we are confident that no legit bugs are behind them, I'm
>> in favour of going beta.
>>
>> I still remain in favour of somehow incentivising reducing other flakies
>> as well. Flakies that expose poor/limited CI infra, and/or tests that are
>> not as resilient as they could be, are still noise that indirectly reduce
>> our QA (and increase efforts to find and tackle those legit runtime
>> problems). Interested in hearing input from others here that have been
>> spending a lot of time on this front.
>>
>> Could it work if we say: all flakies must be ticketed, and test/infra
>> related flakies do not block a beta release so long as there are fewer than
>> the previous release? The intent here being pragmatic, but keeping us on a
>> "keep the campground cleaner" trajectory…
>>
>>
>

[DISCUSS] Remove Dead Pull Requests

[Claude Warren, Jr via dev]

At the moment we have 222 open pull requests.  Some dating back 4 years.
For some the repository from which they were pulled from has been deleted.
For many there are branch conflicts.

Now, I am new here so please excuse any misstatements and attribute to
ignorance not malice any offence.

I would like to propose the  following:


   1. We accept simple typo corrections without a ticket.
   2. Add a "Propose Close" label
   3. We "Propose Close" any pull request for which the originating
   repository has been deleted.
   4. We "Propose Close" any ticket, other than simple typo corrections,
   that has been labeled missing-ticket for more than 30 days.
   5. We Close any pull request that has been in the "Propose Close" state
   for more than 30 days.

I don't have access to make any of these changes.  If granted access I
would be willing to manage the process.

Claude

Re: key apt by apache cassandra seems deprecated and cqlsh is broken

[Claude Warren, Jr via dev]

Could this be related to the deprecation of apt-key on your system?  You
don't specify what version of which distribution you are using.  However,
there is a good example of how to solve the issue at
https://www.linuxuprising.com/2021/01/apt-key-is-deprecated-how-to-add.html



On Tue, Aug 9, 2022 at 11:51 AM Dorian ROSSE  wrote:

> hello,
>
>
> i am getting this error when i try to update my system :
>
> '''W: http://www.apache.org/dist/cassandra/debian/dists/40x/InRelease:
> Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the
> DEPRECATION section in apt-key(8) for details.
> '''
>
> (for this error on my ubuntu system i have reach the ubuntu launchpad
> suport if that hold by their side)
>
> the apache cassandra is installed but the subprogram cqlsh is broken thus
> i have tried to install the module python asked but it seems unexisted :
>
> '''~$ sudo cqlsh
> Traceback (most recent call last):
>   File "/usr/bin/cqlsh.py", line 20, in 
> import cqlshlib
> ModuleNotFoundError: No module named 'cqlshlib'
> ~$ sudo pip3 install cqlshlib
> ERROR: Could not find a version that satisfies the requirement cqlshlib
> (from versions: none)
> ERROR: No matching distribution found for cqlshlib
> ~$ sudo python3 install cqlshlib
> python3: can't open file '/home/dorianrosse/install': [Errno 2] No such
> file or directory
> ~$ sudo pip install cqlshlib
> ERROR: Could not find a version that satisfies the requirement cqlshlib
> (from versions: none)
> ERROR: No matching distribution found for cqlshlib
> ~$ sudo python install cqlshlib
> python: can't open file 'install': [Errno 2] No such file or directory
> '''
>
> thanks you in advance to help myself repair both errors,
>
> regards.
>
>
> Dorian ROSSE.
>