Re: [Discuss] CQLSH should left-align numbers, right-align text (CASSANDRA-19150)

[Derek Chen-Becker]

Actually, now that I'm looking at the original email on my browser and not
my phone (and can see the formatting properly), I think we have the
nomenclature backward here. Left-alignment in the printing world means that
text in each cell starts at the left-most column for the cell, but in your
examples you're calling that right-aligned (and vice-versa). Along the
lines of what Stefan said, I think this probably came about more as a
"we'll just keep things simple and use the same alignment everywhere"
rather than an intentional right-alignment of text for a specific purpose.
I would actually be fine with left-aligning text to fit what appears to be
standard practice in other systems.

Cheers,

Derek

On Tue, Jan 9, 2024 at 7:34 AM Brad  wrote:

> CQLSH currently left-aligns all output, affecting both numbers and text.
> While this works well for numbers, a better approach adopted by many is to
> left align numbers and right align text.
>
> For example, both Excel and Postgres shell use the later:
>
> psql
>
> # select * from employee;
>
>  empid |  name   |dept
>
> ---+-+
>
>  1 | Clark   | Sales
>
>200 | Dave| Accounting
>
> 33 | Johnson | Sales
>
>
> while CQLSH simply left aligns all the columns
>
> cqlsh> select * from employee;
>
>  empid | dept   | name
>
> ---++-
>
> 33 |  Sales | Johnson
>
>  1 |  Sales |   Clark
>
>200 | Accounting |Dave
>
>
>
> Left aligned text looks much worse on text values which share common
> prefixes
>
>
> cqlsh> select * from system_views.system_properties limit 7 ;
>
>
>  name   | value
>
>
> +
>
>   JAVA_HOME |
>   /Users/brad/.jenv/versions/17
>
>cassandra.jmx.local.port |
>   7199
>
>cassandra.logdir |
> /usr/local/cassandra-5.0-beta1/bin/../logs
>
>cassandra.storagedir |
> /usr/local/cassandra-5.0-beta1/bin/../data
>
>   com.sun.management.jmxremote.authenticate |
> false
>
>  com.sun.management.jmxremote.password.file |
>   /etc/cassandra/jmxremote.password
>
> io.netty.transport.estimateSizeOnSubmit |
> false
>
>
>
> The Jira CASSANDRA-19150
> <https://issues.apache.org/jira/browse/CASSANDRA-19150> discusses this in
> further detail with some additional examples.
>
>
> I wanted to raise the issue here to propose changing CQLSH to right-align
> text while continue to left-align numbers.
>
>
> Regards,
>
>
> Brad Schoening
>
>
> ReplyForward
> Add reaction
>


-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [Discuss] CQLSH should left-align numbers, right-align text (CASSANDRA-19150)

[Derek Chen-Becker]

In the ticket itself there's an example of left aligned being better for
prefix strings (e.g. fully qualified class names), and I suspect this is
similarly useful for other things like file paths, etc. I would also agree
with Stefan that it would be nice to know why the current convention was
chosen in the first place.

Cheers,

Derek

On Tue, Jan 9, 2024 at 8:18 AM Brad  wrote:

> Derek,
>
> I'm proposing a switch or blanket change to a convention of right aligned
> text and left aligned numbers in CQLSH.
>
> I took a look at two other examples, Excel and Postgres shell and that's
> how they work when displaying tabular data.  The Jira was originally to
> make right or left alignment an option, but making it configurable seems
> less useful than choosing a better standard.
>
> On Tue, Jan 9, 2024 at 9:58 AM Derek Chen-Becker 
> wrote:
>
>> Just to clarify, per the ticket you're proposing a configuration option
>> to control this on a per-column basis, correct? Your email makes it sound
>> like a blanket change.
>>
>> Cheers,
>>
>> Derek
>>
>> On Tue, Jan 9, 2024 at 7:34 AM Brad  wrote:
>>
>>> CQLSH currently left-aligns all output, affecting both numbers and
>>> text.  While this works well for numbers, a better approach adopted by many
>>> is to left align numbers and right align text.
>>>
>>> For example, both Excel and Postgres shell use the later:
>>>
>>> psql
>>>
>>> # select * from employee;
>>>
>>>  empid |  name   |dept
>>>
>>> ---+-+
>>>
>>>  1 | Clark   | Sales
>>>
>>>200 | Dave| Accounting
>>>
>>> 33 | Johnson | Sales
>>>
>>>
>>> while CQLSH simply left aligns all the columns
>>>
>>> cqlsh> select * from employee;
>>>
>>>  empid | dept   | name
>>>
>>> ---++-
>>>
>>> 33 |  Sales | Johnson
>>>
>>>  1 |  Sales |   Clark
>>>
>>>200 | Accounting |Dave
>>>
>>>
>>>
>>> Left aligned text looks much worse on text values which share common
>>> prefixes
>>>
>>>
>>> cqlsh> select * from system_views.system_properties limit 7 ;
>>>
>>>
>>>  name   | value
>>>
>>>
>>> +
>>>
>>>   JAVA_HOME |
>>>   /Users/brad/.jenv/versions/17
>>>
>>>cassandra.jmx.local.port |
>>> 7199
>>>
>>>cassandra.logdir |
>>> /usr/local/cassandra-5.0-beta1/bin/../logs
>>>
>>>cassandra.storagedir |
>>> /usr/local/cassandra-5.0-beta1/bin/../data
>>>
>>>   com.sun.management.jmxremote.authenticate |
>>>   false
>>>
>>>  com.sun.management.jmxremote.password.file |
>>>   /etc/cassandra/jmxremote.password
>>>
>>> io.netty.transport.estimateSizeOnSubmit |
>>>   false
>>>
>>>
>>>
>>> The Jira CASSANDRA-19150
>>> <https://issues.apache.org/jira/browse/CASSANDRA-19150> discusses this
>>> in further detail with some additional examples.
>>>
>>>
>>> I wanted to raise the issue here to propose changing CQLSH to
>>> right-align text while continue to left-align numbers.
>>>
>>>
>>> Regards,
>>>
>>>
>>> Brad Schoening
>>>
>>>
>>> ReplyForward
>>> Add reaction
>>>
>>
>>
>> --
>> +---+
>> | Derek Chen-Becker |
>> | GPG Key available at https://keybase.io/dchenbecker and   |
>> | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
>> | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
>> +---+
>>
>>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [Discuss] CQLSH should left-align numbers, right-align text (CASSANDRA-19150)

[Derek Chen-Becker]

Just to clarify, per the ticket you're proposing a configuration option to
control this on a per-column basis, correct? Your email makes it sound like
a blanket change.

Cheers,

Derek

On Tue, Jan 9, 2024 at 7:34 AM Brad  wrote:

> CQLSH currently left-aligns all output, affecting both numbers and text.
> While this works well for numbers, a better approach adopted by many is to
> left align numbers and right align text.
>
> For example, both Excel and Postgres shell use the later:
>
> psql
>
> # select * from employee;
>
>  empid |  name   |dept
>
> ---+-+
>
>  1 | Clark   | Sales
>
>200 | Dave| Accounting
>
> 33 | Johnson | Sales
>
>
> while CQLSH simply left aligns all the columns
>
> cqlsh> select * from employee;
>
>  empid | dept   | name
>
> ---++-
>
> 33 |  Sales | Johnson
>
>  1 |  Sales |   Clark
>
>200 | Accounting |Dave
>
>
>
> Left aligned text looks much worse on text values which share common
> prefixes
>
>
> cqlsh> select * from system_views.system_properties limit 7 ;
>
>
>  name   | value
>
>
> +
>
>   JAVA_HOME |
>   /Users/brad/.jenv/versions/17
>
>cassandra.jmx.local.port |
>   7199
>
>cassandra.logdir |
> /usr/local/cassandra-5.0-beta1/bin/../logs
>
>cassandra.storagedir |
> /usr/local/cassandra-5.0-beta1/bin/../data
>
>   com.sun.management.jmxremote.authenticate |
> false
>
>  com.sun.management.jmxremote.password.file |
>   /etc/cassandra/jmxremote.password
>
> io.netty.transport.estimateSizeOnSubmit |
> false
>
>
>
> The Jira CASSANDRA-19150
> <https://issues.apache.org/jira/browse/CASSANDRA-19150> discusses this in
> further detail with some additional examples.
>
>
> I wanted to raise the issue here to propose changing CQLSH to right-align
> text while continue to left-align numbers.
>
>
> Regards,
>
>
> Brad Schoening
>
>
> ReplyForward
> Add reaction
>


-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Moving Semver4j from test to main dependencies

[Derek Chen-Becker]

+1

Semver4j seems reasonable to me. I looked through the code and it's
relatively easy to understand. I'm not sure how easy it would be to replace
CassandraVersion, but that's not an immediate concern I guess.

Cheers,

Derek

On Fri, Dec 15, 2023 at 2:56 AM Jacek Lewandowski <
lewandowski.ja...@gmail.com> wrote:

> Hi,
>
> I'd like to add Semver4j to the production dependencies. It is currently
> on the test classpath. The library is pretty lightweight, licensed with MIT
> and has no transitive dependencies.
>
> We need to represent the kernel version somehow in CASSANDRA-19196 and
> Semver4j looks as the right tool for it. Maybe at some point we can replace
> our custom implementation of CassandraVersion as well.
>
> Thanks,
> - - -- --- -  -
> Jacek Lewandowski
>


-- 
+-----------+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [Discuss] cleaning up build temp files

[Derek Chen-Becker]

Hi,

Would someone be able to review the trunk patch for
https://issues.apache.org/jira/browse/CASSANDRA-18750? I'm fighting
CircleCI to try and get anything to pass (everything fails with
"Concurrency limit"), but builds and tests work locally (via "ant test" and
the docker "run-tests.sh").

Cheers,

Derek

On Tue, Aug 15, 2023 at 8:13 AM Derek Chen-Becker 
wrote:

> Thanks, I thought I had updated against trunk but apparently missed it.
> It's working now.
>
> On Tue, Aug 15, 2023 at 7:27 AM Brandon Williams  wrote:
>
>> On Tue, Aug 15, 2023 at 8:13 AM Derek Chen-Becker 
>> wrote:
>> >
>> > Also, I noticed that the instructions on
>> https://cassandra.apache.org/_/development/how_to_commit.html mention
>> "ant jar check" as a recommended step, but there's not "check" target. Does
>> it mean "checkstyle"?
>>
>> It exists in the cassandra-5.0 and trunk branches after CASSANDRA-18618
>>
>
>
> --
> +---+
> | Derek Chen-Becker |
> | GPG Key available at https://keybase.io/dchenbecker and   |
> | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
> | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
> +---+
>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [Discuss] cleaning up build temp files

[Derek Chen-Becker]

Thanks, I thought I had updated against trunk but apparently missed it.
It's working now.

On Tue, Aug 15, 2023 at 7:27 AM Brandon Williams  wrote:

> On Tue, Aug 15, 2023 at 8:13 AM Derek Chen-Becker 
> wrote:
> >
> > Also, I noticed that the instructions on
> https://cassandra.apache.org/_/development/how_to_commit.html mention
> "ant jar check" as a recommended step, but there's not "check" target. Does
> it mean "checkstyle"?
>
> It exists in the cassandra-5.0 and trunk branches after CASSANDRA-18618
>


-- 
+-------+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [Discuss] cleaning up build temp files

[Derek Chen-Becker]

Thanks Brandon, that did make things simple. PR against trunk is here for
review, and once we get it ship-shape I can create patches against the
other version branches:

https://github.com/apache/cassandra/pull/2593

Also, I noticed that the instructions on
https://cassandra.apache.org/_/development/how_to_commit.html mention "ant
jar check" as a recommended step, but there's not "check" target. Does it
mean "checkstyle"?

Thanks,

Derek

On Tue, Aug 15, 2023 at 4:34 AM Brandon Williams  wrote:

> I think most of those use InMemoryAuditLogger so it's not a problem,
> but we can call DatabaseDescriptor.setAuditLoggingOptions to change
> the dir, the same way the AuditLoggerTest.testJMXArchive method does.
>
> Kind Regards,
> Brandon
>
> On Mon, Aug 14, 2023 at 11:23 PM Derek Chen-Becker
>  wrote:
> >
> > I think I'm 98% done with the needed changes (or at least a first pass;
> I'm sure I'll get feedback), but I've hit an issue with AuditLoggerTest.
> While AuditLogOptions has a String property for "audit_logs_dir", the test
> calls StorageService.enableAuditLogs throught the test methods, and that
> method does not take a parameter for the log directory. As a result, the
> AuditLogOptions instance that is created always uses the default audit log
> directory, which just happens to be the "audit" subdirectory of the project
> root. This is getting into territory I'm not that familiar with, but it
> seems like there might be three ways to fix this:
> >
> > 1. Write a new overload of StorageService.enableAuditLog that takes an
> AuditLogOptions and delegate the other overloads to it
> > 2. Add a new overload that also takes a Path or File pointing to the
> audit log directory
> > 3. Set the "cassandra.logdir.audit" property in the ant build, similar
> to "tmp.dir"/"java.io.tmpdir"
> >
> > Thoughts?
> >
> > Derek
> >
> > On Sun, Aug 13, 2023 at 3:12 PM Josh McKenzie 
> wrote:
> >>
> >> There's also tests that hardcode
> >>
> >> I started mentally twitching when I hit that point in the sentence.
> >>
> >> Kill them with fire.
> >>
> >> On Sun, Aug 13, 2023, at 4:51 PM, Mick Semb Wever wrote:
> >>
> >>
> >>
> https://github.com/apache/cassandra/blob/trunk/test/unit/org/apache/cassandra/db/DirectoriesTest.java#L717-L719
> >>
> https://github.com/apache/cassandra/blob/trunk/test/unit/org/apache/cassandra/db/DirectoriesTest.java#L757-L759
> >>
> >> Can I open a ticket to track fixes for these and any other issues I run
> into while moving to using "build/tmp"?
> >>
> >>
> >>
> >> Go for it. :-)
> >> There's also tests that hardcode other paths that breaks the use of
> `build.dir`
> >>
> >>
> >
> >
> > --
> > +-------+
> > | Derek Chen-Becker |
> > | GPG Key available at https://keybase.io/dchenbecker and   |
> > | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
> > | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
> > +---+
> >
>


-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [Discuss] cleaning up build temp files

[Derek Chen-Becker]

I think I'm 98% done with the needed changes (or at least a first pass; I'm
sure I'll get feedback), but I've hit an issue with AuditLoggerTest. While
AuditLogOptions has a String property for "audit_logs_dir", the test calls
StorageService.enableAuditLogs throught the test methods, and that method
does not take a parameter for the log directory. As a result, the
AuditLogOptions instance that is created always uses the default audit log
directory, which just happens to be the "audit" subdirectory of the project
root. This is getting into territory I'm not that familiar with, but it
seems like there might be three ways to fix this:

1. Write a new overload of StorageService.enableAuditLog that takes an
AuditLogOptions and delegate the other overloads to it
2. Add a new overload that also takes a Path or File pointing to the audit
log directory
3. Set the "cassandra.logdir.audit" property in the ant build, similar to
"tmp.dir"/"java.io.tmpdir"

Thoughts?

Derek

On Sun, Aug 13, 2023 at 3:12 PM Josh McKenzie  wrote:

> There's also tests that hardcode
>
> I started mentally twitching when I hit that point in the sentence.
>
> *Kill them with fire.*
>
> On Sun, Aug 13, 2023, at 4:51 PM, Mick Semb Wever wrote:
>
>
>
> https://github.com/apache/cassandra/blob/trunk/test/unit/org/apache/cassandra/db/DirectoriesTest.java#L717-L719
>
> https://github.com/apache/cassandra/blob/trunk/test/unit/org/apache/cassandra/db/DirectoriesTest.java#L757-L759
>
> Can I open a ticket to track fixes for these and any other issues I run
> into while moving to using "build/tmp"?
>
>
>
> Go for it. :-)
> There's also tests that hardcode other paths that breaks the use of
> `build.dir`
>
>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [Discuss] cleaning up build temp files

[Derek Chen-Becker]

OK, I already found some places where "/tmp" is hard-coded:

https://github.com/apache/cassandra/blob/trunk/test/unit/org/apache/cassandra/db/DirectoriesTest.java#L717-L719
https://github.com/apache/cassandra/blob/trunk/test/unit/org/apache/cassandra/db/DirectoriesTest.java#L757-L759

Can I open a ticket to track fixes for these and any other issues I run
into while moving to using "build/tmp"?

Thanks,

Derek

On Sun, Aug 13, 2023 at 10:13 AM Josh McKenzie  wrote:

> I think we want/need relative paths, e.g. "build/tmp", and if the path is
> in a mounted volume there can be another container still running.
>
> Sure. The specifics of *what* path isn't interesting to me.
>
> The pattern of:
> 1. Let env declare where TEMP lives
> 2. Write things to TEMP
> 3. Delete things from TEMP every time we run a new suite or do "ant clean"
>
> Is.
>
> Could also take it a step further and let env declare RESULTS_PATH for
> things they want to be durable and add an "ant clean-results" target.
>
> On Sun, Aug 13, 2023, at 11:33 AM, Derek Chen-Becker wrote:
>
> Nevermind, I found "tmp.dir"
>
> On Sun, Aug 13, 2023 at 9:29 AM Derek Chen-Becker 
> wrote:
>
> Cool,
>
> I'm a little confused. Is "tmp.dir" a custom Java property that we expose?
> I thought that the standard "property was "java.io.tmpdir". Let me take a
> stab at setting tmp.dir to build/tmp and see if I run into any issues (or
> still see any files in /tmp).
>
> Cheers,
>
> Derek
>
> On Sun, Aug 13, 2023 at 8:24 AM Mick Semb Wever  wrote:
>
>
> While doing some local testing, I noticed that my /tmp drive completely
> filled with test artifact files (e.g. data directories, logs, commit logs,
> etc). Mick pointed out that we do attempt to do some "find" based cleanup
> in CI (
> https://github.com/apache/cassandra-builds/blob/trunk/jenkins-dsl/cassandra_job_dsl_seed.groovy#L437-439),
> but I was wondering if it might be better to do the following for direct
> ant builds:
>
> 1. If TMPDIR is set, use it. It does not appear to be honored, currently,
> so I need to do some analysis of what would need to be done here
> 2. If TMPDIR is not set, use "mktemp" to create a temp directory and set
> TMPDIR with that directory
> 3. Update the "ant clean" task to delete TMPDIR when we've generated it,
> or attempt the find-based cleanup if TMPDIR was provided
>
> Does anyone know if there are any hard-coded assumptions that test files
> will live directly under /tmp?
>
>
>
> This will need testing with in-tree scripts, ci-cassandra, and circleci  :(
>
> What comes to mind:
>  - TMPDIR works best today with the python and scripting stuff
>  - setting TMPDIR can break tests, hence unit test script set instead
> $TMP_DIR which is passed to `-Dtmp.dir=…`
>  - /tmp is often set up to be a more appropropriate fs (and volume size)
>  - it is hard to customise everything
>  - it needs to work locally on your machine as well as in docker
> containers, as well as CI
>
> If we want something that is wiped by `ant clean` I would suggest using
> the build/tmp directory by default.
> In-tree scripts do this for unit tests:
> https://github.com/apache/cassandra/blob/trunk/.build/run-tests.sh#L160
>  but are not yet doing it for the dtests:
> https://github.com/apache/cassandra/blob/trunk/.build/run-python-dtests.sh#L58
>
>
> So I don't think we need (3). If the caller has specified TMPDIR it is
> then their responsibility to clean it.
>
> We can probably avoid trying to set TMPDIR, instead defaulting the
> `tmp.dir` property to  the build/tmp directory.
>
> The goal of any changes in build.xml should be, in addition to providing
> the best dev exp, to simplify the testing and CI layers above it.
>
>
>
> --
> +---+
> | Derek Chen-Becker     |
> | GPG Key available at https://keybase.io/dchenbecker and   |
> | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
> | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
> +---+
>
>
>
> --
> +---+
> | Derek Chen-Becker     |
> | GPG Key available at https://keybase.io/dchenbecker and   |
> | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
> | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
> +---+
>
>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [Discuss] cleaning up build temp files

[Derek Chen-Becker]

Nevermind, I found "tmp.dir"

On Sun, Aug 13, 2023 at 9:29 AM Derek Chen-Becker 
wrote:

> Cool,
>
> I'm a little confused. Is "tmp.dir" a custom Java property that we expose?
> I thought that the standard "property was "java.io.tmpdir". Let me take a
> stab at setting tmp.dir to build/tmp and see if I run into any issues (or
> still see any files in /tmp).
>
> Cheers,
>
> Derek
>
> On Sun, Aug 13, 2023 at 8:24 AM Mick Semb Wever  wrote:
>
>>
>> While doing some local testing, I noticed that my /tmp drive completely
>>> filled with test artifact files (e.g. data directories, logs, commit logs,
>>> etc). Mick pointed out that we do attempt to do some "find" based cleanup
>>> in CI (
>>> https://github.com/apache/cassandra-builds/blob/trunk/jenkins-dsl/cassandra_job_dsl_seed.groovy#L437-439),
>>> but I was wondering if it might be better to do the following for direct
>>> ant builds:
>>>
>>> 1. If TMPDIR is set, use it. It does not appear to be honored,
>>> currently, so I need to do some analysis of what would need to be done here
>>> 2. If TMPDIR is not set, use "mktemp" to create a temp directory and set
>>> TMPDIR with that directory
>>> 3. Update the "ant clean" task to delete TMPDIR when we've generated it,
>>> or attempt the find-based cleanup if TMPDIR was provided
>>>
>>> Does anyone know if there are any hard-coded assumptions that test files
>>> will live directly under /tmp?
>>>
>>
>>
>> This will need testing with in-tree scripts, ci-cassandra, and circleci
>>  :(
>>
>> What comes to mind:
>>  - TMPDIR works best today with the python and scripting stuff
>>  - setting TMPDIR can break tests, hence unit test script set instead
>> $TMP_DIR which is passed to `-Dtmp.dir=…`
>>  - /tmp is often set up to be a more appropropriate fs (and volume size)
>>  - it is hard to customise everything
>>  - it needs to work locally on your machine as well as in docker
>> containers, as well as CI
>>
>> If we want something that is wiped by `ant clean` I would suggest using
>> the build/tmp directory by default.
>> In-tree scripts do this for unit tests:
>> https://github.com/apache/cassandra/blob/trunk/.build/run-tests.sh#L160
>>  but are not yet doing it for the dtests:
>> https://github.com/apache/cassandra/blob/trunk/.build/run-python-dtests.sh#L58
>>
>>
>> So I don't think we need (3). If the caller has specified TMPDIR it is
>> then their responsibility to clean it.
>>
>> We can probably avoid trying to set TMPDIR, instead defaulting the
>> `tmp.dir` property to  the build/tmp directory.
>>
>> The goal of any changes in build.xml should be, in addition to providing
>> the best dev exp, to simplify the testing and CI layers above it.
>>
>>
>
> --
> +-------+
> | Derek Chen-Becker |
> | GPG Key available at https://keybase.io/dchenbecker and   |
> | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
> | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
> +---+
>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [Discuss] cleaning up build temp files

[Derek Chen-Becker]

Cool,

I'm a little confused. Is "tmp.dir" a custom Java property that we expose?
I thought that the standard "property was "java.io.tmpdir". Let me take a
stab at setting tmp.dir to build/tmp and see if I run into any issues (or
still see any files in /tmp).

Cheers,

Derek

On Sun, Aug 13, 2023 at 8:24 AM Mick Semb Wever  wrote:

>
> While doing some local testing, I noticed that my /tmp drive completely
>> filled with test artifact files (e.g. data directories, logs, commit logs,
>> etc). Mick pointed out that we do attempt to do some "find" based cleanup
>> in CI (
>> https://github.com/apache/cassandra-builds/blob/trunk/jenkins-dsl/cassandra_job_dsl_seed.groovy#L437-439),
>> but I was wondering if it might be better to do the following for direct
>> ant builds:
>>
>> 1. If TMPDIR is set, use it. It does not appear to be honored, currently,
>> so I need to do some analysis of what would need to be done here
>> 2. If TMPDIR is not set, use "mktemp" to create a temp directory and set
>> TMPDIR with that directory
>> 3. Update the "ant clean" task to delete TMPDIR when we've generated it,
>> or attempt the find-based cleanup if TMPDIR was provided
>>
>> Does anyone know if there are any hard-coded assumptions that test files
>> will live directly under /tmp?
>>
>
>
> This will need testing with in-tree scripts, ci-cassandra, and circleci  :(
>
> What comes to mind:
>  - TMPDIR works best today with the python and scripting stuff
>  - setting TMPDIR can break tests, hence unit test script set instead
> $TMP_DIR which is passed to `-Dtmp.dir=…`
>  - /tmp is often set up to be a more appropropriate fs (and volume size)
>  - it is hard to customise everything
>  - it needs to work locally on your machine as well as in docker
> containers, as well as CI
>
> If we want something that is wiped by `ant clean` I would suggest using
> the build/tmp directory by default.
> In-tree scripts do this for unit tests:
> https://github.com/apache/cassandra/blob/trunk/.build/run-tests.sh#L160
>  but are not yet doing it for the dtests:
> https://github.com/apache/cassandra/blob/trunk/.build/run-python-dtests.sh#L58
>
>
> So I don't think we need (3). If the caller has specified TMPDIR it is
> then their responsibility to clean it.
>
> We can probably avoid trying to set TMPDIR, instead defaulting the
> `tmp.dir` property to  the build/tmp directory.
>
> The goal of any changes in build.xml should be, in addition to providing
> the best dev exp, to simplify the testing and CI layers above it.
>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

[Discuss] cleaning up build temp files

[Derek Chen-Becker]

Hi,

While doing some local testing, I noticed that my /tmp drive completely
filled with test artifact files (e.g. data directories, logs, commit logs,
etc). Mick pointed out that we do attempt to do some "find" based cleanup
in CI (
https://github.com/apache/cassandra-builds/blob/trunk/jenkins-dsl/cassandra_job_dsl_seed.groovy#L437-439),
but I was wondering if it might be better to do the following for direct
ant builds:

1. If TMPDIR is set, use it. It does not appear to be honored, currently,
so I need to do some analysis of what would need to be done here
2. If TMPDIR is not set, use "mktemp" to create a temp directory and set
TMPDIR with that directory
3. Update the "ant clean" task to delete TMPDIR when we've generated it, or
attempt the find-based cleanup if TMPDIR was provided

Does anyone know if there are any hard-coded assumptions that test files
will live directly under /tmp?

Thanks,

Derek

-- 
+-------+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSSION] Shall we remove ant javadoc task?

[Derek Chen-Becker]

Oh, whoops, I guess I'm the only one that thinks Javadoc is just the tool
and/or it's output (not the markup itself) :P If anything, the codebase
could use a little more package/class/method markup in some places, so I'm
definitely only in favor of getting rid of the ant task. I should amend my
statement to be "...I suspect most people are not *opening their browsers
and *looking at Javadoc..." :)

Cheers,

Derek



On Wed, Aug 2, 2023, 1:30 PM Josh McKenzie  wrote:

> most people are not looking at Javadoc when working on the codebase.
>
> I definitely use it extensively *inside the IDE*. But never as a compiled
> set of external docs.
>
> Which is to say, I'm +1 on removing the target and I'd ask everyone to
> keep javadoccing your classes and methods where things are non-obvious or
> there's a logical coupling with something else in the system. :)
>
> On Wed, Aug 2, 2023, at 2:08 PM, Derek Chen-Becker wrote:
>
> +1. If a need comes up for Javadoc we can fix it at that point, but I
> suspect most people are not looking at Javadoc when working on the codebase.
>
> Cheers,
>
> Derek
>
> On Wed, Aug 2, 2023 at 11:11 AM Brandon Williams  wrote:
>
> I don't think even if it works anyone is going to use the output, so
> I'm good with removal.
>
> Kind Regards,
> Brandon
>
> On Wed, Aug 2, 2023 at 11:50 AM Ekaterina Dimitrova
>  wrote:
> >
> > Hi everyone,
> > We were looking into a user report around our ant javadoc task recently.
> > That made us realize it is not run in CI; it finishes successfully even
> if there are hundreds of errors, some potentially breaking doc pages.
> >
> > There was a ticket discussion where a few community members mentioned
> that this task was probably unnecessary. Can we remove it, or shall we fix
> it?
> >
> > Best regards,
> > Ekaterina
>
>
>
> --
> +---+
> | Derek Chen-Becker |
> | GPG Key available at https://keybase.io/dchenbecker and   |
> | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
> | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
> +---+
>
>
>

Re: [DISCUSSION] Shall we remove ant javadoc task?

[Derek Chen-Becker]

+1. If a need comes up for Javadoc we can fix it at that point, but I
suspect most people are not looking at Javadoc when working on the codebase.

Cheers,

Derek

On Wed, Aug 2, 2023 at 11:11 AM Brandon Williams  wrote:

> I don't think even if it works anyone is going to use the output, so
> I'm good with removal.
>
> Kind Regards,
> Brandon
>
> On Wed, Aug 2, 2023 at 11:50 AM Ekaterina Dimitrova
>  wrote:
> >
> > Hi everyone,
> > We were looking into a user report around our ant javadoc task recently.
> > That made us realize it is not run in CI; it finishes successfully even
> if there are hundreds of errors, some potentially breaking doc pages.
> >
> > There was a ticket discussion where a few community members mentioned
> that this task was probably unnecessary. Can we remove it, or shall we fix
> it?
> >
> > Best regards,
> > Ekaterina
>


-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: CASSANDRA-18554 - mTLS based client and internode authenticators

[Derek Chen-Becker]

EC - eventual consensus?

On Tue, Jul 11, 2023 at 4:03 PM Dinesh Joshi  wrote:

> folks - I think we’ve achieved lazy consensus here. Please continue with
> feedback on the jira.
>
> Thanks,
>
> Dinesh
>
>
> On Jul 7, 2023, at 12:23 PM, Jyothsna Konisa 
> wrote:
>
> 
> Hi Yuki, Jeremiah & Christopher,
>
> Thank you very much for the feedback.
>
> Regarding removing superuser check for adding/removing identities, I have
> relaxed that check and added permissions check instead. With this change
> only users with appropriate permissions to add/drop identities can perform
> that action.
>
> About extending `Create Role` cqlsh statement, we have a couple of reasons
> for not doing that. We designed the mTLS authenticator in such a way that a
> single role can be associated with multiple identities, EX: there can be
> several identities which are read_only users. Also, having a separate cqlsh
> statement for identities makes it more pluggable and independent. If we
> still think that extending the create role statement would be a convenient
> feature, we can add it as required in the followup patches.
>
> Christopher, I will be acting upon your feedback regarding having identity
> in the cassandra.yaml optionally configurable.
>
> Thanks,
> Jyothsna Konisa.
>
> On Thu, Jul 6, 2023 at 5:30 PM Dinesh Joshi  wrote:
>
>> > On Jun 30, 2023, at 1:09 PM, Jeremiah Jordan 
>> wrote:
>> >
>> > I don’t think users necessarily need to be able to update their own
>> identities.  I just don’t want to have to use the super user role.  The
>> super user role has all power over all things in the data base.  I don’t
>> want to have to give that much power to the person who manages identities,
>> I just want to give them the power to manage identities.
>>
>> Makes sense. I think Jyothsna already pushed an update to the PR to relax
>> the restriction. Please feel free to take a look at it.
>>
>> Dinesh
>>
>>
>>
>>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

[Derek Chen-Becker]

A strong +1 to getting to a single CI system. CircleCI definitely has some
niceties and I understand why it's currently used, but right now we get 2
CI systems for twice the price. +1 on the proposed subsets.

Derek

On Mon, Jul 10, 2023 at 9:37 AM Josh McKenzie  wrote:

> I'm personally not thinking about CircleCI at all; I'm envisioning a world
> where all of us have 1 CI *software* system (i.e. reproducible on any
> env) that we use for pre-commit validation, and then post-commit happens on
> reference ASF hardware.
>
> So:
> 1: Pre-commit subset of tests (suites + matrices + env) runs. On green,
> merge.
> 2: Post-commit tests (all suites, matrices, env) runs. If failure, link
> back to the JIRA where the commit took place
>
> Circle would need to remain in lockstep with the requirements for point 1
> here.
>
> On Mon, Jul 10, 2023, at 1:04 AM, Berenguer Blasi wrote:
>
> +1 to Josh which is exactly my line of thought as well. But that is only
> valid if we have a solid Jenkins that will eventually run all test configs.
> So I think I lost track a bit here. Are you proposing:
>
> 1- CircleCI: Run pre-commit a single (the most common/meaningful, TBD)
> config of tests
>
> 2- Jenkins: Runs post-commit _all_ test configs and emails/notifies you in
> case of problems?
>
> Or sthg different like having 1 also in Jenkins?
> On 7/7/23 17:55, Andrés de la Peña wrote:
>
> I think 500 runs combining all configs could be reasonable, since it's
> unlikely to have config-specific flaky tests. As in five configs with 100
> repetitions each.
>
> On Fri, 7 Jul 2023 at 16:14, Josh McKenzie  wrote:
>
> Maybe. Kind of depends on how long we write our tests to run doesn't it? :)
>
> But point taken. Any non-trivial test would start to be something of a
> beast under this approach.
>
> On Fri, Jul 7, 2023, at 11:12 AM, Brandon Williams wrote:
>
> On Fri, Jul 7, 2023 at 10:09 AM Josh McKenzie 
> wrote:
> > 3. Multiplexed tests (changed, added) run against all JDK's and a
> broader range of configs (no-vnode, vnode default, compression, etc)
>
> I think this is going to be too heavy...we're taking 500 iterations
> and multiplying that by like 4 or 5?
>
>
>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

[Derek Chen-Becker]

Ultimately I think we have to invest in two directions: first, choose a
consistent, representative subset of stable tests that we feel give us a
reasonable level of confidence in return for a reasonable amount of
runtime. Second, we need to invest in figuring out why certain tests fail.
I strongly dislike the term "flaky" because it suggests that it's some
inconsequential issue causing problems. The truth is that a test that fails
is either a bug in the service code or a bug in the test. I've come to
realize that the CI and build framework is way too complex for me to be
able to help with much, but I would love to start chipping away at failing
test bugs. I'm getting settled into my new job and I should be able to
commit some regular time each week to triage and fixing starting in August,
and if there are any other folks who are interested let me know.

Cheers,

Derek

On Mon, Jul 3, 2023, 12:30 PM Josh McKenzie  wrote:

> Instead of running all the tests through available CI agents every time we
> can have presets of tests:
>
> Back when I joined the project in 2014, unit tests took ~ 5 minutes to run
> on a local machine. We had pre-commit and post-commit tests as a
> distinction as well, but also had flakes in the pre and post batch. I'd
> love to see us get back to a unit test regime like that.
>
> The challenge we've always had is flaky tests showing up in either the
> pre-commit or post-commit groups and difficulty in attribution on a flaky
> failure to where it was introduced (not to lay blame but to educate and
> learn and prevent recurrence). While historically further reduced smoke
> testing suites would just mean more flakes showing up downstream, the rule
> of multiplexing new or changed tests might go a long way to mitigating that.
>
> Should we mention in this concept how we will build the sub-projects (e.g.
> Accord) alongside Cassandra?
>
> I think it's an interesting question, but I also think there's no real
> dependency of process between primary mainline branches and feature
> branches. My intuition is that having the same bar (green CI, multiplex,
> don't introduce flakes, smart smoke suite tiering) would be a good idea on
> feature branches so there's not a death march right before merge, squashing
> flakes when you have to multiplex hundreds of tests before merge to
> mainline (since presumably a feature branch would impact a lot of tests).
>
> Now that I write that all out it does sound Painful. =/
>
> On Mon, Jul 3, 2023, at 10:38 AM, Maxim Muzafarov wrote:
>
> For me, the biggest benefit of keeping the build scripts and CI
> configurations as well in the same project is that these files are
> versioned in the same way as the main sources do. This ensures that we
> can build past releases without having any annoying errors in the
> scripts, so I would say that this is a pretty necessary change.
>
> I'd like to mention the approach that could work for the projects with
> a huge amount of tests. Instead of running all the tests through
> available CI agents every time we can have presets of tests:
> - base tests (to make sure that your design basically works, the set
> will not run longer than 30 min);
> - pre-commit tests (the number of tests to make sure that we can
> safely commit new changes and fit the run into the 1-2 hour build
> timeframe);
> - nightly builds (scheduled task to build everything we have once a
> day and notify the ML if that build fails);
>
>
> My question here is:
> Should we mention in this concept how we will build the sub-projects
> (e.g. Accord) alongside Cassandra?
>
> On Fri, 30 Jun 2023 at 23:19, Josh McKenzie  wrote:
> >
> > Not everyone will have access to such resources, if all you have is 1
> such pod you'll be waiting a long time (in theory one month, and you
> actually need a few bigger pods for some of the more extensive tests, e.g.
> large upgrade tests)….
> >
> > One thing worth calling out: I believe we have a lot of low hanging
> fruit in the domain of "find long running tests and speed them up". Early
> 2022 I was poking around at our unit tests on CASSANDRA-17371 and found
> that 2.62% of our tests made up 20.4% of our runtime (
> https://docs.google.com/spreadsheets/d/1-tkH-hWBlEVInzMjLmJz4wABV6_mGs-2-NNM2XoVTcA/edit#gid=1501761592).
> This kind of finding is pretty consistent; I remember Carl Yeksigian at
> NGCC back in like 2015 axing an hour plus of aggregate runtime by just
> devoting an afternoon to looking at a few badly behaving tests.
> >
> > I'd like to see us move from "1 pod 1 month" down to something a lot
> more manageable. :)
> >
> > Shout-out to Berenger's work on CASSANDRA-16951 for dtest cluster reuse
> (not yet merged), and I have CASSANDRA-15196 to remove the CDC vs. non
> segment allocator distinction and axe the test-cdc target entirely.
> >
> > Ok. Enough of that. Don't want to derail us, just wanted to call out
> that the state of things today isn't the way it has to be.
> >
> > On Fri, Jun 30, 2023, at 4:41 PM, 

Re: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

[Derek Chen-Becker]

Thanks Josh, this looks great! I think the constraints you've outlined are
reasonable for an initial attempt. We can always evolve if we run into
issues.

Cheers,

Derek

On Fri, Jun 30, 2023 at 11:19 AM Josh McKenzie  wrote:

> Context: we're looking to get away from having split CircleCI and ASF CI
> as well
> as getting ASF CI to a stable state. There's a variety of reasons why it's
> flaky
> (orchestration, heterogenous hardware, hardware failures, flaky tests,
> non-deterministic runs, noisy neighbors, etc), many of which Mick has been
> making great headway on starting to address.
>
> If you're curious see:
> - Mick's 2023/01/09 email thread on CI:
> https://lists.apache.org/thread/fqdvqkjmz6w8c864vw98ymvb1995lcy4
> - Mick's 2023/04/26 email thread on CI:
> https://lists.apache.org/thread/xb80v6r857dz5rlm5ckcn69xcl4shvbq
> - CASSANDRA-18137: epic for "Repeatable ci-cassandra.a.o":
> https://issues.apache.org/jira/browse/CASSANDRA-18137
> - CASSANDRA-18133: In-tree build scripts:
> https://issues.apache.org/jira/browse/CASSANDRA-18133
>
> What's fallen out from this: the new reference CI will have the following
> logical layers:
> 1. ant
> 2. build/test scripts that setup the env. See run-tests.sh and
> run-python-dtests.sh here:
>
> https://github.com/thelastpickle/cassandra/tree/0aecbd873ff4de5474fe15efac4cdde10b603c7b/.build
> 3. dockerized build/test scripts that have containerized the flow of 1 and
> 2. See:
>
> https://github.com/thelastpickle/cassandra/tree/0aecbd873ff4de5474fe15efac4cdde10b603c7b/.build/docker
> 4. CI integrations. See generation of unified test report in build.xml:
>
> https://github.com/thelastpickle/cassandra/blame/mck/18133/trunk/build.xml#L1794-L1817
> )
> 5. Optional full CI lifecycle w/Jenkins running in a container (full stack
> setup, run, teardown, pending)
>
>
> *I want to let everyone know the high level structure of how this is
> shaping up,*
>
> *as this is a change that will directly impact the work of *all of us* on
> the*
> *project.*
>
> In terms of our goals, the chief goals I'd like to call out in this
> context are:
> * ASF CI needs to be and remain consistent
> * contributors need a turnkey way to validate their work before merging
> that
> they can accelerate by throwing resources at it.
>
> We as a project need to determine what is *required* to run in a CI
> environment
> to consider that run certified for merge. Where Mick and I landed
> through a lot
> of back and forth is that the following would be required:
> 1. used ant / pytest to build and run tests
> 2. used the reference scripts being changed in CASSANDRA-18133 (in-tree
> .build/)
> to setup and execute your test environment
> 3. constrained your runtime environment to the same hardware and time
> constraints we use in ASF CI, within reason (CPU count independent of
> speed,
> memory size and disk size independent of hardware specs, etc)
> 4. reported test results in a unified fashion that has all the information
> we
> normally get from a test run
> 5. (maybe) Parallelized the tests across the same split lines as upstream
> ASF
> (i.e. no weird env specific neighbor / scheduling flakes)
>
> Last but not least is the "What do we do with CircleCI?" angle. The current
> thought is we allow people to continue using it with the stated goal of
> migrating the circle config over to using the unified build scripts as
> well and
> get it in compliance with the above requirements.
>
> For reference, here's a gdoc where we've hashed this out:
>
> https://docs.google.com/document/d/1TaYMvE5ryOYX03cxzY6XzuUS651fktVER02JHmZR5FU/edit?usp=sharing
>
> So my questions for the community here:
> 1. What's missing from the above conceptualization of the problem?
> 2. Are the constraints too strong? Too weak? Just right?
>
> Thanks everyone, and happy Friday. ;)
>
> ~Josh
>


-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Adding wiremock to test dependencies

[Derek Chen-Becker]

Generally +1 on this one because it looks different, but am I correct that
we already have a couple of mocking libs in our dependencies? I guess my
concern is that adding a dependency is probably as much about adding some
documentation on why it's included and how/when to use it as it is about
updating the build files.

Cheers,

Derek

On Tue, Jun 20, 2023 at 9:24 AM Josh McKenzie  wrote:

> Speaking only to the "we don't want to add a dependency on something
> that's unstable or likely to fizzle out", looks good there to me:
>
>- Long-term project health / activity looks robust:
>https://github.com/wiremock/wiremock/graphs/contributors
>- Pretty diverse set of contributors in the last couple of years:
>
> https://github.com/wiremock/wiremock/graphs/contributors?from=2020-12-29=2023-06-20=c
>
>
> On Tue, Jun 20, 2023, at 9:05 AM, Brandon Williams wrote:
>
> I was concerned about 'jre8' being in the dependency name and looked
> into which java versions were supported, and it looks like 17 is, so
> we should verify but I am +1 if that is the case.
>
> https://github.com/wiremock/wiremock/issues/1655
>
> Kind Regards,
> Brandon
>
> On Tue, Jun 20, 2023 at 6:35 AM Miklosovic, Stefan
>  wrote:
> >
> > Hi,
> >
> > we want to introduce wiremock library (1) into the project as a test
> dependency to test CASSANDRA-16555.
> >
> > In that patch, (wip here (2)), we want to test how would such snitch
> behave based on what Amazon EC2 Identity Service of version 2 returned to
> that snitch. AWS Identity service of version 2 is necessary to call in
> order to get a token with which a snitch is going to get AZ of a node it is
> called from.
> >
> > The last comment of mine in (3) elaborates about approaches we were
> considering and mocking http communication / requests with wiremock seems
> to be like the most comfortable and straightforward solution.
> >
> > Wiremock is Apache licence 2.0 (4) and is well maintained.
> >
> > Are people OK with us introducing this to the build?
> >
> > (1) https://wiremock.org/
> > (2)
> https://github.com/apache/cassandra/pull/2403/files#diff-dc04778c6659040f1c00f37e97a9b1530a532d3d1e3620427bd6628d1b2ec048
> > (3) https://issues.apache.org/jira/browse/CASSANDRA-16555
> > (4) https://github.com/wiremock/wiremock/blob/master/LICENSE.txt
> >
> > Regards
>
>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [VOTE] CEP-8 Datastax Drivers Donation

[Derek Chen-Becker]

+1

On Tue, Jun 13, 2023 at 8:15 AM Jeremy Hanna 
wrote:

> Calling for a vote on CEP-8 [1].
>
> To clarify the intent, as Benjamin said in the discussion thread [2], the
> goal of this vote is simply to ensure that the community is in favor of
> the donation. Nothing more.
> The plan is to introduce the drivers, one by one. Each driver donation
> will need to be accepted first by the PMC members, as it is the case for
> any donation. Therefore the PMC should have full control on the pace at
> which new drivers are accepted.
>
> If this vote passes, we can start this process for the Java driver under
> the direction of the PMC.
>
> Jeremy
>
> 1.
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-8%3A+Datastax+Drivers+Donation
> 2. https://lists.apache.org/thread/opt630do09phh7hlt28odztxdv6g58dp
>


-- 
+-------+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: CASSANDRA-18554 - mTLS based client and internode authenticators

[Derek Chen-Becker]

Sounds great, thanks for the clarification!

Cheers,

Derek

On Sat, Jun 3, 2023 at 12:48 AM Dinesh Joshi  wrote:

> On Jun 2, 2023, at 9:06 PM, Derek Chen-Becker 
> wrote:
>
> This certainly looks like a nice addition to the operator's tools for
> securing cluster access. Out of curiosity, is there anything in this work
> that would *preclude* a different authentication scheme for internode at
> some point in the future? Has there ever been discussion of pluggability
> similar to the client protocol?
>
>
> This is a pluggable implementation so it's not mandatory to use it and
> doesn't preclude one from using a different mechanism in the future. We
> haven't explicitly discussed pluggability i.e. part of protocol negotiation
> in the past for internode connections. However, this work also does not
> preclude us from implementing such changes. If we do add negotiation this
> could be one of the authentication mechanisms. So it would be complimentary.
>
>
> Also, am I correct in understanding that this would allow for multiple
> certificates for the same identity (e.g. distinct cert per node)? I
> certainly understand the decision to keep things simple and have all nodes
> share identity from the perspective of operational simplicity, but I also
> don't want to get in a situation where a single compromised node would
> require an invalidation and redeployment on all nodes in the cluster.
>
>
> I don't recommend all nodes share the same certificate. Each node in the
> cluster should obtain a unique certificate with the same SPIFFE. In the
> event a node is compromised, the operator can revoke that node's
> certificate without having to redeploy to all nodes in the cluster.
>
> thanks,
>
> Dinesh
>


-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: CASSANDRA-18554 - mTLS based client and internode authenticators

[Derek Chen-Becker]

Hi Dinesh,

This certainly looks like a nice addition to the operator's tools for
securing cluster access. Out of curiosity, is there anything in this work
that would *preclude* a different authentication scheme for internode at
some point in the future? Has there ever been discussion of pluggability
similar to the client protocol? Also, am I correct in understanding that
this would allow for multiple certificates for the same identity (e.g.
distinct cert per node)? I certainly understand the decision to keep things
simple and have all nodes share identity from the perspective of
operational simplicity, but I also don't want to get in a situation where a
single compromised node would require an invalidation and redeployment on
all nodes in the cluster.

Cheers,

Derek

On Fri, Jun 2, 2023 at 12:57 PM Dinesh Joshi  wrote:

>
>1. Is there an expectation that this would be used alongside internode
>and client TLS? Would the certificates be the same, different, or is that
>an implementation detail for the specific deployment to determine?
>
>
> I am not sure what you mean by this would be used alongside internode and
> client TLS? The mutual TLS authentication allows the server to authenticate
> the client's identity using a client TLS certificate. The authenticators
> we're adding enable this functionality. There isn't an expectation that the
> same certificates be used. In fact, clients should not use the same
> certificates as the internode.
>
>
>1. For some reason I'm having trouble understanding the internode
>authentication portion of this ticket (authenticating a client with a
>certificate makes sense vs just authenticating the connection). Why is this
>needed on top of the connection-level TLS we have now?
>
>
> Current connection-level TLS only secures the TLS connection. It doesn't
> authenticate the peer. This adds the ability to authenticate the peer in
> addition to securing the TLS connection.
>
>
>1. When an operator or DBA is looking to add a new identity is that
>just handled as part of the new CQL statement or is there some certificate
>management required on the nodes? I assume it's just the CA that needs to
>be placed on the nodes to establish trust in the certificate itself then
>authz happening within C* after determining the certificate can be trusted,
>but want to be certain.
>
>
> Each deployment is different so certificate management isn't scope of the
> database itself. However, the operator can rotate the certificates using an
> external agent and Cassandra will pick them up through SSL hot reloading.
> We don't just rely on a CA trusting the client certificate, we extract the
> identity from the certificate (for example: SPIFFE) and ensure that it is
> allowed to access the cluster. This means someone can't just obtain a
> certificate signed by a CA that Cassandra cluster trusts and connect to it.
>
>
>1. As a minor nit, should we include static certificates in the test
>data? I see they expire in 2033 which is a fair way off, but I wonder if it
>would make sense to generate the certificates as part of the test setup.
>
> Thanks for the feedback!
>


-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSS] CEP-31 negotiated authentication

[Derek Chen-Becker]

Hi Jacek,

I took a quick look through the CEP and I think I understand the
implementation you're donating. I don't think that the approach you're
taking and the approach I proposed are contradictory, but I want to make
sure I'm understanding some aspects of the CEP:

1. Is there any mechanism for discovery so that the client knows which
authenticators are supported? The main use case I see here is that since
the client drives selection of the authenticator, the client probably wants
to utilize the strongest mutually supported mechanism
2. Can you specify the client/server exchange in a state diagram or more
clearly detail which messages are involved? The CEP states that "The driver
sends an additional preamble along with the initial SASL authentication
message". Is the "initial SASL auth message" the AUTH_RESPONSE? Are you
basically saying that the server sends the AUTHENTICATE message with a
class name, so does the client basically respond with "No, here's the
authenticator I want to use" in the preamble?
3. Does the donated code for the server already handle hot reconfiguration
of authenticators? The CEP states "We want to make it possible to add, ..."
so I wasn't sure if that was future work or not

I think I need to re-read and digest, but on first run-through this looks
really interesting!

Cheers,

Derek

On Fri, May 26, 2023 at 8:09 AM Jacek Lewandowski <
lewandowski.ja...@gmail.com> wrote:

> Hi,
>
> I'd like to start a discussion on negotiated authentication and
> improvements to authentication, authorization, and role management in
> general. A draft of proposed changes is included in CEP-31.
>
>
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-31+%28DRAFT%29+Negotiated+authentication+and+authorization
>
> thanks,
> - - -- --- -  -
> Jacek Lewandowski
>


-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [CASSANDRA-11471] Authentication mechanism negotiation (OPTIONS/SUPPORTED)

[Derek Chen-Becker]

Hi Jacek, thanks for sharing! I'm super busy until the middle of next week
but I've blocked out some time to read through this. I'm looking forward to
the discussion :)

Cheers,

Derek

On Fri, May 26, 2023 at 2:08 AM Jacek Lewandowski <
lewandowski.ja...@gmail.com> wrote:

> Hi,
>
> I'm happy that this discussion has started because we wanted to propose
> something we have been working on. I've created
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-31+%28DRAFT%29+Negotiated+authentication+and+authorization
> with some details. It is a draft and we are open to suggestions and
> cooperation on that.
>
> Basically what we have been using in DataStax for years already is based
> on embedding negotiations in SASL messages. In particular, the
> authentication mechanism chosen by the client is sent in a preamble of the
> first authentication message. It does not require a new protocol -
> everything we need is internal to authentication message exchange. It is
> also backward compatible as if the server does not receive the mechanism
> preamble, it will just continue assuming the default authentication
> mechanism.
>
> Please have a look and let me know what you think. I'll start a separate
> thread on that topic soon.
>
> thanks
> - - -- --- -  -
> Jacek Lewandowski
>
>
> pt., 26 maj 2023 o 04:08 Dinesh Joshi  napisał(a):
>
>> Leaving the naming aside (the hardest part of any software), I am
>> generally positive about your idea. A protocol version bump may be
>> avoidable like you suggested. Perhaps a prototype of this idea is in order
>> to help shape the idea? Would you like to take it on?
>>
>> On May 21, 2023, at 4:21 AM, Derek Chen-Becker 
>> wrote:
>>
>> We had a recent discussion in Slack about how to potentially use the
>> OPTIONS and SUPPORTED messages in the existing CQL protocol to allow the
>> server to advertise more than one authentication method and allow the
>> client to then choose which authenticator to use. The primary use case here
>> is to allow seamless migration to a new authenticator without having to
>> have all parties involved agree on a single class (and avoid a disruptive
>> change). There's already a ticket open that was focused on making a change
>> to the binary protocol (
>> https://issues.apache.org/jira/browse/CASSANDRA-11471) but I think that
>> we can accomplish this in a backwards compatible way that avoids a change
>> to the protocol itself.
>>
>> What I propose is to allow a server configured for this graceful auth
>> change to send an additional value in the [string multimap] body of the
>> SUPPORTED message that indicates which authenticators are supported, in
>> descending priority order. For example, if I wanted to migrate my server to
>> support both PlainTextAuthProvider and some new MyAwesomeAuthProvider, I
>> would configure my client to query options and the server would respond with
>>
>> 'AUTHENTICATORS': ['MyAwesomeAuthProvider', 'PlainTextAuthProvider']
>>
>> The client can then choose from its own supported providers and send it
>> as part of the STARTUP message [string map] body:
>>
>> 'AUTHENTICATOR': 'MyAwesomeAuthenticator'
>>
>> I'm not good with naming so feel free to propose a different key for
>> either of these map entries. In any case, the server then validates that
>> the client-chosen authenticator is actually supported and would then
>> proceed with the AUTHENTICATE message. In the case where the client sends
>> an invalid/unsupported authenticator choice, the server can simply respond
>> with an AUTHENTICATE using the most-preferred configured authenticator.
>>
>> I think this is a better approach than changing the binary protocol
>> because the mechanism already exists for negotiating options and this seems
>> like a natural use case that avoids having to create an entirely new
>> version of the protocol. It does not appear to conflict with the existing
>> protocol definition but I'm not 100% certain. Section 4.1.1 discusses
>> "Possible options"  for the STARTUP message (
>> https://github.com/apache/cassandra/blob/trunk/doc/native_protocol_v4.spec#L296),
>> but that's an unfortunate use of English that's ambiguous as to whether it
>> means "The only ones supported" or "Supported but not exclusively".
>>
>> I've taken a look at the Java and Python driver source so far and I can't
>> find anything that would seem to cause a problem by returning a SUPPORTED
>> multimap entry that the client isn't aware of (in both they would appear to
>> ignore it), but I'll a

[CASSANDRA-11471] Authentication mechanism negotiation (OPTIONS/SUPPORTED)

[Derek Chen-Becker]

We had a recent discussion in Slack about how to potentially use the
OPTIONS and SUPPORTED messages in the existing CQL protocol to allow the
server to advertise more than one authentication method and allow the
client to then choose which authenticator to use. The primary use case here
is to allow seamless migration to a new authenticator without having to
have all parties involved agree on a single class (and avoid a disruptive
change). There's already a ticket open that was focused on making a change
to the binary protocol (
https://issues.apache.org/jira/browse/CASSANDRA-11471) but I think that we
can accomplish this in a backwards compatible way that avoids a change to
the protocol itself.

What I propose is to allow a server configured for this graceful auth
change to send an additional value in the [string multimap] body of the
SUPPORTED message that indicates which authenticators are supported, in
descending priority order. For example, if I wanted to migrate my server to
support both PlainTextAuthProvider and some new MyAwesomeAuthProvider, I
would configure my client to query options and the server would respond with

'AUTHENTICATORS': ['MyAwesomeAuthProvider', 'PlainTextAuthProvider']

The client can then choose from its own supported providers and send it as
part of the STARTUP message [string map] body:

'AUTHENTICATOR': 'MyAwesomeAuthenticator'

I'm not good with naming so feel free to propose a different key for either
of these map entries. In any case, the server then validates that the
client-chosen authenticator is actually supported and would then proceed
with the AUTHENTICATE message. In the case where the client sends an
invalid/unsupported authenticator choice, the server can simply respond
with an AUTHENTICATE using the most-preferred configured authenticator.

I think this is a better approach than changing the binary protocol because
the mechanism already exists for negotiating options and this seems like a
natural use case that avoids having to create an entirely new version of
the protocol. It does not appear to conflict with the existing protocol
definition but I'm not 100% certain. Section 4.1.1 discusses "Possible
options"  for the STARTUP message (
https://github.com/apache/cassandra/blob/trunk/doc/native_protocol_v4.spec#L296),
but that's an unfortunate use of English that's ambiguous as to whether it
means "The only ones supported" or "Supported but not exclusively".

I've taken a look at the Java and Python driver source so far and I can't
find anything that would seem to cause a problem by returning a SUPPORTED
multimap entry that the client isn't aware of (in both they would appear to
ignore it), but I'll also admit that this is the first time I've looked at
this part of the client code and I could be missing something. Is anyone
aware of possible problems that would be caused by using this approach? In
particular, if there are clients that strictly validate all entries in the
SUPPORTED map then this could cause a problem.

Worst case, we may still need a protocol version bump if the enumeration of
STARTUP options is intended to be strict, but at least this would not
require a new message type and would fit into the existing framework for
negotiation between client and server.

Thoughts, questions, or concerns would be appreciated.

Cheers,

Derek

-- 
+-------+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [POLL] Vector type for ML

[Derek Chen-Becker]

LOL, I'm holding you to that at the summit :) In all seriousness, I'm glad
to see a robust debate around it. I guess for completeness, my order of
preference is

1 - NONNULL FROZEN>
2 - NONNULL TYPE (which part of this implies frozen? The NONNULL or the
cardinality?)
3 - DENSE_VECTOR

I guess my main concern with just "VECTOR" is that it's such an overloaded
term. Maybe in ML it means something specific, but for anyone coming from
C++, Rust, Java, etc, a Vector is both mutable and can carry null (or
equivalent, e.g. None, in Rust). If the argument hadn't also been made that
we should be working toward something that's not ML-specific maybe I would
be less concerned.

Cheers,

Derek


Cheers,

Derek

On Fri, May 5, 2023 at 11:14 AM Patrick McFadin  wrote:

> Derek, despite your preference, I would hang out with you at a party.
>
> On Fri, May 5, 2023 at 9:44 AM Derek Chen-Becker 
> wrote:
>
>> Speaking as someone who likes Erlang, maybe that's why I also like
>> NONNULL FROZEN>. It's unambiguous what Cassandra is going to do
>> with that type. DENSE VECTOR means I need to go read docs (and then
>> probably double-check in the source to be sure) to be sure what exactly is
>> going on.
>>
>> Cheers,
>>
>> Derek
>>
>> On Fri, May 5, 2023 at 9:54 AM Patrick McFadin 
>> wrote:
>>
>>> I hope we are willing to consider developers that use our system because
>>> if I had to teach people to use "NON-NULL FROZEN" I'm pretty sure
>>> the response would be:
>>>
>>> Did you tell me to go write a distributed map-reduce job in Erlang? I
>>> beleive I did, Bob.
>>>
>>> On Fri, May 5, 2023 at 8:05 AM Josh McKenzie 
>>> wrote:
>>>
>>>> Idiomatically, to my mind, there's a question of "what space are we
>>>> thinking about this datatype in"?
>>>>
>>>> - In the context of mathematics, nullability in a vector would be 0
>>>> - In the context of Cassandra, nullability tends to mean a tombstone
>>>> (or nothing)
>>>> - In the context of programming languages, it's all over the place
>>>>
>>>> Given many models are exploring quantizing to int8 and other data
>>>> types, there's definitely the "support other data types easily in the
>>>> future" piece to me we need to keep in mind.
>>>>
>>>> So with the above and the "meet the user where they are and don't make
>>>> them understand more of Cassandra than absolutely critical to use it", I
>>>> lean:
>>>>
>>>> 1. DENSE_VECTOR
>>>> 2. VECTOR
>>>> 3. type[dimension]
>>>>
>>>> This leaves the path open for us to expand on it in the future with
>>>> sparse support and allows us to introduce some semantics that indicate
>>>> idioms around nullability for the users coming from a different space.
>>>>
>>>> "NON-NULL FROZEN" is strictly correct, however it requires
>>>> understanding idioms of how Cassandra thinks about data (nulls mean
>>>> different things to us, we have differences between frozen and non-frozen
>>>> due to constraints in our storage engine and materialization of data, etc)
>>>> that get in the way of users doing things in the pattern they're familiar
>>>> with without learning more about the DB than they're probably looking to
>>>> learn. Historically this has been a challenge for us in adoption; the
>>>> classic "Why can't I just write and delete and write as much as I want? Why
>>>> are deletes filling up my disk?" problem comes to mind.
>>>>
>>>> I'd also be happy with us supporting:
>>>> * NON-NULL FROZEN
>>>> * DENSE_VECTOR as syntactic sugar for the above
>>>>
>>>> If getting into the "built-in syntactic sugar mapping for communities
>>>> and specific use-cases" is something we're willing to consider.
>>>>
>>>> On Fri, May 5, 2023, at 7:26 AM, Patrick McFadin wrote:
>>>>
>>>> I think we are still discussing implementation here when I'm talking
>>>> about developer experience. I want developers to adopt this quickly, easily
>>>> and be successful. Vector search is already a thing. People use it every
>>>> day. A successful outcome, in my view, is developers picking up this
>>>> feature without reading a manual. (Because they don't anyway and get in
>>>> trouble) I did some more extensive research about what other DBs are 

Re: [POLL] Vector type for ML

[Derek Chen-Becker]

nitial agreement.
>>
>> For syntax, I think one option was just FLOAT[N]. In VECTOR FLOAT[N],
>> VECTOR is redundant - FLOAT[N] is fully descriptive by itself. I don’t
>> think VECTOR should be used to simply imply non-null, as this would be very
>> unintuitive. More logical would be NONNULL, if this is the only condition
>> being applied. Alternatively for arrays we could default to NONNULL and
>> later introduce NULLABLE if we want to permit nulls.
>>
>> If the word vector is to be used it makes more sense to make it look like
>> a list, so VECTOR as here the word VECTOR is clearly not
>> redundant.
>>
>> So, I vote:
>>
>> 1) (NON NULL) FLOAT[N]
>> 2) FLOAT[N]   (Non null by default)
>> 3) VECTOR
>>
>>
>>
>> On 4 May 2023, at 08:52, Mick Semb Wever  wrote:
>>
>> 
>>
>>
>> Did we agree on a CQL syntax?
>>
>> I don’t believe there has been a pool on CQL syntax… my understanding
>> reading all the threads is that there are ~4-5 options and non are -1ed, so
>> believe we are waiting for majority rule on this?
>>
>>
>>
>>
>> Re-reading that thread, IIUC the valid choices remaining are…
>>
>> 1. VECTOR FLOAT[n]
>> 2. FLOAT VECTOR[n]
>> 3. VECTOR
>> 4. VECTOR[n]
>> 5. ARRAY
>> 6. NON-NULL FROZEN
>>
>>
>> Yes I'm putting my preference (1) first ;) because (banging on) if the
>> future of CQL will have FLOAT[n] and FROZEN, where the VECTOR
>> keyword is: for general cql users; just meaning "non-null and frozen",
>> these gel best together.
>>
>> Options (5) and (6) are for those that feel we can and should provide
>> this type without introducing the vector keyword.
>>
>>
>>
>>
>>
>> --
>> [image: DataStax Logo Square] <https://www.datastax.com/>
>> *Mike Adamson*
>> Engineering
>> +1 650 389 6000 <16503896000> | datastax.com <https://www.datastax.com/>
>> Find DataStax Online:
>> [image: LinkedIn Logo]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_datastax=DwMFaQ=adz96Xi0w1RHqtPMowiL2g=IFj3MdIKYLLXIUhYdUGB0cTzTlxyCb7_VUmICBaYilU=uHzE4WhPViSF0rsjSxKhfwGDU1Bo7USObSc_aIcgelo=akx0E6l2bnTjOvA-YxtonbW0M4b6bNg4nRwmcHNDo4Q=>
>>[image: Facebook Logo]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_datastax=DwMFaQ=adz96Xi0w1RHqtPMowiL2g=IFj3MdIKYLLXIUhYdUGB0cTzTlxyCb7_VUmICBaYilU=uHzE4WhPViSF0rsjSxKhfwGDU1Bo7USObSc_aIcgelo=ncMlB41-6hHuqx-EhnM83-KVtjMegQ9c2l2zDzHAxiU=>
>>[image: Twitter Logo] <https://twitter.com/DataStax>   [image: RSS
>> Feed] <https://www.datastax.com/blog/rss.xml>   [image: Github Logo]
>> <https://github.com/datastax>
>>
>>
>>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSS] CEP-28: Reading and Writing Cassandra Data with Spark Bulk Analytics

[Derek Chen-Becker]

On Tue, Mar 28, 2023 at 9:03 AM Joseph Lynch  wrote:
...

I think we might be underselling how valuable JVM isolation is,
> especially for analytics queries that are going to pass the entire
> dataset through heap somewhat constantly.
>

Big +1 here. The JVM simply does not have significant granularity of
control for resource utilization, but this is explicitly a feature of
separate processes. Add in being able to separate GC domains and you can
avoid a lot of noisy neighbor in-VM behavior for the disparate workloads.

Cheers,

Derek


-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Welcome our next PMC Chair Josh McKenzie

[Derek Chen-Becker]

Congratulations, Josh!

On Thu, Mar 23, 2023, 4:23 AM Mick Semb Wever  wrote:

> It is time to pass the baton on, and on behalf of the Apache Cassandra
> Project Management Committee (PMC) I would like to welcome and congratulate
> our next PMC Chair Josh McKenzie (jmckenzie).
>
> Most of you already know Josh, especially through his regular and valuable
> project oversight and status emails, always presenting a balance and
> understanding to the various views and concerns incoming.
>
> Repeating Paulo's words from last year: The chair is an administrative
> position that interfaces with the Apache Software Foundation Board, by
> submitting regular reports about project status and health. Read more about
> the PMC chair role on Apache projects:
> - https://www.apache.org/foundation/how-it-works.html#pmc
> - https://www.apache.org/foundation/how-it-works.html#pmc-chair
> - https://www.apache.org/foundation/faq.html#why-are-PMC-chairs-officers
>
> The PMC as a whole is the entity that oversees and leads the project and
> any PMC member can be approached as a representative of the committee. A
> list of Apache Cassandra PMC members can be found on:
> https://cassandra.apache.org/_/community.html
>

Re: [DISCUSS] New dependencies with Chronicle-Queue update

[Derek Chen-Becker]

Deletion is the highest form of engineering ;)

On Thu, Mar 16, 2023 at 7:09 AM Mick Semb Wever  wrote:

> >  asm-analysis-9.4.jar
> >  asm-commons-9.4.jar
> >  asm-tree-9.4.jar
> >  asm-util-9.4.jar
>
>
> FYI, on further inspection of the posix dependency, i've excluded
> these four asm* dependencies.
>


-- 
+-------+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: hsqldb test dependency in simulator code

[Derek Chen-Becker]

Do we have the hsqldb library just for a primitive oriented equivalent of
Map? If this is test code only that seems like a pretty easy
thing to replace and I would vote to implement that so we can fully drop
the dependency.

Cheers,

Derek

On Tue, Mar 14, 2023 at 5:49 AM Miklosovic, Stefan <
stefan.mikloso...@netapp.com> wrote:

> Hi list,
>
> while removing Hadoop code in trunk, as agreed on ML recently, I did that
> but we need to do this (1). By removing all Hadoop dependencies, we also
> removed hsqldb library we surprisingly rely on so I am putting it back.
>
> Honestly, I do not know if people are even aware of the fact we are
> depending on hsqldb in simulator package. Can not we rewrite that so we can
> drop this dependency entirely?
>
> Do we want to spend time on rewriting this and then we remove Hadoop
> libraries or I just add this library among the test ones?
>
> Thanks
>
> (1) https://github.com/apache/cassandra/pull/2212/files#r1133841928



-- 
+-------+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSSION] Cassandra + Java 17

[Derek Chen-Becker]

f Unsafe? History around the choice
>>>> of using it in C* and future plans I might not know of?
>>>>
>>>
>>> It was used mainly to get around the fact that Java did not offer other
>>> means to do certain things.
>>> Outside of trying to anticipate some of the restrictions of that API and
>>> make sure that the JDK offers a suitable replacement for us. I am not sure
>>> that there is much that we can do. But I might misunderstand your question.
>>>
>>> Le mer. 1 mars 2023 à 21:16, Ekaterina Dimitrova 
>>> a écrit :
>>>
>>>> Hi everyone,
>>>> Some updates and questions around JDK 17 below.
>>>> First of all, I wanted to let people know that currently Cassandra
>>>> trunk can be already compiled and run with J8 + J11 + J17. This is the
>>>> product of the realization that the feature branch makes it harder for
>>>> working on JDK17 related tickets due to the involvement of too many moving
>>>> parts. Agreement reached in [1] that new JDK introduction can be done
>>>> incrementally. Scripted UDFs removed, hooks to be added in a follow up
>>>> ticket.
>>>> What does this mean?
>>>> - Currently you can compile and run Cassandra trunk  with JDK
>>>> 17(further to 8+11). You can run unit and java distributed tests already
>>>> with JDK17
>>>> - CASSANDRA-18106 in progress,  enabling CCM to handle JDK8, 11 and 17
>>>> with trunk and when that is ready we will be able to run also Python tests;
>>>> After that one lands it comes CASSANDRA-18247 ; its goal is to add CircleCI
>>>> config (separate of the one we have for 8+11) for 11+17 which can be used
>>>> from people who work on JDK17 related issues. Patch proposal already in the
>>>> ticket. Final version we will have when we do the switch 8+11 to 11+17,
>>>> things will go through evolution.
>>>>
>>>> What does this mean? Anyone who is interested to test or to help with
>>>> JDK17 effort can easily do it directly from trunk. Jenkins and CircleCI are
>>>> not switched from 8+11 to 11+17 until we are ready. Only test experimental
>>>> additional CircleCI config will be added, temporary to make it easier for
>>>> testing
>>>>
>>>> To remind you - the umbrella ticket for the JDK17 builds is
>>>> CASSANDRA-16895.
>>>> Good outstanding candidate still not assigned - CASSANDRA-18180, if
>>>> anyone has cycles, please, take a look at it. CASSANDRA-18263 might be also
>>>> of interest to someone.
>>>>
>>>> In other news, I added already to the JDK17 jvm options certain
>>>> imports/exports which are needed at this point but as we agreed in the past
>>>> - it will be good to try to eliminate as many of them as we can. Consider
>>>> those experimental in my opinion.
>>>> Some of them though are related to:
>>>> -- some were added already from 11; thoughts?
>>>> *-- *some will be eliminated with some maintenance in progress
>>>> *-- *some are related to
>>>> https://chronicle.software/chronicle-support-java-17. I guess we are
>>>> cornered with those until Chronicle eliminates the need for those.
>>>> (CASSANDRA-18049)
>>>> -- Find a way to get FileDescriptor.fd and
>>>> sun.nio.ch.FileChannelImpl.fd without opening internals (
>>>> CASSANDRA-17850)
>>>> -- we also use setAccessible at numerous places.
>>>> And I am sure our CI will tell me I am missing something, especially
>>>> when trunk is alive...
>>>>
>>>> A few other questions:
>>>> - thoughts around the usage/future of Unsafe? History around the choice
>>>> of using it in C* and future plans I might not know of?
>>>> - ECJ - It seems the compiler artifacts are moved from here
>>>> <https://mvnrepository.com/artifact/org.eclipse.jdt.core.compiler/ecj>
>>>> to here <https://mvnrepository.com/artifact/org.eclipse.jdt/ecj> and
>>>> there is change of license from EPL1.0 to EPL2.0 too. But if I read
>>>> correctly here
>>>> <https://www.apache.org/legal/resolved.html#weak-copyleft-licenses> that
>>>> should not affect us. I am dealing with this in CASSANDRA-18190. Please let
>>>> me know if you see any problem with this that I might be missing.
>>>> - Looking at the history of tickets around JMXServerUtils class I guess
>>>> it was accepted that we might have breakages (and we already had
>>>> CASSANDRA-14173) - JmxRegistry extends sun.rmi.registry.RegistryImpl?
>>>>
>>>> Best regards,
>>>> Ekaterina
>>>>
>>>> [1] https://lists.apache.org/thread/c39yrbdszgz9s34vr17wpjdhv6h2oo61
>>>>
>>>>
>>>>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Role of Hadoop code in Cassandra 5.0

[Derek Chen-Becker]

Honestly, I don't think moving it out in its current state is a win,
either. I'm +1 to deprecation in 4.1.x and removal in 5.0. If someone in
the community wants or needs the Hadoop code it should be in a separate
repo/package just like the Spark Connector.

Derek

On Thu, Mar 9, 2023 at 10:07 AM Miklosovic, Stefan <
stefan.mikloso...@netapp.com> wrote:

> Derek,
>
> I have couple more points ... I do not think that extracting it to a
> separate repository is "win". That code is on Hadoop 1.0.3. We would be
> spending a lot of work on extracting it just to extract 10 years old code
> with occasional updates (in my humble opinion just to make it compilable
> again if the code around changes). What good is in that? We would have one
> more place to take care of ... Now we at least have it all in one place.
>
> I believe we have four options:
>
> 1) leave it there so it will be like this is for next years with
> questionable and diminishing usage
> 2) update it to Hadoop 3.3 (I wonder who is going to do that)
> 3) 2) and extract it to a separate repository but if we do 2) we can just
> leave it there
> 4) remove it
>
> 
> From: Derek Chen-Becker 
> Sent: Thursday, March 9, 2023 15:55
> To: dev@cassandra.apache.org
> Subject: Re: Role of Hadoop code in Cassandra 5.0
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> I think the question isn't "Who ... is still using that?" but more "are we
> actually going to support it?" If we're on a version that old it would
> appear that we've basically abandoned it, although there do appear to have
> been refactoring (for other things) commits in the last couple of years. I
> would be in favor of removal from 5.0, but at the very least, could it be
> moved into a separate repo/package so that it's not pulling a relatively
> large dependency subtree from Hadoop into our main codebase?
>
> Cheers,
>
> Derek
>
> On Thu, Mar 9, 2023 at 6:44 AM Miklosovic, Stefan <
> stefan.mikloso...@netapp.com<mailto:stefan.mikloso...@netapp.com>> wrote:
> Hi list,
>
> I stumbled upon Hadoop package again. I think there was some discussion
> about the relevancy of Hadoop code some time ago but I would like to ask
> this again.
>
> Do you think Hadoop code (1) is still relevant in 5.0? Who in the industry
> is still using that?
>
> We might drop a lot of code and some Hadoop dependencies too (3) (even
> their scope is "provided"). The version of Hadoop we build upon is 1.0.3
> which was released 10 years ago. This code does not have any tests nor
> documentation on the website.
>
> There seems to be issues like this (2) and it seems like the solution is
> to, basically, use Spark Cassandra connector instead which I would say is
> quite reasonable.
>
> Regards
>
> (1)
> https://github.com/apache/cassandra/tree/trunk/src/java/org/apache/cassandra/hadoop
> (2) https://lists.apache.org/thread/jdy5hdc2l7l29h04dqol5ylroqos1y2p
> (3)
> https://github.com/apache/cassandra/blob/trunk/.build/parent-pom-template.xml#L507-L589
>
>
> --
> +---+
> | Derek Chen-Becker |
> | GPG Key available at https://keybase.io/dchenbecker and   |
> | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
> | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
> +---+
>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Role of Hadoop code in Cassandra 5.0

[Derek Chen-Becker]

I think the question isn't "Who ... is still using that?" but more "are we
actually going to support it?" If we're on a version that old it would
appear that we've basically abandoned it, although there do appear to have
been refactoring (for other things) commits in the last couple of years. I
would be in favor of removal from 5.0, but at the very least, could it be
moved into a separate repo/package so that it's not pulling a relatively
large dependency subtree from Hadoop into our main codebase?

Cheers,

Derek

On Thu, Mar 9, 2023 at 6:44 AM Miklosovic, Stefan <
stefan.mikloso...@netapp.com> wrote:

> Hi list,
>
> I stumbled upon Hadoop package again. I think there was some discussion
> about the relevancy of Hadoop code some time ago but I would like to ask
> this again.
>
> Do you think Hadoop code (1) is still relevant in 5.0? Who in the industry
> is still using that?
>
> We might drop a lot of code and some Hadoop dependencies too (3) (even
> their scope is "provided"). The version of Hadoop we build upon is 1.0.3
> which was released 10 years ago. This code does not have any tests nor
> documentation on the website.
>
> There seems to be issues like this (2) and it seems like the solution is
> to, basically, use Spark Cassandra connector instead which I would say is
> quite reasonable.
>
> Regards
>
> (1)
> https://github.com/apache/cassandra/tree/trunk/src/java/org/apache/cassandra/hadoop
> (2) https://lists.apache.org/thread/jdy5hdc2l7l29h04dqol5ylroqos1y2p
> (3)
> https://github.com/apache/cassandra/blob/trunk/.build/parent-pom-template.xml#L507-L589



-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Degradation of availability when using NTS and RF > number of racks

[Derek Chen-Becker]

there is already some logic around this scenario
> (1) but the implementation is not entirely correct. This solution is not
> computing the replica placement correctly so the above problem would be
> addressed.
> >>
> >> We created a draft here (2, 3) which fixes it.
> >>
> >> There is also a test which simulates this scenario. When I assign 256
> tokens to each node randomly (by same mean as generatetokens command uses)
> and I try to compute natural replicas for 1 billion random tokens and I
> compute how many cases there will be when 3 replicas out of 5 are inserted
> in the same rack (so by losing it we would lose quorum), for above setup I
> get around 6%.
> >>
> >> For 12 nodes, 3 racks, 4 nodes per rack, rf = 5, this happens in 10%
> cases.
> >>
> >> To interpret this number, it basically means that with such topology,
> RF and CL, when a random rack fails completely, when doing a random read,
> there is 6% chance that data will not be available (or 10%, respectively).
> >>
> >> One caveat here is that NTS is not compatible with this new strategy
> anymore because it will place replicas differently. So I guess that fixing
> this in NTS will not be possible because of upgrades. I think people would
> need to setup completely new keyspace and somehow migrate data if they wish
> or they just start from scratch with this strategy.
> >>
> >> Questions:
> >>
> >> 1) do you think this is meaningful to fix and it might end up in trunk?
> >>
> >> 2) should not we just ban this scenario entirely? It might be possible
> to check the configuration upon keyspace creation (rf > num of racks) and
> if we see this is problematic we would just fail that query? Guardrail
> maybe?
> >>
> >> 3) people in the ticket mention writing "CEP" for this but I do not see
> any reason to do so. It is just a strategy as any other. What would that
> CEP would even be about? Is this necessary?
> >>
> >> Regards
> >>
> >> (1)
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java#L126-L128
> >> (2) https://github.com/apache/cassandra/pull/2191
> >> (3) https://issues.apache.org/jira/browse/CASSANDRA-16203
> >
>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Degradation of availability when using NTS and RF > number of racks

[Derek Chen-Becker]

1) It does seem a like a big footgun. I think it violates the principle of
least surprise if someone has configured NTS thinking that they are
improving availability
2) I don't know that we want to ban it outright, since maybe there's a case
for someone to be using a different CL that would be OK with the loss of a
majority of replicas (e.g. ONE). For example, we don't fail if someone uses
ALL or EACH_QUORUM with a problematic setup, do we? Would we warn on
keyspace creation with RF > racks or are you suggesting that the warning
would be at query time?
3) agreed, this doesn't seem like an enhancement as much as it is
identifying legal but likely incorrect configuration

Cheers,

Derek

On Mon, Mar 6, 2023 at 3:52 AM Miklosovic, Stefan <
stefan.mikloso...@netapp.com> wrote:

> Hi all,
>
> some time ago we identified an issue with NetworkTopologyStrategy. The
> problem is that when RF > number of racks, it may happen that NTS places
> replicas in such a way that when whole rack is lost, we lose QUORUM and
> data are not available anymore if QUORUM CL is used.
>
> To illustrate this problem, lets have this setup:
>
> 9 nodes in 1 DC, 3 racks, 3 nodes per rack. RF = 5. Then, NTS could place
> replicas like this: 3 replicas in rack1, 1 replica in rack2, 1 replica in
> rack3. Hence, when rack1 is lost, we do not have QUORUM.
>
> It seems to us that there is already some logic around this scenario (1)
> but the implementation is not entirely correct. This solution is not
> computing the replica placement correctly so the above problem would be
> addressed.
>
> We created a draft here (2, 3) which fixes it.
>
> There is also a test which simulates this scenario. When I assign 256
> tokens to each node randomly (by same mean as generatetokens command uses)
> and I try to compute natural replicas for 1 billion random tokens and I
> compute how many cases there will be when 3 replicas out of 5 are inserted
> in the same rack (so by losing it we would lose quorum), for above setup I
> get around 6%.
>
> For 12 nodes, 3 racks, 4 nodes per rack, rf = 5, this happens in 10% cases.
>
> To interpret this number, it basically means that with such topology, RF
> and CL, when a random rack fails completely, when doing a random read,
> there is 6% chance that data will not be available (or 10%, respectively).
>
> One caveat here is that NTS is not compatible with this new strategy
> anymore because it will place replicas differently. So I guess that fixing
> this in NTS will not be possible because of upgrades. I think people would
> need to setup completely new keyspace and somehow migrate data if they wish
> or they just start from scratch with this strategy.
>
> Questions:
>
> 1) do you think this is meaningful to fix and it might end up in trunk?
>
> 2) should not we just ban this scenario entirely? It might be possible to
> check the configuration upon keyspace creation (rf > num of racks) and if
> we see this is problematic we would just fail that query? Guardrail maybe?
>
> 3) people in the ticket mention writing "CEP" for this but I do not see
> any reason to do so. It is just a strategy as any other. What would that
> CEP would even be about? Is this necessary?
>
> Regards
>
> (1)
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java#L126-L128
> (2) https://github.com/apache/cassandra/pull/2191
> (3) https://issues.apache.org/jira/browse/CASSANDRA-16203



-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSS] Merging incremental feature work

[Derek Chen-Becker]

 and nobody else picks it up.
>>
>>
>>
>> So, returning back to this day and this database... Basically what you
>> want to avoid is to paint yourself into a corner, and particularly the
>> wrong corner. So the way I would answer this question is that large bodies
>> of work should:
>>  - Refactoring that is a) harmless, and/or b) improves the codebase
>> anyway, should be merged early into trunk.
>>  - The main body of the new functionality should be developed in a
>> feature branch up until some kind of MVP stage. This means that by the time
>> it is proposed for merge, a) it has been tested to both be of good quality
>> and that it actually provides the benefit it set out to implement. This
>> means that merging it to trunk will be a net improvement, always.
>>  - After that first big MVP merge, additional functionality of course
>> could be developed directly against trunk.
>>  - For patches that are very clean and self contained, for example in
>> their own Java package, it doesn't really matter, because they are easy to
>> roll back if needed. They can be developed against trunk.
>>
>> So applying this to Josh's examples:
>>
>> 1) I assume JDK17 support is invasive, so that would suggest a feature
>> branch. However, the next question is, is there any risk involved in this
>> work (like Falcon for MySQL). Hypothetically it could be that Java 17 has
>> worse performance than Java 11, or some other blocking problem is
>> encountered. But in practice we probably estimate that this risk is small.
>> In such a case JDK17 support could indeed be developed with small patches
>> directly against trunk, but this would be an exception to the rule!
>>
>> 2) To take an example of an approved CEP, surprisingly enough, the
>> humongous Accord patch is actually very clean and self contained, and would
>> be easy to remove (or disable with a feature flag, which has been done). So
>> it could have been developed against trunk. (But I'm not sure that was
>> obvious in the beginning of development?)
>>
>> 3) The work on SSTable tries and Memtable tries was even explicitly split
>> into separate CEPs for the API refactor and the new functionality.
>>
>>
>> Perhaps Linus Torvalds said the above more succintly than me:
>>
>>
>>
>> *So the name of the game is to _avoid_ decisions, at least the big and
>> painful ones. Making small and non-consequential decisions is fine, and
>> makes you look like you know what you're doing, so what a kernel manager
>> needs to do is to turn the big and painful ones into small things where
>> nobody really cares.It helps to realize that the key difference between a
>> big decision and a small one is whether you can fix your decision
>> afterwards. Any decision can be made small by just always making sure that
>> if you were wrong (and you _will_ be wrong), you can always undo the damage
>> later by backtracking. Suddenly, you get to be doubly managerial for making
>> _two_ inconsequential decisions - the wrong one _and_ the right one.*
>>
>>
>>
>> https://www.openlife.cc/onlinebook/epilogue-linux-kernel-management-style-linus-torvalds
>>
>>
>> (I particularly like the last sentence!)
>>
>> henrik
>>
>>
>> On Fri, Feb 3, 2023 at 2:06 PM Josh McKenzie 
>> wrote:
>>
>>
>> The topic of how we handle merging large complex bodies of work came up
>> recently with the CEP-15 merge and JDK17, and we've faced this question in
>> the past as well (CASSANDRA-8099 comes to mind).
>>
>> The times we've done large bodies of work separately from trunk and then
>> merged them in have their own benefits and costs, and the examples I can
>> think of where we've merged in work to trunk incrementally with something
>> flagged experimental have markedly different cost/benefits. Further, the
>> two approaches have shaped the *way* we approached work quite
>> differently with how we architected and tested things.
>>
>> My current thinking: I'd like to propose we all agree to move to merge
>> work into trunk incrementally if it's either:
>> 1) New JDK support
>> 2) An approved CEP
>>
>> The bar for merging anything into trunk should remain:
>> 1) 2 +1's from committers
>> 2) Green CI (presently circle or ASF, in the future ideally ASF or an ASF
>> analog env)
>>
>> I don't know if this is a generally held opinion and we just haven't
>> discussed it and switched our general behavior yet, or if this is more
>> controversial, so I won't burden this email with enumerating pros and cons
>> of the two approaches until I get a gauge of the community's temperature.
>>
>> So - what do we think?
>>
>>
>>
>> --
>>
>>
>>
>> *Henrik Ingo*
>>
>> *c*. +358 40 569 7354
>>
>> *w*. *www.datastax.com <http://www.datastax.com>*
>>
>> <https://www.facebook.com/datastax>  <https://twitter.com/datastax>
>> <https://www.linkedin.com/company/datastax/>
>> <https://github.com/datastax/>
>>
>>
>>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Welcome Patrick McFadin as Cassandra Committer

[Derek Chen-Becker]

Congrats!

On Thu, Feb 2, 2023 at 10:58 AM Benjamin Lerer  wrote:

> The PMC members are pleased to announce that Patrick McFadin has accepted
> the invitation to become committer today.
>
> Thanks a lot, Patrick, for everything you have done for this project and
> its community through the years.
>
> Congratulations and welcome!
>
> The Apache Cassandra PMC members
>


-- 
+-------+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Intra-project dependencies

[Derek Chen-Becker]

Actually, re-reading the thread, I think I missed the initial point
brought up and got lost in the discussion specific to submodules. What
is the technical reason for bringing Accord in-tree? While I think
submodules are the best way to include source in-tree, I'm not sure
this is actually the correct thing to do in this case. Don't we
already have mechanisms to deal with snapshot versions of library
dependencies in the build? Do we need release votes for snapshots?

Thanks,

Derek

On Tue, Jan 17, 2023 at 9:25 AM Derek Chen-Becker  wrote:
>
> I'd like to go back to Benedict's initial point: if we have a new
> consensus protocol that other projects would potentially be interested
> in, then by all means it should be its own project. Let's start with
> that as a basis for discussion, because from my reading it seems like
> people might be disagreeing with that initial premise.
>
> If we agree that Accord should be independent, I'm +1 for git
> submodules primarily because that's a standard way of doing things and
> I don't think we need yet another bespoke solution to a problem that
> hundreds, if not thousands of other software projects encounter. I've
> worked with lots of projects using submodules and while they're not a
> panacea, they've never been a significant problem to work with.
>
> It's also a little confusing to see people argue about HEAD in
> relation to any of this, since that's just an alias to the latest
> commit for a given branch. In every project I've worked with that uses
> submodules you would never use HEAD, because the submodule itself
> already records the *exact* commit associated with the parent.
>
> Cheers,
>
> Derek
>
> On Tue, Jan 17, 2023 at 2:28 AM Benedict  wrote:
> >
> > The answer to all your questions is “like any other library” - this is a 
> > procedural hack to ease development. There are alternative isomorphic 
> > hacks, like compiling source jars from Accord and including them in the C* 
> > tree, if it helps your mental model.
> >
> > > you stated that a goal was to avoid maintaining multiple branches.
> >
> > No, I stated that a goal was to *decouple* development of Accord from C*. I 
> > don’t see why you would take that to mean there are no branches of Accord, 
> > as that would quite clearly be incompatible with the C* release strategy.
> >
> >
> >
> > On 17 Jan 2023, at 07:36, Mick Semb Wever  wrote:
> >
> > 
> >>
> >> … extrapolating this experience to multiple C* versions
> >
> >
> > To include forward-merging, bisecting old history, etc etc. that's a leap 
> > of faith that I believe deserves the discussion.
> >
> >> - patches are off submodule SHAs, not the submodule's HEAD,
> >>
> >>
> >> A SHA would point to the HEAD of a given branch, at the time of merge, 
> >> just by SHA? I’ve no idea what you imagine here, but this just ensures 
> >> that a given SHA of the importing project continues to compile correctly 
> >> when it is no longer HEAD. It does not mean there’s no HEAD that 
> >> corresponds directly to the SHA of the importing project’s HEAD.
> >
> >
> >
> > That wasn't my concern. Rather that you need to know in advance when the 
> > SHA is not HEAD. You can't commit off a past SHA. Once you find out (and 
> > how does this happen?) that the submodule code is not HEAD what do you then 
> > do? What if fast-forwarding the submodule to HEAD's SHA breaks things, do 
> > you now have to fix that or introduce branching in the submodule? If the 
> > submodule doesn't have releases, is it doing versioning, and if not how are 
> > branches distinguished?
> >
> > Arn't these all fair enquiries to raise?
> >
> >> - you need to be making commits to all branches (and forward merging) 
> >> anyway to update submodule SHAs,
> >>
> >>
> >> Exactly as you would any library upgrade?
> >
> >
> >
> > Correct. submodules does not solve/remove the need to commit to multiple 
> > branches and forward merge.
> > Furthermore submodules means at least one additional commit, and possibly 
> > twice as many commits.
> >
> >
> >> - if development is active on trunk, and then you need an update on an 
> >> older branch, you have to accommodate to backporting all those trunk 
> >> changes (or introduce the same branching in the submodule),
> >>
> >>
> >> If you do feature development against Accord then you will obviously 
> >> branch it? You would only make bug fixes to a bug fix branch. I’m not sure 
> >> what you think is wrong 

Re: Intra-project dependencies

[Derek Chen-Becker]

I'd like to go back to Benedict's initial point: if we have a new
consensus protocol that other projects would potentially be interested
in, then by all means it should be its own project. Let's start with
that as a basis for discussion, because from my reading it seems like
people might be disagreeing with that initial premise.

If we agree that Accord should be independent, I'm +1 for git
submodules primarily because that's a standard way of doing things and
I don't think we need yet another bespoke solution to a problem that
hundreds, if not thousands of other software projects encounter. I've
worked with lots of projects using submodules and while they're not a
panacea, they've never been a significant problem to work with.

It's also a little confusing to see people argue about HEAD in
relation to any of this, since that's just an alias to the latest
commit for a given branch. In every project I've worked with that uses
submodules you would never use HEAD, because the submodule itself
already records the *exact* commit associated with the parent.

Cheers,

Derek

On Tue, Jan 17, 2023 at 2:28 AM Benedict  wrote:
>
> The answer to all your questions is “like any other library” - this is a 
> procedural hack to ease development. There are alternative isomorphic hacks, 
> like compiling source jars from Accord and including them in the C* tree, if 
> it helps your mental model.
>
> > you stated that a goal was to avoid maintaining multiple branches.
>
> No, I stated that a goal was to *decouple* development of Accord from C*. I 
> don’t see why you would take that to mean there are no branches of Accord, as 
> that would quite clearly be incompatible with the C* release strategy.
>
>
>
> On 17 Jan 2023, at 07:36, Mick Semb Wever  wrote:
>
> 
>>
>> … extrapolating this experience to multiple C* versions
>
>
> To include forward-merging, bisecting old history, etc etc. that's a leap of 
> faith that I believe deserves the discussion.
>
>> - patches are off submodule SHAs, not the submodule's HEAD,
>>
>>
>> A SHA would point to the HEAD of a given branch, at the time of merge, just 
>> by SHA? I’ve no idea what you imagine here, but this just ensures that a 
>> given SHA of the importing project continues to compile correctly when it is 
>> no longer HEAD. It does not mean there’s no HEAD that corresponds directly 
>> to the SHA of the importing project’s HEAD.
>
>
>
> That wasn't my concern. Rather that you need to know in advance when the SHA 
> is not HEAD. You can't commit off a past SHA. Once you find out (and how does 
> this happen?) that the submodule code is not HEAD what do you then do? What 
> if fast-forwarding the submodule to HEAD's SHA breaks things, do you now have 
> to fix that or introduce branching in the submodule? If the submodule doesn't 
> have releases, is it doing versioning, and if not how are branches 
> distinguished?
>
> Arn't these all fair enquiries to raise?
>
>> - you need to be making commits to all branches (and forward merging) anyway 
>> to update submodule SHAs,
>>
>>
>> Exactly as you would any library upgrade?
>
>
>
> Correct. submodules does not solve/remove the need to commit to multiple 
> branches and forward merge.
> Furthermore submodules means at least one additional commit, and possibly 
> twice as many commits.
>
>
>> - if development is active on trunk, and then you need an update on an older 
>> branch, you have to accommodate to backporting all those trunk changes (or 
>> introduce the same branching in the submodule),
>>
>>
>> If you do feature development against Accord then you will obviously branch 
>> it? You would only make bug fixes to a bug fix branch. I’m not sure what you 
>> think is wrong here.
>
>
>
> That's not obvious, you stated that a goal was to avoid maintaining multiple 
> branches. Sure there's benefits to a lazy branching approach, but it 
> contradicts your initial motivations and introduces methodology changes that 
> are worth pointing out. What happens when there are multiple consumers of 
> Accord, and (like the situation we face with jamm) its HEAD is well in front 
> of anything C* is using.
>
> As Henrik states, the underlying problem doesn't change, we're just choosing 
> between trade-offs. My concern is that we're not even doing a very good job 
> of choosing between the trade-offs. Based on past experiences with 
> submodules: that started with great excitement and led to tears and 
> frustration after a few years; I'm only pushing for a more thorough 
> discussion and proposal.
>
>
>
>


-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSSION] Cassandra's code style and source code analysis

[Derek Chen-Becker]

I vote for #4. I've always used the convention of having stdlib stuff
first, external stuff second, and same-project imports last. I guess
increasing order of specificity?

Happy Holidays!

Derek

On Thu, Dec 22, 2022 at 7:52 AM Maxim Muzafarov  wrote:
>
> Hello everyone, have a great vacation and happy holidays to all!
>
>
> I've completed a small research about how the classe's import order
> rule are spread in the Apache projects. Some of the projects don't
> have any restrictions over the imports even if they are using the
> checkstyle configuration. The other ones may have only the consensus
> over the imports, but they are not reflected in the checkstyle yet
> (e.g. Kafka). The conclusion here can only be that there is a very
> large variability in the classe's import order, so we have to agree on
> the order on our own.
>
> You can find the projects, IDEs and frameworks and their corresponding
> classe's import order below:
> https://mmuzaf.github.io/blog/Java_Import_Orders.html
>
>
> Most of the time during development in an IDE the classe's imports
> remains collapsed, so from my point of view the following things
> related to the classe's import comes into the first place to consider:
>
> - a PR review: newly imports must be clearly visible;
> - try to minimize the total amount of affected files;
> - the import order rule must be implemented in a simple way and well
> supported by IDEs and its plugins;
>
> In addition to the last mentioned option, the checkstyle itself has
> some limitations also. For instance, the ImportOrder has a limitation
> by design to enforce an empty line between groups ("java", "javax"),
> or CustomImportOrder may have only up to 4 custom groups separated by
> a blank line.
>
>
>
> Based on all of the above I can propose the following classe's order.
> All of them are tested on the latest changes from the trunk branch
> (commit hash: b171b4ba294126e985d0ee629744516f19c8644e)
>
>
> 1. Total 2 groups, 3072 files to change
>
> ```
> all other imports
> [blank line]
> static all other imports
> ```
>
> 2. Total 3 groups, 2345 files to change
>
> ```
> java.*
> javax.*
> [blank line]
> all other imports
> [blank line]
> static all other imports
> ```
>
> 3. Total 5 groups, 2968 files to change
>
> ```
> org.apache.cassandra.*
> [blank line]
> java.*
> [blank line]
> javax.*
> [blank line]
> all other imports
> [blank line]
> static all other imports
> ```
>
> 4. Total 5 groups, 1792 files to change
>
> ```
> java.*
> javax.*
> [blank line]
> com.*
> net.*
> org.*
> [blank line]
> org.apache.cassandra.*
> [blank line]
> all other imports
> [blank line]
> static all other imports
> ```
>
> 5. Total 2 groups, 3114 files to change
>
> ```
> java.*
> javax.*
> org.apache.cassandra.*
> all other imports
> [blank line]
> static all other imports
> ```
>
>
> Of course, any suggestions are really appreciated.
> Please, share your thoughts.
>
> On Thu, 15 Dec 2022 at 17:48, Mick Semb Wever  wrote:
> >>
> >> Another angle I forgot to mention is that this is quite a big patch and 
> >> there are quite big pieces of work coming, being it CEP-15, for example. 
> >> So I am trying to figure out if we are ok to just merge this work first 
> >> and devs doing CEP-15 will need to rework their imports or we merge this 
> >> after them so we will fix their stuff. I do not know what is more 
> >> preferable.
> >
> >
> >
> > Thank you for bringing this point up Stefan.
> >
> > I would be actively reaching out to all those engaged with current CEPs, 
> > asking them the rebase impact this would cause and if they are ok with it. 
> > The CEPs are our priority, and we have a significant amount of them in 
> > progress compared to anything we've had for many years.
> >
> >
> >



-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSSION] If we fix code that used default encoding to now be UTF-8... is this a regression?

[Derek Chen-Becker]

As an initial step, could we introduce some sort of log warning,
metric or other indicator for operators to determine if they're
running with a non-UTF-8 encoding?

On Mon, Nov 28, 2022 at 1:21 PM David Capwell  wrote:
>
> It probably has to be done on a  case-by-case basis
>
>
> Yeah, this is what I feel as well…
>
> Does the linter provide more detail than just the list?
>
>
> Not really, it shows how to fix but can’t really say if the fix will cause 
> issues… If you are not running with UTF-8 we do the right thing most of the 
> time, but some files “may” break… this would also be true if you 
> backup/restore these files on a different environment...
>
>
> On Nov 10, 2022, at 12:44 PM, Derek Chen-Becker  wrote:
>
> This seems fraught with peril. I think that it should be fixed, but I
> also wonder what the testing requirements would be to validate no
> regression. It probably has to be done on a  case-by-case basis. Is it
> as simple as auditing places where we're calling getBytes or
> PrintReader/PrintWriter without an explicit encoding? Some of them,
> like 
> https://github.com/apache/cassandra/blob/30ad754d7e95501ffa916bf986e4cfda1aa5e441/src/java/org/apache/cassandra/tools/HashPassword.java#L128,
> look like that would be easy to address, but others seem like they
> could be complicated.
>
> Does the linter provide more detail than just the list?
>
> Cheers,
>
> Derek
>
> On Fri, Nov 4, 2022 at 2:09 PM David Capwell  wrote:
>
>
> Testing out linter trying to see if it can solve a case for Simulator and see 
> we have 25 cases where we don’t add the encoding and rely on default, which 
> is based off the system…
>
> If we attempt to fix these cases, I am wondering if this is a regression… it 
> “might” be the case someone set -Dfile.encoding=ascii or updated env LANG to 
> something non-UTF based…
>
> Here is the list reported
>
> org.apache.cassandra.cql3.functions.JavaBasedUDFunction since first 
> historized release
> org.apache.cassandra.db.ColumnFamilyStore since first historized release
> org.apache.cassandra.db.compaction.CompactionLogger$CompactionLogSerializer 
> since first historized release
> org.apache.cassandra.db.filter.RowFilter$CustomExpression since first 
> historized release
> org.apache.cassandra.db.lifecycle.LogTransaction since first historized 
> release
> org.apache.cassandra.gms.FailureDetector since first historized release
> org.apache.cassandra.index.sasi.analyzer.StandardTokenizerImpl since first 
> historized release
> org.apache.cassandra.io.sstable.SSTable since first historized release
> org.apache.cassandra.io.util.FileReader since first historized release
> org.apache.cassandra.io.util.FileReader since first historized release
> org.apache.cassandra.io.util.FileWriter since first historized release
> org.apache.cassandra.io.util.FileWriter since first historized release
> org.apache.cassandra.metrics.SamplingManager since first historized release
> org.apache.cassandra.metrics.SamplingManager since first historized release
> org.apache.cassandra.schema.IndexMetadata since first historized release
> org.apache.cassandra.security.PEMBasedSslContextFactory since first 
> historized release
> org.apache.cassandra.tools.HashPassword since first historized release
> org.apache.cassandra.tools.JMXTool$Dump$Format$3 since first historized 
> release
> org.apache.cassandra.tools.NodeTool$NodeToolCmd since first historized release
> org.apache.cassandra.tools.SSTableMetadataViewer since first historized 
> release
> org.apache.cassandra.transport.Client since first historized release
> org.apache.cassandra.utils.ByteArrayUtil since first historized release
> org.apache.cassandra.utils.FBUtilities since first historized release
> org.apache.cassandra.utils.GuidGenerator since first historized release
> org.apache.cassandra.utils.HeapUtils since first historized release
>
>
>
> --
> +---+
> | Derek Chen-Becker |
> | GPG Key available at https://keybase.io/dchenbecker and   |
> | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
> | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
> +---+
>
>


-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSS] Adding dependency to the Big-Math library from eobermuhlner

[Derek Chen-Becker]

Overall the library appears to be high quality, even going so far as
to have regression tests between versions. I do, however, think that
the long-term maintenance risk needs to be acknowledged here. I think
the *absolute* worst case scenario would be corruption or deletion of
source and artifacts, requiring the Cassandra community to either
re-implement the functions or remove them from CQL support. More
likely (having seen it happen once or twice) would be abandonment by
the author, requiring a fork and maintenance along with some
dependency modification (assuming the fork would have to be published
under a different group/artifact ID). I'm +1 for the addition of the
dependency on the basis of the apparent stability of the project, and
with the understanding that the functionality offered here is
ancillary to core Cassandra functionality.

Cheers,

Derek

On Mon, Nov 28, 2022 at 5:30 AM Josh McKenzie  wrote:
>
> I'm pleased with the rigor he shows on his explanations of implementation and 
> performance: http://obermuhlner.ch/wordpress/2016/06/02/bigdecimalmath/
>
> Seems like it's probably stable given the infrequency of changes to it and 
> he's still actively merging patches submit by others: 
> https://github.com/eobermuhlner/big-math/commits/master as of 8 days ago. 
> Only 4 issues open on the repo at this time as well for a reasonably starred 
> / forked library.
>
> I guess my one concern: this appears to be a library maintained primarily by 
> 1 person; that's a worst-case bus factor. Should he abandon the project is it 
> something we'd plan to fork and bring into tree and maintain ourselves? Given 
> how mature and stable it is I wouldn't be too worried, but worth considering 
> the worst-case.
>
>
> On Mon, Nov 28, 2022, at 3:48 AM, Benjamin Lerer wrote:
>
> Hi everybody,
>
> I wanted to discuss the addition of the Big-Math 
> library(http://eobermuhlner.github.io/big-math/)  as a dependency by  
> CASSANDRA-17221 which add support for abs, exp, log, log10, and round Math 
> function. The library was added for providing those functions for the 
> Cassandra decimal type (java BigDecimal).
>
> This patch has been started a long time ago and went through multiple rounds 
> of reviews and rework. In my enthusiasm to finally commit this patch I forgot 
> to raise the discussion to the mailing list about the dependency. I apologize 
> for that.
>
> Does anybody have some concerns with the addition of that Library as a 
> dependency?
>
>


-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Should we change 4.1 to G1 and offheap_objects ?

[Derek Chen-Becker]

On Thu, Nov 17, 2022 at 2:01 PM Josh McKenzie  wrote:
> 3) Expert: Leave me alone. I tune my own GC

This is increasingly not a thing. I haven't looked at ZGC, but G1 and
Shenandoah provide a lot of knobs...that the collector will happily
ignore if it decides it knows better :)

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Should we change 4.1 to G1 and offheap_objects ?

[Derek Chen-Becker]

I wouldn't recommend Shenandoah or ZGC, period. They're not designed
for the kind of workload you'll typically see running a database (high
throughput of objects that don't tenure) and both will fall over in
interesting ways under high allocation rate. GenShen is intended to
combine the generational goodness of G1 with the concurrent collection
of Shenandoah, and will likely perform better than G1 in terms of
pause time and better than Shenandoah for allocation rate and heap
utilization. GenShen, however, is experimental. Right now I would say
G1 is the best collector generally available.

In terms of providing data (beyond anecdotes), do we even agree on
what the baseline load test looks like? Are we going off of something
that's in dtest, or do we have a defined benchmarking suite somewhere?

Cheers,

Derek

On Thu, Nov 17, 2022 at 1:47 PM J. D. Jordan  wrote:
>
> -1 on providing a bunch of choices and forcing users to pick one. We should 
> have a default and it should be “good enough” for most people. The people who 
> want to dig in and try other gc settings can still do it, and we could 
> provide them some profiles to start from, but there needs to be a default.  
> We need to be asking new operators less questions on install, not more.
>
> Re:experience with Shenandoah under high load, I have in the past seen the 
> exact same thing for both Shenandoah and ZGC. Both of them have issues at 
> high loads while performing great at moderate loads. I have not seen G1 ever 
> have such issues. So I would not be fine with a switch to Shenandoah or ZGC 
> as the default without extensive testing on current JVM versions that have 
> hopefully improved the behavior under load.
>
> > On Nov 17, 2022, at 9:39 AM, Joseph Lynch  wrote:
> > It seems like this is a choice most users might not know how to make?
> >
> > On Thu, Nov 17, 2022 at 7:06 AM Josh McKenzie  wrote:
> >>
> >> Have we ever discussed including multiple profiles that are simple to swap 
> >> between and documented for their tested / intended use cases?
> >>
> >> Then the burden of having a “sane” default for the wild variance of 
> >> workloads people use it for would be somewhat mitigated. Sure, there’s 
> >> always going to be folks that run the default and never think to change it 
> >> but the UX could be as simple as a one line config change to swap between 
> >> GC profiles and we could add and deprecate / remove over time.
> >>
> >> Concretely, having config files such as:
> >>
> >> jvm11-CMS-write.options
> >> jvm11-CMS-mixed.options
> >> jvm11-CMS-read.options
> >> jvm11-G1.options
> >> jvm11-ZGC.options
> >> jvm11-Shen.options
> >>
> >>
> >> Arguably we could take it a step further and not actually allow a C* node 
> >> to startup without pointing to one of the config files from your primary 
> >> config, and provide a clean mechanism to integrate that selection on 
> >> headless installs.
> >>
> >> Notably, this could be a terrible idea. But it does seem like we keep 
> >> butting up against the complexity and mixed pressures of having the One 
> >> True Way to GC via the default config and the lift to change that.
> >>
> >> On Wed, Nov 16, 2022, at 9:49 PM, Derek Chen-Becker wrote:
> >>
> >> I'm fine with not including G1 in 4.1, but would we consider inclusion
> >> for 4.1.X down the road once validation has been done?
> >>
> >> Derek
> >>
> >>
> >> On Wed, Nov 16, 2022 at 4:39 PM David Capwell  wrote:
> >>> Getting poked in Slack to be more explicit in this thread…
> >>> Switching to G1 on trunk, +1
> >>> Switching to G1 on 4.1, -1.  4.1 is about to be released and this isn’t a 
> >>> bug fix but a perf improvement ticket and as such should go through 
> >>> validation that the perf improvements are seen, there is not enough time 
> >>> left for that added performance work burden so strongly feel it should be 
> >>> pushed to 4.2/5.0 where it has plenty of time to be validated against.  
> >>> The ticket even asks to avoid validating the claims; saying 'Hoping we 
> >>> can skip due diligence on this ticket because the data is "in the past” 
> >>> already”'.  Others have attempted both shenandoah and ZGC and found mixed 
> >>> results, so nothing leads me to believe that won’t be true here either.
> >>>> On Nov 16, 2022, at 9:15 AM, J. D. Jordan  
> >>>> wrote:
> >>>> Heap -
> >>>> +1 for G1 in trunk
> >>>> +0 for G1 i

Re: Should we change 4.1 to G1 and offheap_objects ?

[Derek Chen-Becker]

I'm fine with not including G1 in 4.1, but would we consider inclusion
for 4.1.X down the road once validation has been done?

Derek


On Wed, Nov 16, 2022 at 4:39 PM David Capwell  wrote:
>
> Getting poked in Slack to be more explicit in this thread…
>
> Switching to G1 on trunk, +1
> Switching to G1 on 4.1, -1.  4.1 is about to be released and this isn’t a bug 
> fix but a perf improvement ticket and as such should go through validation 
> that the perf improvements are seen, there is not enough time left for that 
> added performance work burden so strongly feel it should be pushed to 4.2/5.0 
> where it has plenty of time to be validated against.  The ticket even asks to 
> avoid validating the claims; saying 'Hoping we can skip due diligence on this 
> ticket because the data is "in the past” already”'.  Others have attempted 
> both shenandoah and ZGC and found mixed results, so nothing leads me to 
> believe that won’t be true here either.
>
> > On Nov 16, 2022, at 9:15 AM, J. D. Jordan  wrote:
> >
> > Heap -
> > +1 for G1 in trunk
> > +0 for G1 in 4.1 - I think it’s worthwhile and fairly well tested but I 
> > understand pushback against changing this so late in the game.
> >
> > Memtable -
> > -1 for off heap in 4.1. I think this needs more testing and isn’t something 
> > to change at the last minute.
> > +1 for running performance/fuzz tests against the alternate memtable 
> > choices in trunk and switching if they don’t show regressions.
> >
> >> On Nov 16, 2022, at 10:48 AM, Josh McKenzie  wrote:
> >>
> >> 
> >> To clarify: -0 here on G1 as default for 4.1 as well; I'd like us to 
> >> prioritize digging into G1's behavior on small heaps vs. CMS w/our default 
> >> tuning sooner rather than later. With that info I'd likely be a strong +1 
> >> on the shift.
> >>
> >> -1 on switching to offheap_objects for 4.1 RC; again, think this is just a 
> >> small step away from being a +1 w/some more rigor around seeing the 
> >> current state of the technology's intersections.
> >>
> >> On Wed, Nov 16, 2022, at 7:47 AM, Aleksey Yeshchenko wrote:
> >>> All right. I’ll clarify then.
> >>>
> >>> -0 on switching the default to G1 *this late* just before RC1.
> >>> -1 on switching the default offheap_objects *for 4.1 RC1*, but all for it 
> >>> in principle, for 4.2, after we run some more test and resolve the 
> >>> concerns raised by Jeff.
> >>>
> >>> Let’s please try to avoid this kind of super late defaults switch going 
> >>> forward?
> >>>
> >>> —
> >>> AY
> >>>
> >>> > On 16 Nov 2022, at 03:27, Derek Chen-Becker  
> >>> > wrote:
> >>> >
> >>> > For the record, I'm +100 on G1. Take it with whatever sized grain of
> >>> > salt you think appropriate for a relative newcomer to the list, but
> >>> > I've spent my last 7-8 years dealing with the intersection of
> >>> > high-throughput, low latency systems and their interaction with GC and
> >>> > in my personal experience G1 outperforms CMS in all cases and with
> >>> > significantly less work (zero work, in many cases). The only things
> >>> > I've seen perform better *with a similar heap footprint* are GenShen
> >>> > (currently experimental) and Rust (beyond the scope of this topic).
> >>> >
> >>> > Derek
> >>> >
> >>> > On Tue, Nov 15, 2022 at 4:51 PM Jon Haddad  
> >>> > wrote:
> >>> >>
> >>> >> I'm curious what it would take for folks to be OK with merging this 
> >>> >> into 4.1?  How much additional time would you want to feel comfortable?
> >>> >>
> >>> >> I should probably have been a little more vigorous in my +1 of Mick's 
> >>> >> PR.  For a little background - I worked on several hundred clusters 
> >>> >> while at TLP, mostly dealing with stability and performance issues.  A 
> >>> >> lot of them stemmed partially or wholly from the GC settings we ship 
> >>> >> in the project. Par New with CMS and small new gen results in a lot of 
> >>> >> premature promotion leading to high pause times into the hundreds of 
> >>> >> ms which pushes p99 latency through the roof.
> >>> >>
> >>> >> I'm a big +1 in favor of G1 because it's not just better for most 
> >>> >>

Re: Should we change 4.1 to G1 and offheap_objects ?

[Derek Chen-Becker]

For the record, I'm +100 on G1. Take it with whatever sized grain of
salt you think appropriate for a relative newcomer to the list, but
I've spent my last 7-8 years dealing with the intersection of
high-throughput, low latency systems and their interaction with GC and
in my personal experience G1 outperforms CMS in all cases and with
significantly less work (zero work, in many cases). The only things
I've seen perform better *with a similar heap footprint* are GenShen
(currently experimental) and Rust (beyond the scope of this topic).

Derek

On Tue, Nov 15, 2022 at 4:51 PM Jon Haddad  wrote:
>
> I'm curious what it would take for folks to be OK with merging this into 4.1? 
>  How much additional time would you want to feel comfortable?
>
> I should probably have been a little more vigorous in my +1 of Mick's PR.  
> For a little background - I worked on several hundred clusters while at TLP, 
> mostly dealing with stability and performance issues.  A lot of them stemmed 
> partially or wholly from the GC settings we ship in the project. Par New with 
> CMS and small new gen results in a lot of premature promotion leading to high 
> pause times into the hundreds of ms which pushes p99 latency through the roof.
>
> I'm a big +1 in favor of G1 because it's not just better for most people but 
> it's better for _every_ new Cassandra user.  The first experience that people 
> have with the project is important, and our current GC settings are quite bad 
> - so bad they lead to problems with stability in production.  The G1 settings 
> are mostly hands off, result in shorter pause times and are a big improvement 
> over the status quo.
>
> Most folks don't do GC tuning, they use what we supply, and what we currently 
> supply leads to a poor initial experience with the database.  I think we owe 
> the community our best effort even if it means pushing the release back 
> little bit.
>
> Just for some additional context, we're (Netflix) running 25K nodes on G1 
> across a variety of hardware in AWS with wildly varying workloads, and I 
> haven't seen G1 be the root cause of a problem even once.  The settings that 
> Mick is proposing are almost identical to what we use (we use half of heap up 
> to 30GB).
>
> I'd really appreciate it if we took a second to consider the community effect 
> of another release that ships settings that cause significant pain for our 
> users.
>
> Jon
>
> On 2022/11/10 21:49:36 Mick Semb Wever wrote:
> > >
> > > In case of GC, reasonably extensive performance testing should be the
> > > expectations. Potentially revisiting some of the G1 params for the 4.1
> > > reality - quite a lot has changed since those optional defaults where
> > > picked.
> > >
> >
> >
> > I've put our battle-tested g1 opts (from consultants at TLP and DataStax)
> > in the patch for CASSANDRA-18027
> >
> > In reality it is really not much of a change, g1 does make it simple.
> > Picking the correct ParallelGCThreads and ConcGCThreads and the floor to
> > the new heap (XX:NewSize) is still required, though we could do a much
> > better job of dynamic defaults to them.
> >
> > Alex Dejanovski's blog is a starting point:
> > https://thelastpickle.com/blog/2020/06/29/cassandra_4-0_garbage_collectors_performance_benchmarks.html
> > where this gc opt set was used (though it doesn't prove why those options
> > are chosen)
> >
> > The bar for objection to sneaking these into 4.1 was intended to be low,
> > and I stand by those that raise concerns.
> >



-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSSION] If we fix code that used default encoding to now be UTF-8... is this a regression?

[Derek Chen-Becker]

This seems fraught with peril. I think that it should be fixed, but I
also wonder what the testing requirements would be to validate no
regression. It probably has to be done on a  case-by-case basis. Is it
as simple as auditing places where we're calling getBytes or
PrintReader/PrintWriter without an explicit encoding? Some of them,
like 
https://github.com/apache/cassandra/blob/30ad754d7e95501ffa916bf986e4cfda1aa5e441/src/java/org/apache/cassandra/tools/HashPassword.java#L128,
look like that would be easy to address, but others seem like they
could be complicated.

Does the linter provide more detail than just the list?

Cheers,

Derek

On Fri, Nov 4, 2022 at 2:09 PM David Capwell  wrote:
>
> Testing out linter trying to see if it can solve a case for Simulator and see 
> we have 25 cases where we don’t add the encoding and rely on default, which 
> is based off the system…
>
> If we attempt to fix these cases, I am wondering if this is a regression… it 
> “might” be the case someone set -Dfile.encoding=ascii or updated env LANG to 
> something non-UTF based…
>
> Here is the list reported
>
> org.apache.cassandra.cql3.functions.JavaBasedUDFunction since first 
> historized release
> org.apache.cassandra.db.ColumnFamilyStore since first historized release
> org.apache.cassandra.db.compaction.CompactionLogger$CompactionLogSerializer 
> since first historized release
> org.apache.cassandra.db.filter.RowFilter$CustomExpression since first 
> historized release
> org.apache.cassandra.db.lifecycle.LogTransaction since first historized 
> release
> org.apache.cassandra.gms.FailureDetector since first historized release
> org.apache.cassandra.index.sasi.analyzer.StandardTokenizerImpl since first 
> historized release
> org.apache.cassandra.io.sstable.SSTable since first historized release
> org.apache.cassandra.io.util.FileReader since first historized release
> org.apache.cassandra.io.util.FileReader since first historized release
> org.apache.cassandra.io.util.FileWriter since first historized release
> org.apache.cassandra.io.util.FileWriter since first historized release
> org.apache.cassandra.metrics.SamplingManager since first historized release
> org.apache.cassandra.metrics.SamplingManager since first historized release
> org.apache.cassandra.schema.IndexMetadata since first historized release
> org.apache.cassandra.security.PEMBasedSslContextFactory since first 
> historized release
> org.apache.cassandra.tools.HashPassword since first historized release
> org.apache.cassandra.tools.JMXTool$Dump$Format$3 since first historized 
> release
> org.apache.cassandra.tools.NodeTool$NodeToolCmd since first historized release
> org.apache.cassandra.tools.SSTableMetadataViewer since first historized 
> release
> org.apache.cassandra.transport.Client since first historized release
> org.apache.cassandra.utils.ByteArrayUtil since first historized release
> org.apache.cassandra.utils.FBUtilities since first historized release
> org.apache.cassandra.utils.GuidGenerator since first historized release
> org.apache.cassandra.utils.HeapUtils since first historized release
>


-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Should we change 4.1 to G1 and offheap_objects ?

[Derek Chen-Becker]

There's a lot of work that's gone into G1 to the point where for
almost all workloads it will perform better than CMS. However, there
are almost no knobs to tune (most G1 params are "advisory" and G1 will
happily ignore them if it wants to), so there may not be a great
replacement if people are tuning CMS heavily for a specific workload.
Can you define "friendlier" in the context of CMS?

Derek

On Wed, Nov 9, 2022 at 3:10 PM Brandon Williams  wrote:
>
> If CMS is gone, is there a friendlier alternative to G1?
>
> On Wed, Nov 9, 2022 at 3:53 PM Josh McKenzie  wrote:
> >
> > My recollection (and brief sleuthing now) surfaces: we've gone back and 
> > forth on the G1 vs. CMS debate over the years and I think we settled on "it 
> > all depends on your environment, workload, and you need to tune it anyway. 
> > It might be worth having a 'default' mode that selects one of the two based 
> > on heap size unless otherwise specified".
> >
> > I certainly wouldn't make changes to any defaults on a release between beta 
> > and rc personally.
> >
> > On Wed, Nov 9, 2022, at 4:20 PM, Jeff Jirsa wrote:
> >
> > G1 you can argue for with the changes in the JDK, though it's MUCH  less 
> > friendly to small heaps (e.g. probably our default simple user).
> >
> > Offheap memtables are different though. If someone wants to attest that 
> > offheap_objects get the same level of rigorous testing as the existing 
> > default, that'd be useful, but I'm pretty sure that's not true, and bugs 
> > like https://issues.apache.org/jira/browse/CASSANDRA-12125  (which remains 
> > undiagnosed) reinforce that it's less commonly used and may have latent 
> > undiscovered bugs for default users.
> >
> >
> >
> >
> >
> > On Wed, Nov 9, 2022 at 11:23 AM Mick Semb Wever  wrote:
> >
> > Any objections to making these changes, at the very last minute, for 
> > 4.1-rc1 ?
> > This is CASSANDRA-12029 and CASSANDRA-7486
> >
> > Provided we figure out patches for them in the next day or two.
> >
> >



-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSSION] Add SpotBugs to the build

[Derek Chen-Becker]

"ant skip-code-checks" might be more discoverable (and less intimidating)
than "ant unsafe". I'm a +100 on enabling more static
analysis/linting/coverage reporting for full builds.

On Tue, Nov 8, 2022, 6:42 AM Josh McKenzie  wrote:

> We used to do linting years ago, and our approach was basically "the past
> is past, but look for deltas of new bad things added and notify the
> author". At least for me personally the signal / noise ratio was quite
> valuable (and not just for my own code; i.e. this wasn't *just* a "Josh
> Problem" ;) )
>
> I was thinking about this w/the checkstyle additions; we need the ability
> to bypass these easily for incremental change / test cycles while working
> on something so devs don't have to pay the "analyze the entire code-base"
> tax on each small local change. I'm a strong +1 on having these tools in
> here and on by default for build or jar targets.
>
> Maybe it'd be more friendly to have something like "ant unsafe" (or "ant
> jar-unsafe" so it's adjacent) that just does a quick jar w/out that stuff
> so it's more discoverable than multiple -D params?
>
> On Tue, Nov 8, 2022, at 5:40 AM, Miklosovic, Stefan wrote:
>
> I confused Jacoco to be code analysis tool instead of code coverage one.
> Early morning I guess :) I was thinking more about tooling like Sonar. What
> do you think about that? Any pros / cons to Spotbugs?
>
> 
> From: Miklosovic, Stefan 
> Sent: Tuesday, November 8, 2022 11:36
> To: dev@cassandra.apache.org
> Subject: Re: [DISCUSSION] Add SpotBugs to the build
>
> Hi David, all
>
> what is the position of Jacoco in Cassandra as this point? I see it in
> build.xml there are some targets and infrastructure around that.
>
> What is wrong with using Jacoco instead more heavily for static analysis?
> Why do we need to introduce this?
>
> Regards
>
> 
> From: David Capwell 
> Sent: Tuesday, November 8, 2022 0:10
> To: dev
> Subject: [DISCUSSION] Add SpotBugs to the build
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
>
> I was thinking that it would be good to add SpotBugs (
> https://spotbugs.github.io) into our build to help find bugs earlier in
> the life cycle.  SpotBugs is LGPL but as it is used only in the build and
> not to within this project, then this should be fine with Apache.
>
> The motivation for adding this was from CASSANDRA-17178; the Simulator has
> issues with Serializable classes missing serialVersionUID (as we deal with
> ClassLoaders; this field is strongly recommend in general for all
> Serializable classes), but this project can add more value as there are a
> large collection of potential bugs to look out for; below are a few
> examples found.
>
> * Number.valueOf vs Number.parse.  In many parts of the code we do
> valueOf which returns a boxed value; we then unbox for the usage; this adds
> more garbage that isn’t needed
> * Using Number.compareTo rather than primitive compare functions (causing
> boxing)
> * Ignoring return value for functions that don’t have a side effect.  This
> happens in a few cases where we are building a StringBuilder where we call
> .toString but ignore the string… then call it later on
> * use of putIfAbsent without looking at the return.  This was found in
> CacheService where we add the SSTable reader to the cache and assume we win
> the race and start using it… rather than using the object that won the race
>
>
>

Re: [DISCUSSION] Add SpotBugs to the build

[Derek Chen-Becker]

I'm always in favor of having the compiler/runtime do more work for
us, but I guess in the interest of gauging impact to dev productivity,
does this add much overhead? I guess we'll need to discuss what it
finds after it runs, as well.

Cheers,

Derek


On Mon, Nov 7, 2022 at 4:10 PM David Capwell  wrote:
>
> I was thinking that it would be good to add SpotBugs 
> (https://spotbugs.github.io) into our build to help find bugs earlier in the 
> life cycle.  SpotBugs is LGPL but as it is used only in the build and not to 
> within this project, then this should be fine with Apache.
>
> The motivation for adding this was from CASSANDRA-17178; the Simulator has 
> issues with Serializable classes missing serialVersionUID (as we deal with 
> ClassLoaders; this field is strongly recommend in general for all 
> Serializable classes), but this project can add more value as there are a 
> large collection of potential bugs to look out for; below are a few examples 
> found.
>
> * Number.valueOf vs Number.parse.  In many parts of the code we do 
> valueOf which returns a boxed value; we then unbox for the usage; this adds 
> more garbage that isn’t needed
> * Using Number.compareTo rather than primitive compare functions (causing 
> boxing)
> * Ignoring return value for functions that don’t have a side effect.  This 
> happens in a few cases where we are building a StringBuilder where we call 
> .toString but ignore the string… then call it later on
> * use of putIfAbsent without looking at the return.  This was found in 
> CacheService where we add the SSTable reader to the cache and assume we win 
> the race and start using it… rather than using the object that won the race



--
+-------+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: A proposal for refactoring the CircleCI config

[Derek Chen-Becker]

> For the parallel param logic, sounds fine to me.  Not sure if this would also 
> work for resource_type, but I still argue that xlarge isn’t needed in 90% of 
> the cases its used… so fixing this may be better than param there…. So yes, I 
> would be cool with this change if it basically removes the patching logic… I 
> had another JIRA to have a python script rewrite the YAML, but this method 
> may solve in a cleaner way.

Almost any part of a CircleCI definition can be replaced with a
parameter, so basically we want config-2_1.yml to be a template, and
we plug different values in as desired. Would you mind sending a link
to that JIRA so I can understand that use case?

> About matrix jobs; I don’t know them in circle but have used in other places, 
> this sounds good to me.  I would also enhance to argue that JVM is just 1 
> config and we sadly have many more:
>
> JVM: [8, 11, 17]
> VNODE: [true, false]
> CDC: [true, false]
> COMPRESSION: [true, false]
> MEMTABLE: [skiplist, shardedskiplist, trie]

My understanding is that we could parameterize all of these such that
we could use a matrix as long as all combinations are valid. Let me
get parameterization of basic configuration reviewed first, and then
we can take a look at how to matricize things.

Cheers,

Derek

-- 
+-------+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

A proposal for refactoring the CircleCI config

[Derek Chen-Becker]

ed.
  One possible workaround, if we do not want to require a minimum
  circleci CLI version, would be to define the parameters in their own
  file and concatenate the level-specific definition file with the
  config-2_1.yml file prior to processing.

  Either approach (yaml file override or concatenation) provides benefit
  since there would be no need to modify the level-specific file as the
  main configuration is changed, unlike with patch files.

  The changes needed to implement this are relatively small, and are
  easy to test, since a change to use pipeline parameters (along with
  requisite changes to `generate.sh') should not result in any changes
  to the config.yml files.


Other benefits for future investigation
═══

  Beyond simple replacement of the patch files, there are some
  additional capabilities in CircleCI configuration that may benefit the
  project long-term. One of the most interesting is the concept of
  matrix jobs. Matrix jobs allow you to parameterize a job over multiple
  values. In our case, this could potentially allow us to parameterize
  jobs over different version of the JDK. This would reduce the overall
  size of the configuration, because the current approach duplicates
  jobs for each Java version. It would also make it simpler to add/test
  new Java versions by making a relatively small number of changes to
  the matrix instead of creating a whole set of version-specific jobs.
  We would still need to create version-specific executors, but if we
  move things like parallelism and resource class out of the executor
  and into the job definition, we can parameterize in the workflow and
  reduce the number of executors to one per Java version. For example:

  ┌
  │ jobs:
  │   j8_unit_tests:
  │ parameters:
  │   executor:
  │ type: executor
  │ executor: << parameters.executor >>
  │ parallelism: << pipeline.parameters.unit_test_parallelism >>
  │ resource_class: << pipeline.parameters.unit_test_resclass >>
  │ steps:
  │   - attach_workspace:
  │   at: /home/cassandra
  │   - create_junit_containers
  │   - log_environment
  │   - run_parallel_junit_tests
  │
  │ workflows:
  │   pre-commit_tests:
  │ jobs:
  │   - j8_unit_tests:
  │   requires:
  │ - build
  │   matrix:
  │ parameters:
  │   executor: [java8-executor, java11-executor, java17-executor]
  └

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Using bash instead of sh in generate.sh?

[Derek Chen-Becker]

I don't think this quite rises to the level of a CEP, but I wanted to get
input on this. I'm working on the CircleCI build tooling and the
generate.sh script uses "/bin/sh" for its execution. There are a couple of
things that could be cleaned up and simplified if we used bash instead. I'm
all for making sure that tooling is portable, but I think that in this case
it's probably safe to assume bash is available for dev work. For example,
all of the shell scripts in the cassandra-builds repo use bash, not sh.

Thanks,

Derek

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSS] Potential circleci config and workflow changes

[Derek Chen-Becker]

I'm writing up a more complete proposal, but here are some examples.
Parameters can be set from either the UI (not my intent) or via the
circleci CLI. Effectively, the config-2_1.yml can have parameters specified
like:

parameters:
  run_extra_test:
type: boolean
default: false


jobs:
  extra_test:
steps:
  when:
condition: << pipeline.parameters.run_extra_test >>
steps:
  - run : # something
  - run : # something

And then jobs can conditionally execute or not. The "when" clause can also
be applied at the workflows level. When you generate the configuration, you
pass a 2nd yaml file (or text) to the circleci CLI and it will override any
parameters with what you provide. For example, if I wanted to disable the
test:

circleci config process --pipeline-parameters "run_extra_test: false"
config-2_1.yml

The tradeoff is that the config will be more verbose with conditionals. My
first proposal is actually to just use parameters to get rid of the patch
files, since the current patch files simply change values. In this case,
parameters allow us to use the config yaml like a template, which is less
error prone than patch files. However, things like conditional execution
and matrix jobs might allow us to streamline the config, especially as we
support more JDK versions.

Cheers,

Derek

On Tue, Oct 25, 2022 at 10:40 AM David Capwell  wrote:

> This could also be a pipeline parameter instead of hacking it in
> generate.sh
>
>
> Curious how this works… I run a script that deletes all the approvals and
> removes the testing workflows… I really don’t want to use the UI at all….
> I assumed pipeline params are a UI thing, but I think the goal here for
> many of us are to ignore the UI other than looking at the results… and even
> that can be scripted...
>
>
> On Oct 24, 2022, at 4:44 PM, Derek Chen-Becker 
> wrote:
>
> This could also be a pipeline parameter instead of hacking it in
> generate.sh. I promise I'll have a proposal before the end of the week.
>
> Derek
>
> On Mon, Oct 24, 2022 at 2:13 PM Josh McKenzie 
> wrote:
>
>> @Ekaterina: I recall us going back and forth on whether default should be
>> require approval or not and there not being a consensus. I'm fine not
>> changing the status quo and just parameterizing that in generate.sh so
>> folks can locally script how they want to setup when they alias up
>> generate.sh.
>>
>> I'll add C-17113 to the epic as well and any other tickets anyone has in
>> flight we can link up.
>>
>> Maybe we should remove them from the workflow when the free option is used
>>
>> That'd put us in the position of having a "smoke testing suite" for free
>> tier users and the expectation of a committer running the full suite
>> pre-merge. Which, now that I type it out, is a lot more representative of
>> our current reality so we should probably do that.
>>
>> Noted re: the -f flag; I could have checked that but just hacked that out
>> in the email spur of the moment. We could just default to low / free /
>> smoke test and have -p for paid tier.
>>
>>
>> On Mon, Oct 24, 2022, at 3:23 PM, Andrés de la Peña wrote:
>>
>> - Ticket for: remove -h, have -f and -p (free and paid)
>>
>>
>> +1 to this, probably there isn't anyone using -h. There are some jobs
>> that can't pass with the free option. Maybe we should remove them from the
>> workflow when the free option is used. Perhaps that could save new
>> contributors some confusion. Or should we leave them because a subset of
>> the tests inside those jobs can still pass even with the free tier?
>>
>> By the way, the generate.sh script already accepts a -f flag. It's used
>> to stop checking that the specified environment variables are known. It was
>> meant to be a kind of general "--force" flag.
>>
>> On Mon, 24 Oct 2022 at 20:07, Ekaterina Dimitrova 
>> wrote:
>>
>> Seems like my email crashed with Andres’ one.
>> My understanding is we will use the ticket CASSANDRA-17113 as
>> placeholder, the work there will be rebased/reworked etc depending on what
>> we agree with.
>> I also agree with the other points he made. Sounds reasonable to me
>>
>> On Mon, 24 Oct 2022 at 15:03, Ekaterina Dimitrova 
>> wrote:
>>
>> Thank you Josh
>>
>> So about push with/without a single click, I guess you mean to
>> parameterize whether the step build needs approval or not? Pre-commit the
>> new flag will use the “no-approval” version, but during development we
>> still will be able to push the tests without immediately starting all
>> tests, right?
>> - p

Re: [DISCUSS] Potential circleci config and workflow changes

[Derek Chen-Becker]

 if f.endswith('Test.java') and
> include(os.path.join(root, f), f):
> > num_files += 1
> > return math.floor(num_files / num_file_in_worker)
> >
> > def fix_parallelism(args, contents):
> > jobs = contents['jobs']
> >
> > unit_parallelism= java_parallelism(args.src,
> 'unit', 20)
> > jvm_dtest_parallelism   = java_parallelism(args.src,
> 'distributed', 4, lambda full, name: 'upgrade' not in full)
> > jvm_dtest_upgrade_parallelism   = java_parallelism(args.src,
> 'distributed', 2, lambda full, name: 'upgrade' in full)
> - `TL;DR - I find all test files we are going to run, and based off a
> pre-defined variable that says “idea” number of files per worker, I then
> calculate how many workers we need.  So unit tests are num_files / 20 ~= 35
> workers.  Can I be “smarter” by knowing which files have higher cost?
> Sure… but the “perfect” and the “average” are too similar that it wasn’t
> worth it...`
> - Ticket to combine pre-commit jobs into 1 pipeline for all JDK's
> - Path to activate all supported JDK's for pre-commit at root
> (one-click pre-merge full validation)
> - Path to activate per JDK below that (interim work partial validation)
> - Ticket to rename jobs in circleci
> - Reference comment:
> https://issues.apache.org/jira/browse/CASSANDRA-17939?focusedCommentId=17617016=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17617016
> - (buildjdk)_(runjdk)_(testsuite) format:
> - j8_j8_jvm_dtests
> - j8_j11_jvm_dtests
> - j11_j11_jvm_dtest_vnode
> etc
> - Ticket for flag in generate.sh to support auto run on push (see response
> above)
> - Ticket for: remove -h, have -f and -p (free and paid) (probably
> intersects with https://issues.apache.org/jira/browse/CASSANDRA-17600)
>
> Anything wrong w/the above or anything missed? If not, I'll go do some
> JIRA'ing.
>
>
> ~Josh
>
>
> On Fri, Oct 21, 2022, at 3:50 PM, Josh McKenzie wrote:
>
> I am cool with removing circle if apache CI is stable and works, we do
> need to solve the non-committer issue but would argue that partially exists
> in circle today (you can be a non-commuter with a paid account, but you
> can’t be a non-committer with a free account)
>
> There's a few threads here:
> 1. non-committers should be able to run ci
> 2. People that have resources and want to run ci faster should be able to
> do so (assuming the ci of record could serve to be faster)
> 3. ci should be stable
>
> Thus far we haven't landed on 1 system that satisfies all 3. There's some
> background discussions brainstorming how to get there; when / if things
> come from that they'll as always be brought to the list for discussion.
>
> On Fri, Oct 21, 2022, at 1:44 PM, Ekaterina Dimitrova wrote:
>
> I agree with David with one caveat - last time I checked only some Python
> tests lack enough resources with the free tier. The rest run slower than
> with a paid account, but they do fine. In fact I use the free tier if I
> want to test only unit or in-jvm tests sometimes. I guess that is what he
> meant by partially but even being able to run the non-Python tests is a win
> IMHO. If we find a solution for all tests though… even better.
> @Derek your idea sounds interesting, I will be happy to see a proposal.
> Thank you
>
> On Fri, 21 Oct 2022 at 13:39, David Capwell  wrote:
>
> I am cool with removing circle if apache CI is stable and works, we do
> need to solve the non-committer issue but would argue that partially exists
> in circle today (you can be a non-commuter with a paid account, but you
> can’t be a non-committer with a free account)
>
>
>
> On Oct 20, 2022, at 2:20 PM, Josh McKenzie  wrote:
>
> I believe it's original intention to be just about CircleCI.
>
> It was but fwiw I'm good w/us exploring adjacent things regarding CI here.
> I'm planning on deep diving on the thread tomorrow and distilling a
> snapshot of the work we have a consensus on for circle and summarizing here
> so we don't lose that. Seems like it's fairly non-controversial.
>
> On Thu, Oct 20, 2022, at 5:14 PM, Mick Semb Wever wrote:
>
>
>
> On Thu, 20 Oct 2022 at 22:07, Derek Chen-Becker 
> wrote:
>
> Would the preclusion of non-committers also prevent us from configuring
> Jenkins to auto-test on PR independent of who opens it?
>
> One of my current concerns is that we're maintaining 2x the CI for 1x the
> benefit, and I don't currently see an easy way to unify them (perhaps a
> lack of imagination?). I know there's a long history behind the choice of
> CircleCI, so I'm not trying to be hand-wavy about all of the tho

Re: Some tests are never executed in CI due to their name

[Derek Chen-Becker]

OK, I think the approach you're proposing is fine. As an orthogonal idea,
is there any way to do a one-time audit to narrow down files that we need
to inspect, or any way to automate confirmation of a file containing tests?
For example:

* Start with all of the files under the `test` directory (is it safe to
assume that this is the only location for test classes?) => 1457 files
* Remove any files that already end in "Test.java" => down to 396 files
* Is it safe to assume that only files having "Test" somewhere in their
name are test classes? That reduces the list down to 60 files

Alternatively, are we only talking about unit tests? Would it be sufficient
to look for files that don't end in "Test.java" but contain "@Test"
annotations? That's a significantly smaller set, so maybe we could fix unit
tests first:

❯❯❯ for fl in test/unit/**/*.java; do if [[ $fl != *Test.java ]]; then if
grep -qF '@Test' $fl; then print $fl; fi; fi; done

 circleci-parity ✱ ◼
test/unit/org/apache/cassandra/auth/CassandraAuthorizerTruncatingTests.java
test/unit/org/apache/cassandra/cql3/BatchTests.java
test/unit/org/apache/cassandra/cql3/KeywordTestBase.java
test/unit/org/apache/cassandra/db/guardrails/GuardrailAllowUncompressedTables.java
test/unit/org/apache/cassandra/db/guardrails/GuardrailConsistencyLevelsTester.java
test/unit/org/apache/cassandra/db/guardrails/GuardrailSecondaryIndexTester.java
test/unit/org/apache/cassandra/db/guardrails/GuardrailSecondaryIndexesPerTable.java
test/unit/org/apache/cassandra/db/guardrails/ThresholdTester.java
test/unit/org/apache/cassandra/dht/PartitionerTestCase.java
test/unit/org/apache/cassandra/utils/UUIDTests.java

Cheers,

Derek


On Mon, Oct 24, 2022 at 9:00 AM Miklosovic, Stefan <
stefan.mikloso...@netapp.com> wrote:

> One more point as I was reading your email in a hurry. The ultimate goal
> is to run all tests. So even if we apply the proposed regexp in checkstyle
> and there is an unfortunate case that MyTestSplit1 would slip through on a
> review and it would not be executed in CI, if we do not figure out some
> better regexp, we might relax "test.name" property to include Split*.java
> as well. Yes, this might be also done.
>
> Enforcing "all tests to end on "Test" " means that we would need to know
> which .java files contain tests and which not without looking into them. I
> do not think that is possible under circumstances we have where (checkstyle
> module).
>
> 
> From: Derek Chen-Becker 
> Sent: Monday, October 24, 2022 15:53
> To: dev@cassandra.apache.org
> Subject: Re: Some tests are never executed in CI due to their name
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> OK, that makes sense, and I missed the rename of "TestBaseImpl" (which I
> agree is in need of fixing). When you say:
>
> > What we are trying to achieve by that is to have a test which ends on
> "Test" and nothing else and it may contain "Test" only in case it does not
> start with it.
>
> Should the regex enforce the "which ends on Test" part? Otherwise, if we
> want to allow tests like MyTestSplit1, do we need to update the "test.name
> <http://test.name>" parameter to allow that in the test runner?
>
> Thanks,
>
> Derek
>
>
> On Mon, Oct 24, 2022 at 7:27 AM Miklosovic, Stefan <
> stefan.mikloso...@netapp.com<mailto:stefan.mikloso...@netapp.com>> wrote:
> Hi Derek,
>
> yes, the regexp is ^(?!(?:Test))(?!.*?(?:Tests)).*$
>
> What we are trying to achieve by that is to have a test which ends on
> "Test" and nothing else and it may contain "Test" only in case it does not
> start with it.
>
> To your second paragraph, I do not think we have any *Base files which are
> tests (that would be a bug) nor they are standalone (not inherited). By
> first approach, TestBaseImpl (which we do have) would start to be invalid,
> we would have to rename it to "DistributedTestBaseImpl" because it was
> starting on "Test" which is no-no. I agree that "TestBaseImpl" is rather
> unfortunate name. Currently, TestBaseImpl itself extends
> "DistributedTestBase" so in the end we would have "DistributedTestBaseImpl
> -> TestBaseImpl -> DistributedTestBase". Please keep in mind that
> "DistributedTestBase" is located in dtest jar (separate project) and that
> is rather inconvenient to rename.
>
> Regards
>
> 
> From: Derek Chen-Becker  de...@che

Re: Some tests are never executed in CI due to their name

[Derek Chen-Becker]

OK, that makes sense, and I missed the rename of "TestBaseImpl" (which I
agree is in need of fixing). When you say:

> What we are trying to achieve by that is to have a test which ends on
"Test" and nothing else and it may contain "Test" only in case it does not
start with it.

Should the regex enforce the "which ends on Test" part? Otherwise, if we
want to allow tests like MyTestSplit1, do we need to update the "test.name"
parameter to allow that in the test runner?

Thanks,

Derek


On Mon, Oct 24, 2022 at 7:27 AM Miklosovic, Stefan <
stefan.mikloso...@netapp.com> wrote:

> Hi Derek,
>
> yes, the regexp is ^(?!(?:Test))(?!.*?(?:Tests)).*$
>
> What we are trying to achieve by that is to have a test which ends on
> "Test" and nothing else and it may contain "Test" only in case it does not
> start with it.
>
> To your second paragraph, I do not think we have any *Base files which are
> tests (that would be a bug) nor they are standalone (not inherited). By
> first approach, TestBaseImpl (which we do have) would start to be invalid,
> we would have to rename it to "DistributedTestBaseImpl" because it was
> starting on "Test" which is no-no. I agree that "TestBaseImpl" is rather
> unfortunate name. Currently, TestBaseImpl itself extends
> "DistributedTestBase" so in the end we would have "DistributedTestBaseImpl
> -> TestBaseImpl -> DistributedTestBase". Please keep in mind that
> "DistributedTestBase" is located in dtest jar (separate project) and that
> is rather inconvenient to rename.
>
> Regards
>
> 
> From: Derek Chen-Becker 
> Sent: Monday, October 24, 2022 15:09
> To: dev@cassandra.apache.org
> Subject: Re: Some tests are never executed in CI due to their name
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> For clarity, what is the final regex you've landed on for the first
> approach? Is it "^(?!(?:Test))(?!.*?(?:Tests)).*$" ? Are we just trying to
> reject anything that *starts* with "Test" or contains "Tests" somewhere in
> the name? It might be more clear to state what we think a valid test name
> should be, not in regex form.
>
> When it comes to "TestBase" files, are there any of these that are
> actually intended to be run as a test? The "Base" would indicate to me that
> it's intended to be inherited, not run directly. Do we have some "TestBase"
> classes somewhere that aren't inherited and are meant to be run directly,
> because that definitely seems like a misleading name that should be fixed,
> invasive or not. I would lean toward the principle of least surprise here,
> even if it means an involved one-time cleanup.
>
> Cheers,
>
> Derek
>
> On Mon, Oct 24, 2022 at 3:02 AM Miklosovic, Stefan <
> stefan.mikloso...@netapp.com<mailto:stefan.mikloso...@netapp.com>> wrote:
> Hi list,
>
> it was reported that some tests are not executed as part of CI because of
> their name (1). There is (2) and (3) so right now it runs only "*Test.java"
> files.
>
> Except of us renaming the affected classes, we should fix this in a more
> robust way - via checkstyle - so we enforce how test files have to be
> called.
>
> We came up with two approaches and we do not know which one to choose so
> we want to gather more feedback / opinions.
>
> The first approach would make this happen (implemented here (4))
>
> class name - passes / fails the checkstyle
>
> TestIt - fails
> TestsThatFeature - fails
> ThisTestsMyFeature - fails
> SomeTests - fails
>
> SomeHelperClassForTesting - passes
> SomeTest - passes
> DistributedTestBase - passes
>
> MyTestSplit1 - passes
>
> We would fix "SplitN" test files by hand (I think they are not executed
> right now in CI either) to something like MySplit1Test and MySplit2Test but
> the regexp we would eventually use would not prevent this from happening in
> the future. I do not know how to construct such a complex regexp yet which
> would cover all cases above plus this splitting one. Feel free to look into
> the ticket where details are discussed.
>
> The second approach would forbid to have any occurrence of "test" in the
> file name but at the end. For example, it would fail on classes like these
>
> MyTestSplit1
> TestHelper
> BurnTestUtil
> FuzzTestBase
> DistributedTestSnitch
> CASCommonTestCases
> ServerTestUtils
> AuthTestUtils
> TestMemtable
>
> While this wo

Re: Some tests are never executed in CI due to their name

[Derek Chen-Becker]

For clarity, what is the final regex you've landed on for the first
approach? Is it "^(?!(?:Test))(?!.*?(?:Tests)).*$" ? Are we just trying to
reject anything that *starts* with "Test" or contains "Tests" somewhere in
the name? It might be more clear to state what we think a valid test name
should be, not in regex form.

When it comes to "TestBase" files, are there any of these that are actually
intended to be run as a test? The "Base" would indicate to me that it's
intended to be inherited, not run directly. Do we have some "TestBase"
classes somewhere that aren't inherited and are meant to be run directly,
because that definitely seems like a misleading name that should be fixed,
invasive or not. I would lean toward the principle of least surprise here,
even if it means an involved one-time cleanup.

Cheers,

Derek

On Mon, Oct 24, 2022 at 3:02 AM Miklosovic, Stefan <
stefan.mikloso...@netapp.com> wrote:

> Hi list,
>
> it was reported that some tests are not executed as part of CI because of
> their name (1). There is (2) and (3) so right now it runs only "*Test.java"
> files.
>
> Except of us renaming the affected classes, we should fix this in a more
> robust way - via checkstyle - so we enforce how test files have to be
> called.
>
> We came up with two approaches and we do not know which one to choose so
> we want to gather more feedback / opinions.
>
> The first approach would make this happen (implemented here (4))
>
> class name - passes / fails the checkstyle
>
> TestIt - fails
> TestsThatFeature - fails
> ThisTestsMyFeature - fails
> SomeTests - fails
>
> SomeHelperClassForTesting - passes
> SomeTest - passes
> DistributedTestBase - passes
>
> MyTestSplit1 - passes
>
> We would fix "SplitN" test files by hand (I think they are not executed
> right now in CI either) to something like MySplit1Test and MySplit2Test but
> the regexp we would eventually use would not prevent this from happening in
> the future. I do not know how to construct such a complex regexp yet which
> would cover all cases above plus this splitting one. Feel free to look into
> the ticket where details are discussed.
>
> The second approach would forbid to have any occurrence of "test" in the
> file name but at the end. For example, it would fail on classes like these
>
> MyTestSplit1
> TestHelper
> BurnTestUtil
> FuzzTestBase
> DistributedTestSnitch
> CASCommonTestCases
> ServerTestUtils
> AuthTestUtils
> TestMemtable
>
> While this would also fix "MyTestSplitN" cases, I consider this to be way
> more invasive to the current codebase because we would have to rename a lot
> of files (like, hundreds, plus a lot of places they are referenced / used
> at) and I do not even think this is necessary and desirable. Like, how
> would we call "FuzzTestBase" to not contain "Test" in it? I think having
> "test" string in the middle is just fine, specifically for helper classes.
> One possible fix would be to replace "Test" with "Tester" so we would have
> "TesterMemtable", "AuthTesterUtils", "TesterHelper",
> "DistributedTesterSnitch" ... Word "Tester" would be explicitly allowed.
>
> I personally lean to the first approach even though cases like
> "MyTestSplit1" will not be spotted in compile time, but hey, we still do
> have a review process on top of this. The obvious violations like
> "TestThisFeature" and "MySuperTests" would be evaluated as violations which
> I would argue happens the most often.
>
> Please keep in mind that we have checkstyle only in 4.1 and trunk. So,
> while we would fix names of classes in all branches we support so they are
> picked up by CI, checkstyle for this would be introduced in 4.1 and trunk
> only.
>
> (1) https://issues.apache.org/jira/browse/CASSANDRA-17964
> (2) https://github.com/apache/cassandra/blob/trunk/build.xml#L61
> (3) https://github.com/apache/cassandra/blob/trunk/build.xml#L1033
> (4) https://github.com/instaclustr/cassandra/commits/CASSANDRA-17964-4.1-2



-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSS] Potential circleci config and workflow changes

[Derek Chen-Becker]

Random thought (and on-topic, even!) now that I'm starting to understand
CircleCI config better: we should use conditionals and parameters so that
we can have a single, uniform config across version branches, and limit the
diffs across branches to version related flags to enable or disable sets of
tests. I would even go so far as to say we might be able to put the config
into a submodule while pulling in branch-specific config from the top level
repo.

Cheers,

Derek

On Thu, Oct 20, 2022, 3:21 PM Josh McKenzie  wrote:

> I believe it's original intention to be just about CircleCI.
>
> It was but fwiw I'm good w/us exploring adjacent things regarding CI here.
> I'm planning on deep diving on the thread tomorrow and distilling a
> snapshot of the work we have a consensus on for circle and summarizing here
> so we don't lose that. Seems like it's fairly non-controversial.
>
> On Thu, Oct 20, 2022, at 5:14 PM, Mick Semb Wever wrote:
>
>
>
> On Thu, 20 Oct 2022 at 22:07, Derek Chen-Becker 
> wrote:
>
> Would the preclusion of non-committers also prevent us from configuring
> Jenkins to auto-test on PR independent of who opens it?
>
> One of my current concerns is that we're maintaining 2x the CI for 1x the
> benefit, and I don't currently see an easy way to unify them (perhaps a
> lack of imagination?). I know there's a long history behind the choice of
> CircleCI, so I'm not trying to be hand-wavy about all of the thought that
> went into that decision, but that decision has costs beyond just a paid
> CircleCI account. My long term, probably naive, goals for CI would be to:
>
> 1. Have a CI system that is *fully* available to *any* contributor, modulo
> safeguards to prevent abuse
>
>
>
> This thread is going off-topic, as I believe it's original intention to be
> just about CircleCI.
>
> But on your point… our community CI won't be allowed (by ASF), nor have
> capacity (limited donated resources), to run pre-commit testing by anyone
> and everyone.
>
> Today, trusted contributors can be handed tokens to ci-cassandra.a.o (make
> sure to label them so they can be revoked easily), but we still face the
> issue that too many pre-commit runs impacts the throughput and quality of
> the post-commit runs (though this has improved recently).
>
> It's on my wishlist to be able to: with a single command line; spin up the
> ci-cassandra.a.o stack on any k8s cluster, run any git sha through it and
> collect results, and tear it down. Variations on this would solve
> non-committers being able to repeat, use, and work on their own (or a
> separately donated) CI system, and folk/companies with money to be able to
> run their own ci-cassandra.a.o stacks for faster pre-commit turnaround
> time. Having this reproducibility of the CI system would make testing
> changes to it easier as well, so I'd expect a positive feedback loop here.
>
> I have some rough ideas on how to get started on this, if anyone would
> like to buddy up on it.
>
>
>

Re: [DISCUSS] Potential circleci config and workflow changes

[Derek Chen-Becker]

Would the preclusion of non-committers also prevent us from configuring
Jenkins to auto-test on PR independent of who opens it?

One of my current concerns is that we're maintaining 2x the CI for 1x the
benefit, and I don't currently see an easy way to unify them (perhaps a
lack of imagination?). I know there's a long history behind the choice of
CircleCI, so I'm not trying to be hand-wavy about all of the thought that
went into that decision, but that decision has costs beyond just a paid
CircleCI account. My long term, probably naive, goals for CI would be to:

1. Have a CI system that is *fully* available to *any* contributor, modulo
safeguards to prevent abuse
2. Have a CI system that is easy to maintain, with clear instructions and
examples for adding a new test suite
3. Have a CI system with reporting that makes it easy to quickly identify
test failures, as well as test *stability* over time

CI isn't glamorous, is often underfunded, and yet is critical to project
velocity and stability. I really appreciate Josh and others driving these
discussions!

Cheers,

Derek

On Thu, Oct 20, 2022 at 1:25 PM Ekaterina Dimitrova 
wrote:

> Sounds like great plan to me.
> Just wanted to mention one caveat. Non-committers do not have access to
> ASF CI. I do not think this will change. While no one of us ever said no to
> push a patch for testing, it is good to have a good backup plan people can
> do it themselves. Currently this is CircleCI. Maybe we can leave it just as
> is at some point and people who want to use it can continue supporting it
> themselves? I don’t have a clear answer now.  I guess time will show
>
> On Thu, 20 Oct 2022 at 14:51, Brandon Williams  wrote:
>
>> On Thu, Oct 20, 2022 at 1:45 PM Josh McKenzie 
>> wrote:
>> >
>> > My high level hope is that we can:
>> >
>> > 1. Unstick mainline yearly releases (vote to accept circle results,
>> make circle more robust <- WE ARE HERE)
>> > 2. Invest resources into the ASF CI environment to get it to being a
>> viable replacement for circle (requirements for this qualification TBD)
>> > 3. Deprecate circle
>> >
>> > I'm doing my best to help make the above a reality. There's a lot of
>> hand-waving in "ASF CI as viable replacement" but it's 2022 and there's a
>> lot of modern build and ci system's learning our industry has gained in the
>> last decade we're not yet taking advantage of.
>>
>> I'm a strong +1 on this plan and will do everything I can to assist.
>> We should have a discussion on how to get this accomplished and
>> delineate the current problems we are facing.
>>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSS] Potential circleci config and workflow changes

[Derek Chen-Becker]

ant is a simple job?
>> >> 4) why do we require people to install “circleci” command to
>> contribute?  If you rename .circleci/config-2_1.yml to .circleci/config.yml
>> then CI will work just fine… we don’t need to call “circleci config
>> process” every time we touch circle config…. Also, seems that w/e someone
>> new to circle config (but not cassandra) touch it they always mutate
>> LOW/MID/HIGH and not .circleci/config-2_1.yml… so I keep going back to fix
>> .circleci/config-2_1.yml….
>> >>
>> >>
>> >> On Oct 19, 2022, at 1:32 PM, Miklosovic, Stefan <
>> stefan.mikloso...@netapp.com> wrote:
>> >>
>> >> 1) would be nice to have. The first thing I do is that I change the
>> parallelism to 20. None of committed config.yaml's are appropriate for our
>> company CircleCI so I have to tweak this manually. I think we can not run
>> more that 25/30 containers in parallel, something like that. HIGHRES has
>> 100 and MIDRES has some jobs having parallelism equal to 50 or so so that
>> is not good either. I would be happy with simple way to modify default
>> config.yaml on parallelism. I use "sed" to change parallelism: 4 to
>> parallelism: 20 and leave parallelism: 1 where it does not make sense to
>> increase it. However I noticed that there is not "4" set everywhere, some
>> jobs have it set to "1" so I have to take extra care of these cases (I
>> consider that to be a bug, I think there are two or three, I do not
>> remember). Once set, I have that config in "git stash" so I just apply it
>> every time I need it.
>> >>
>> >> 5) would be nice too.
>> >> 7) is nice but not crucial, it takes no time to commit that.
>> >>
>> >> 
>> >> From: Josh McKenzie 
>> >> Sent: Wednesday, October 19, 2022 21:50
>> >> To: dev
>> >> Subject: [DISCUSS] Potential circleci config and workflow changes
>> >>
>> >> NetApp Security WARNING: This is an external email. Do not click links
>> or open attachments unless you recognize the sender and know the content is
>> safe.
>> >>
>> >>
>> >>
>> >> While working w/Andres on CASSANDRA-17939 a variety of things came up
>> regarding our circleci config and opportunities to improve it. Figured I'd
>> hit the list up here to see what people's thoughts are since many of us
>> intersect with these systems daily and having your workflow disrupted
>> without having a chance to provide input is bad.
>> >>
>> >> The ideas:
>> >> 1. Tune parallelism levels per job (David and Ekaterina have insight
>> on this)
>> >> 2. Rename jobs on circle to be more indicative of their function
>> >> 3. Unify j8 and j11 workflow pairs into single (for 2 and 3 see:
>> https://issues.apache.org/jira/browse/CASSANDRA-17939?focusedCommentId=17616595=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17616595
>> )
>> >> 4. Update documentation w/guidance on using circle,
>> .circleci/generate.sh examples, etc
>> >> 4a. How to commit:
>> https://cassandra.apache.org/_/development/how_to_commit.html
>> >> 4b. Testing: https://cassandra.apache.org/_/development/testing.html
>> >> 5. Flag on generate.sh to allow auto-run on push
>> >> 6. Clean up the -l, -m, -h flags (test and indicate -l feasibility for
>> all suites, default to -m, deprecate -h?) <- may not be a code-change issue
>> and instead be a documentation issue
>> >> 7. Consider flag on generate.sh to run and commit with "[DO NOT MERGE]
>> temporary circleci config" as the commit message
>> >>
>> >> Curious to see what folks think.
>> >>
>> >> ~Josh
>> >>
>> >>
>>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Website fixes PR

[Derek Chen-Becker]

I put together some minor fixes for the website dev documentation:

https://github.com/apache/cassandra-website/pull/179

I've tested these locally using the docker scripts, although antora seems
to create broken links and navigation (confirmed this behavior even without
my commit). I was told that this didn't need a JIRA ticket but I'm happy to
create one to track if that's more appropriate.

Thanks,

Derek

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Cassandra project status update 2022-10-13

[Derek Chen-Becker]

up.
>
>
> [Dev list Digest]
> https://lists.apache.org/list?dev@cassandra.apache.org:lte=14d:
>
> Pretty busy couple of weeks. There's an ongoing discussion about password
> validation and generation and CEP-24 that is covering some really solid
> ground on just how far is appropriate to go with the feature, what we'd
> like as operators, what we think is industry standard in our field, etc.
> You can chime in on the thread:
> https://lists.apache.org/thread/454tmo2r9238rj69j7h3xv43crygv31m
>
> We have a new episode of the Apache Cassandra Corner on staging that'll go
> live Friday night - if you'd like, feel free to preview it and provide
> Aaron with your feedback:
> https://lists.apache.org/thread/7prkk4dndfvf8oxsb17o74z61wknrn45
>
> Derek Chen-Becker (ApacheCon presenter extraordinaire!) is looking for
> reviewers on a circleci config and doc fix - we have PR's for that linked
> in his email:
> https://lists.apache.org/thread/lt0tplx19k6xvbl9f0pg0q5jql0r4j8t. @Derek
> - do we have JIRA's for this yet? That'd get them visible in our standard
> workflows so I could highlight them from the status emails here.
>
> We drove to consensus on the inclusion of the Agrona library with the
> points being raised about the dangers of overlapping functionality and
> documenting the "preferred and idiomatic Way To Do Things" in a code-base
> where you have multiple first-class options:
> https://lists.apache.org/thread/zk9hjk1rklcof1pmw555no032pmr3001. I don't
> think anyone took away the ToDo item of actually revising our Code Style
> guidelines (https://cassandra.apache.org/_/development/code_style.html)
> about library usage - maybe that'd be something good for someone to do when
> they add a library @Branimir? :)
>
> James reached out about joining the slack, which is what led me to realize
> I should include that little detail above in this status email:
> https://lists.apache.org/thread/53rmbdz8f089b00mwfwsnv5kmbdsjjhd
>
> The thread about whether we should move from 4.2 to 5.0 had a straggler
> from Patrick a couple weeks ago; I'm still firmly in the camp of "we should
> more rigorously document our JDK commitments and signpost upcoming JDK
> support addition and removal w/major versions", but words are cheap and
> Ekaterina's doing the hard work on the JDK17 support for now so I'll just
> shut up on that topic. :)
>
> Berenguer's looking for feedback re:
> https://issues.apache.org/jira/browse/CASSANDRA-14227, extending the
> maximum expiration date. Benedict provided some feedback on the mailing
> list regarding concerns w/memory pressure (which is being finicky because
> of mail threading w/an unrelated reply) -> thread starting w/the reply and
> proposal for delta encoding here:
> https://lists.apache.org/thread/j36ps2tsjjchfm1msl3v8xghox1djgyt. If you
> have a perspective on this please feel free to chime in.
>
> And last but certainly not least Brad keeps fighting the good fight on
> modernizing and cleaning up our python:
> https://lists.apache.org/thread/14wlyv2skmkn6jlg9ojh134c3p20ypg8! No harm
> in being careful and hitting up the dev list about removing and replacing
> libraries.
>
>
> [ASF CI Trends]
> https://butler.cassandra.apache.org/#/
>
> Here's our trends on our branches for the last two weeks:
>
> 3.0: 10 -> 13
> 3.11: 17 -> 22
> 4.0: 5 -> 6
> 4.1: 11 -> 14
> trunk: 11 -> 7
>
> The spike in 4.1 appears to be local socket binding errors:
> https://ci-cassandra.apache.org/job/Cassandra-4.1/181/testReport/dtest-offheap.replication_test/TestSnitchConfigurationUpdate/test_rf_collapse_property_file_snitch/,
> ccmlib.common.UnavailableSocketError: Inet address 127.0.0.2:7000 is not
> available: [Errno 98] Address already in use; a cluster may already be
> running or you may need to add the loopback alias
>
> This is something we saw intermittently in the past but anecdotally
> appears to be becoming more prevalent. Not sure if we have a JIRA for it yet
>
>
> [CircleCI Status]
> Andres de la Peña is working on CASSANDRA-17938 (allow multiple tests to
> be multiplexed in one run) and CASSANDRA-17939 (automatically detect and
> repeat new or modified JUnit tests), Derek's working on getting our test
> suite coverage into parity on CASSANDRA-17950 (enable dtest-offheap in
> circle-ci). We'll need to do another audit of which tests are running in
> ASF CI and not in Circle to ensure that the coverage there matches; several
> conversations have gone on about this in slack, email threads, and JIRA, so
> if I've missed something here please ping the thread. Probably worth
> creating an epic to validate circle as a CI env that we can use to validate
> a release; I'll get on that.
>
>
>

Re: [Discuss] CEP-24 Password validation and generation

[Derek Chen-Becker]

Thanks Jeff, you put it more succinctly than I was able to. I think just
having these logged out in the app log would be sufficient for audit
purposes.

Cheers,

Derek

On Tue, Oct 11, 2022 at 12:56 PM Jeff Jirsa  wrote:

> I don't think logging which policy violation was triggered is that big of
> an ask?
>
> "Password changed for user X, complying with policies (reuse, complexity,
> entropy)"
> "ERROR: Password rejected due to policy violation (reuse)"
> "ERROR: Password rejected due to policy violation (complexity)"
> "ERROR: Password rejected due to policy violation (entropy)"
>
> The success log makes it much easier for users subject to audit controls,
> and the failures make it much easier to tell users what's going on for
> those users who operate the database for other people.
>
>
>
>
> On Tue, Oct 11, 2022 at 11:48 AM Miklosovic, Stefan <
> stefan.mikloso...@netapp.com> wrote:
>
>> I am afraid this is way out of scope of this CEP. I think we would be the
>> very first on the database scene to offer such low-level and fine-grained
>> information. I am not persuaded that this is something we should be giving
>> a lot of attention right now. We have "cassandra / cassandra" credentials
>> combo as default, I would say that is more important to deal with right now
>> than providing very detailed information about what kind of passwords
>> people are using.
>>
>> Thank you very much for (not only your) insights. This is very important
>> feedback and I am glad you participate in this thread. I will try to
>> summarize where we are as it is easy to get lost in these emails.
>>
>> 
>> From: Derek Chen-Becker 
>> Sent: Tuesday, October 11, 2022 18:59
>> To: dev@cassandra.apache.org
>> Subject: Re: [Discuss] CEP-24 Password validation and generation
>>
>> NetApp Security WARNING: This is an external email. Do not click links or
>> open attachments unless you recognize the sender and know the content is
>> safe.
>>
>>
>>
>> Speaking with my operator hat on, I would want to know if there's a
>> common pattern that my end users hit so that I can better educate them or
>> provide tools (e.g. vaults) to help them manage the required complexity.
>>
>> Cheers,
>>
>> Derek
>>
>> On Tue, Oct 11, 2022 at 10:06 AM Miklosovic, Stefan <
>> stefan.mikloso...@netapp.com<mailto:stefan.mikloso...@netapp.com>> wrote:
>> "but we should consider logging why it was rejected."
>>
>> Why? What is the use case? Why is it important for you to know what
>> somebody tried to create a role with password "aa" (it will not be
>> shown), just mentioning that they tried to create a password with a lot of
>> repeating characters? What is the added value here?
>>
>> I need to double check if warnings are logged as well. I'll get back to
>> you.
>>
>>
>> 
>> From: Derek Chen-Becker > de...@chen-becker.org>>
>> Sent: Tuesday, October 11, 2022 17:47
>> To: dev@cassandra.apache.org<mailto:dev@cassandra.apache.org>
>> Subject: Re: [Discuss] CEP-24 Password validation and generation
>>
>> NetApp Security WARNING: This is an external email. Do not click links or
>> open attachments unless you recognize the sender and know the content is
>> safe.
>>
>>
>>
>> I know we log that the password change attempt was made, but we should
>> consider logging why it was rejected. As part of that, we may want to
>> consider how we format that message to make it easy for an auditing system
>> to parse. We should never log the actual password, or even a redacted
>> version; apologies if it sounded like I was suggesting that.
>>
>> Cheers,
>>
>> Derek
>>
>> On Tue, Oct 11, 2022 at 9:36 AM Miklosovic, Stefan <
>> stefan.mikloso...@netapp.com<mailto:stefan.mikloso...@netapp.com>> stefan.mikloso...@netapp.com<mailto:stefan.mikloso...@netapp.com>>>
>> wrote:
>> I dont follow, sorry. What logs do you mean? What would you like to see?
>>
>> The auditing framework we already do have in place will log that somebody
>> tried to create (or alter) a role with this and that password and it failed
>> (password would be redacted). If you use "with generated password", that
>> password will not be even shown in audit log. I am not completely sure if
>> the warning is logged too if password is not valid (I do think that only
>> CQL command i

Looking for reviewers for CircleCI config and doc fix

[Derek Chen-Becker]

Hi,

I have a minor doc fix for the CircleCI generate.sh script:

https://github.com/apache/cassandra/pull/1902

@adelapena made the change to the script that added the flag, so maybe he
could take a quick look. On the CircleCI build itself, I have a small PR to
make the offheap dtests run (they're currently missing):

https://github.com/apache/cassandra/pull/1904

Ekaterina was helping with this but it sounds like she's unavailable until
next week, so I was wondering if someone else familiar with CircleCI could
take a look.

Thanks,

Derek


-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [Discuss] CEP-24 Password validation and generation

[Derek Chen-Becker]

Speaking with my operator hat on, I would want to know if there's a common
pattern that my end users hit so that I can better educate them or provide
tools (e.g. vaults) to help them manage the required complexity.

Cheers,

Derek

On Tue, Oct 11, 2022 at 10:06 AM Miklosovic, Stefan <
stefan.mikloso...@netapp.com> wrote:

> "but we should consider logging why it was rejected."
>
> Why? What is the use case? Why is it important for you to know what
> somebody tried to create a role with password "aa" (it will not be
> shown), just mentioning that they tried to create a password with a lot of
> repeating characters? What is the added value here?
>
> I need to double check if warnings are logged as well. I'll get back to
> you.
>
>
> 
> From: Derek Chen-Becker 
> Sent: Tuesday, October 11, 2022 17:47
> To: dev@cassandra.apache.org
> Subject: Re: [Discuss] CEP-24 Password validation and generation
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> I know we log that the password change attempt was made, but we should
> consider logging why it was rejected. As part of that, we may want to
> consider how we format that message to make it easy for an auditing system
> to parse. We should never log the actual password, or even a redacted
> version; apologies if it sounded like I was suggesting that.
>
> Cheers,
>
> Derek
>
> On Tue, Oct 11, 2022 at 9:36 AM Miklosovic, Stefan <
> stefan.mikloso...@netapp.com<mailto:stefan.mikloso...@netapp.com>> wrote:
> I dont follow, sorry. What logs do you mean? What would you like to see?
>
> The auditing framework we already do have in place will log that somebody
> tried to create (or alter) a role with this and that password and it failed
> (password would be redacted). If you use "with generated password", that
> password will not be even shown in audit log. I am not completely sure if
> the warning is logged too if password is not valid (I do think that only
> CQL command itself is audit-logged).
>
> The configuration of validator is in cassandra.yaml. Folks can review that?
>
> I am sorry if I am missing something here, could you expand what you mean?
>
> To Josh:
>
> You are probably right but it is so easy to bypass that it is questionable
> why it is actually there. All it takes is to do "alter role myself with
> generated password" (literally), 5 times in a row and you can set the
> original one back. One positive fact is that such password, even same as
> the original one, would still have to be valid, but it just might be same
> as it was.
>
> 
> From: Derek Chen-Becker  de...@chen-becker.org>>
> Sent: Tuesday, October 11, 2022 17:14
> To: dev@cassandra.apache.org<mailto:dev@cassandra.apache.org>
> Subject: Re: [Discuss] CEP-24 Password validation and generation
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> On top of what Josh said, a corresponding requirement here would be
> auditing. A password complexity and/or history policy is one piece of
> security posture, but is itself insufficient to be considered "secure".
> What kind of logs will this new policy checker emit that the folks
> responsible for security for a given cluster can parse, process, and report
> on?
>
> Cheers,
>
> Derek
>
> On Tue, Oct 11, 2022 at 9:07 AM Josh McKenzie  <mailto:jmcken...@apache.org><mailto:jmcken...@apache.org jmcken...@apache.org>>> wrote:
> if we do that, we should also restrict the frequency how often a user can
> change the password. Lets think this through. If the depth of historical
> verification is 5 passwords, a user just has to regenerate a password 5
> times in a row an he can use the same one
> I may be misunderstanding, but this strikes me as in the "we can only do
> so much to prevent people from doing stupid things" category. If someone's
> so hell-bent on circumventing their company's attempts at security they're
> probably much more at risk of running unidentified attachments or giving
> out credentials over the phone rather than finding clever ways to work
> around annoying password rotation rules on their db accounts right?
>
> On Mon, Oct 10, 2022, at 4:08 PM, Miklosovic, Stefan wrote:
> Hi Brad,
>
> your link about not enforcing regular password expiration for users is
> spot on. For these reasons I decided to not expand that CEP in that
&g

Re: [Discuss] CEP-24 Password validation and generation

[Derek Chen-Becker]

I know we log that the password change attempt was made, but we should
consider logging why it was rejected. As part of that, we may want to
consider how we format that message to make it easy for an auditing system
to parse. We should never log the actual password, or even a redacted
version; apologies if it sounded like I was suggesting that.

Cheers,

Derek

On Tue, Oct 11, 2022 at 9:36 AM Miklosovic, Stefan <
stefan.mikloso...@netapp.com> wrote:

> I dont follow, sorry. What logs do you mean? What would you like to see?
>
> The auditing framework we already do have in place will log that somebody
> tried to create (or alter) a role with this and that password and it failed
> (password would be redacted). If you use "with generated password", that
> password will not be even shown in audit log. I am not completely sure if
> the warning is logged too if password is not valid (I do think that only
> CQL command itself is audit-logged).
>
> The configuration of validator is in cassandra.yaml. Folks can review that?
>
> I am sorry if I am missing something here, could you expand what you mean?
>
> To Josh:
>
> You are probably right but it is so easy to bypass that it is questionable
> why it is actually there. All it takes is to do "alter role myself with
> generated password" (literally), 5 times in a row and you can set the
> original one back. One positive fact is that such password, even same as
> the original one, would still have to be valid, but it just might be same
> as it was.
>
> 
> From: Derek Chen-Becker 
> Sent: Tuesday, October 11, 2022 17:14
> To: dev@cassandra.apache.org
> Subject: Re: [Discuss] CEP-24 Password validation and generation
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> On top of what Josh said, a corresponding requirement here would be
> auditing. A password complexity and/or history policy is one piece of
> security posture, but is itself insufficient to be considered "secure".
> What kind of logs will this new policy checker emit that the folks
> responsible for security for a given cluster can parse, process, and report
> on?
>
> Cheers,
>
> Derek
>
> On Tue, Oct 11, 2022 at 9:07 AM Josh McKenzie  <mailto:jmcken...@apache.org>> wrote:
> if we do that, we should also restrict the frequency how often a user can
> change the password. Lets think this through. If the depth of historical
> verification is 5 passwords, a user just has to regenerate a password 5
> times in a row an he can use the same one
> I may be misunderstanding, but this strikes me as in the "we can only do
> so much to prevent people from doing stupid things" category. If someone's
> so hell-bent on circumventing their company's attempts at security they're
> probably much more at risk of running unidentified attachments or giving
> out credentials over the phone rather than finding clever ways to work
> around annoying password rotation rules on their db accounts right?
>
> On Mon, Oct 10, 2022, at 4:08 PM, Miklosovic, Stefan wrote:
> Hi Brad,
>
> your link about not enforcing regular password expiration for users is
> spot on. For these reasons I decided to not expand that CEP in that
> direction. Sure, technically possible, but practically questionable. I
> think that all these guides and recommendations should be looked at from
> the perspective of the system they are meant to be implemented in.
> Enforcing password to be changed in a database system is rather interesting
> take. After I briefly took a look, I do not think there is a database on
> the market which is enforcing this. On the other hand, for example, Neo4j
> forces you to change the password on the first login as the default one is
> "neo4j" for user "neo4j". This does make sense to implement for Cassandra
> as well. I do consider password "cassandra" for role "cassandra" very
> insecure and it is not forced by anybody to change it. However, it is quite
> interesting problem how to achieve that.
>
> Also, the reason I want to leave out historical verification of passwords
> in (at least the initial) implementation is that if we do that, we should
> also restrict the frequency how often a user can change the password. Lets
> think this through. If the depth of historical verification is 5 passwords,
> a user just has to regenerate a password 5 times in a row an he can use the
> same one. So implmenting this without restricting how often he can change
> his password does not make sense. We can indeed explore this further but I
> feel like t

Re: [Discuss] CEP-24 Password validation and generation

[Derek Chen-Becker]

when using Cassandra and cqlsh in
> console environment.
>
> (1) https://pages.nist.gov/800-63-3/sp800-63b.html#appA
> 
> From: Brad 
> Sent: Monday, October 10, 2022 17:43
> To: dev@cassandra.apache.org
> Subject: Re: [Discuss] CEP-24 Password validation and generation
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> I would suggest reviewing the guidelines in sec in 5.1.1.2 of NIST Special
> Publication 800-63B<
> https://pages.nist.gov/800-63-3/sp800-63b.html#memsecretver> and the NCSC
> Password policy: updating your approach - NCSC.GOV.UK<
> https://www.ncsc.gov.uk/collection/passwords/updating-your-approach#PasswordGuidance:UpdatingYourApproach-Don'tenforceregularpasswordexpiry
> >
>
> Regards,
>
> Brad
>
>
> On Mon, Sep 19, 2022 at 7:27 AM Miklosovic, Stefan <
> stefan.mikloso...@netapp.com<mailto:stefan.mikloso...@netapp.com>> wrote:
> Hi list,
>
> together with my colleague Jackson Fleming we put together CEP-24 about
> password validation and password generation in Cassandra.
>
> https://cwiki.apache.org/confluence/x/QoueDQ
>
> We are looking forward to discuss this CEP with you in depth.
>
> The outcome of this thread would be to sort out any issues / concerns you
> have so we might eventually vote and implement that in upstream if our
> contribution is found to be useful.
>
> There is a reference implementation provided we would like to build our
> solution on top.
>
> Regards
>
> Stefan Miklosovic
>
>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [VOTE] Revising release gating criteria and CI systems

[Derek Chen-Becker]

+1

I should have an initial patch to add some of the missing tests in circleci
Monday.

Derek

On Sat, Oct 8, 2022, 6:31 AM Josh McKenzie  wrote:

> DISCUSS thread:
> https://lists.apache.org/thread/o166v7nr9lxnzdy5511tv40rr9t6zbrw
>
> Revise Release Lifecycle cwiki page (
> https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle):
>  - Ensure we have parity on jobs run and coverage between circle and asf-ci
>  - Allow usage of circleci as gatekeeper for releases. A release will
> require 1 green run for beta, 3 green runs consecutively for ga
>  - No new consistent regressions on CI for asf compared to prior branches
>  - Explicitly do not consider ci-cassandra asf flaky tests as release
> blockers
>
> Changes to codify into documentation (
> https://cassandra.apache.org/_/development/how_to_commit.html):
>  - On patch before commit, multiplex @500 all new tests, changed tests, or
> expected to be impacted tests ("expected to be impacted" piece pending
> multi-class multiplexing support).
>  - Add support for / documentation for multi-class specification in
> multiplexer and document
>
> Add informal project commitment during next major release lifecycle to
> continue working on bringing asf ci-cassandra up to where it can be formal
> gatekeeper for release.
>
> ---
> The vote for these revisions will run through EoD 10/12/22 to give us the
> weekend + 72 business hours.
>

Re: [DISCUSS] Adding dependency on agrona

[Derek Chen-Becker]

Providing some guidance seems like a reasonable step to take. Would that be
in our coding guidelines (
https://cassandra.apache.org/_/development/code_style.html) or do we have
other places where we provide development guidelines? I like your
optimism on Valhalla's timeline, but I also agree that there's no need to
change things that aren't currently problems :)

Cheers,

Derek

On Sat, Oct 1, 2022 at 6:51 AM Benedict  wrote:

> I agree we need to be cautious about permitting overlapping functionality.
> I think if there are other utilities that duplicate existing functionality
> we should have a similar discussion on list before we begin using it, even
> after including the library. Perhaps even our wording around seeking input
> should be modified to include functionality present in a dependency but
> unused, and that may duplicate existing functionality.
>
> I am unconvinced it would be worthwhile replacing the use of ByteBuffer
> more generally within the codebase though, as Project Valhalla will likely
> cause us to revisit our internal APIs significantly - and that may arrive
> in some form within the next couple of years. So the value of the churn of
> changing implementation is unlikely to be high.
>
> That said, isolated subsystems could probably use it if we think it’s
> superior for the time being.
>
> Just my 2c.
>
> On 29 Sep 2022, at 15:26, Derek Chen-Becker  wrote:
>
> 
> +1 from me with some concerns.
>
> Agrona looks like a nice lib, and I like what I've seen skimming the docs,
> but I think there's a longer-term discussion around whether we want to do
> anything about bringing in a library that effectively duplicates a bunch of
> standard lib constructs. My main concern is cognitive load if there are
> subtle semantic differences, say, between Agrona and stdlib buffers or any
> other construct where we might have both in the codebase. If we think
> Agrona has better buffers (or locking, timers, etc), should we be talking
> about replacing usage of stdlib with Agrona throughout, or making a
> recommendation for one over the other for future work?
>
> Cheers,
>
> Derek
>
> On Thu, Sep 29, 2022 at 12:26 AM Benedict  wrote:
>
>> Also +1
>>
>> On 23 Sep 2022, at 18:17, David Capwell  wrote:
>>
>> +1 from me
>>
>> On Sep 23, 2022, at 1:21 AM, Branimir Lambov <
>> branimir.lam...@datastax.com> wrote:
>>
>> The usage in the trie memtable is only for volatile access to buffers. In
>> this case I chose the library instead of reimplementing the functionality
>> (e.g. as methods in `ByteBufferUtil`) because the relevant interface makes
>> sense and the library is a good quality one that contains a range of other
>> utilities that can be very useful to Cassandra.
>>
>> In other words, I personally would welcome opening Cassandra up to using
>> other parts of Agrona, and am asking if the community shares this sentiment.
>>
>>
>> Regards,
>> Branimir
>>
>> On Wed, Sep 21, 2022 at 9:15 PM Derek Chen-Becker 
>> wrote:
>>
>>> Agrona looks like it has quite a bit more than just buffers, so if we
>>> add this as a dependency for the new memtable, would it potentially open up
>>> use of other parts of Agrona (wittingly or not)? Unless I misunderstood,
>>> wasn't part of the new memtable implementation an interface to allow this
>>> to be pluggable? Could we avoid bringing it in as a full dependency for
>>> Cassandra if the trie memtable were packaged separately as a plugin instead
>>> of being included directly?
>>>
>>> Cheers,
>>>
>>> Derek
>>>
>>> On Wed, Sep 21, 2022 at 6:41 AM Benedict  wrote:
>>>
>>>> In principle no, it’s a high quality library. But it might help to
>>>> briefly outline what it’s used for. I assume it is instead of ByteBuffer?
>>>> In which case it could maybe be worthwhile discussing as a project how we
>>>> foresee interaction with existing buffer machinery, and maybe how we expect
>>>> our buffer use to evolve on the project, as we already have several 
>>>> buffers.
>>>>
>>>> That said, I anticipate our buffer use changing significantly with the
>>>> introduction of value types and native memory improvements coming in future
>>>> Java releases, so my personal inclination is just to accept the dependency.
>>>>
>>>> On 21 Sep 2022, at 13:29, Branimir Lambov  wrote:
>>>>
>>>> 
>>>> Hi everyone,
>>>>
>>>> CASSANDRA-17240 (Trie memtable implementation) introduces a dependency
>>&g

Re: [DISCUSS] Adding dependency on agrona

[Derek Chen-Becker]

+1 from me with some concerns.

Agrona looks like a nice lib, and I like what I've seen skimming the docs,
but I think there's a longer-term discussion around whether we want to do
anything about bringing in a library that effectively duplicates a bunch of
standard lib constructs. My main concern is cognitive load if there are
subtle semantic differences, say, between Agrona and stdlib buffers or any
other construct where we might have both in the codebase. If we think
Agrona has better buffers (or locking, timers, etc), should we be talking
about replacing usage of stdlib with Agrona throughout, or making a
recommendation for one over the other for future work?

Cheers,

Derek

On Thu, Sep 29, 2022 at 12:26 AM Benedict  wrote:

> Also +1
>
> On 23 Sep 2022, at 18:17, David Capwell  wrote:
>
> +1 from me
>
> On Sep 23, 2022, at 1:21 AM, Branimir Lambov 
> wrote:
>
> The usage in the trie memtable is only for volatile access to buffers. In
> this case I chose the library instead of reimplementing the functionality
> (e.g. as methods in `ByteBufferUtil`) because the relevant interface makes
> sense and the library is a good quality one that contains a range of other
> utilities that can be very useful to Cassandra.
>
> In other words, I personally would welcome opening Cassandra up to using
> other parts of Agrona, and am asking if the community shares this sentiment.
>
>
> Regards,
> Branimir
>
> On Wed, Sep 21, 2022 at 9:15 PM Derek Chen-Becker 
> wrote:
>
>> Agrona looks like it has quite a bit more than just buffers, so if we add
>> this as a dependency for the new memtable, would it potentially open up use
>> of other parts of Agrona (wittingly or not)? Unless I misunderstood, wasn't
>> part of the new memtable implementation an interface to allow this to be
>> pluggable? Could we avoid bringing it in as a full dependency for Cassandra
>> if the trie memtable were packaged separately as a plugin instead of being
>> included directly?
>>
>> Cheers,
>>
>> Derek
>>
>> On Wed, Sep 21, 2022 at 6:41 AM Benedict  wrote:
>>
>>> In principle no, it’s a high quality library. But it might help to
>>> briefly outline what it’s used for. I assume it is instead of ByteBuffer?
>>> In which case it could maybe be worthwhile discussing as a project how we
>>> foresee interaction with existing buffer machinery, and maybe how we expect
>>> our buffer use to evolve on the project, as we already have several buffers.
>>>
>>> That said, I anticipate our buffer use changing significantly with the
>>> introduction of value types and native memory improvements coming in future
>>> Java releases, so my personal inclination is just to accept the dependency.
>>>
>>> On 21 Sep 2022, at 13:29, Branimir Lambov  wrote:
>>>
>>> 
>>> Hi everyone,
>>>
>>> CASSANDRA-17240 (Trie memtable implementation) introduces a dependency
>>> on the agrona  library (https://github.com/real-logic/agrona).
>>>
>>> Does anyone have any objections to adding this dependency?
>>>
>>> Regards,
>>> Branimir
>>>
>>>
>>
>> --
>> +---+
>> | Derek Chen-Becker |
>> | GPG Key available at https://keybase.io/dchenbecker
>> <https://urldefense.com/v3/__https://keybase.io/dchenbecker__;!!PbtH5S7Ebw!cY9TyIm1RqAGMkhgyKDjzQcOq6Cy6kzMj_VjvMm40JG9VMm6JgFfH9omG1Spx0UmlkEcGJcFmDtKjcbIGBN7PBunbg$>
>> and   |
>> | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org
>> <https://urldefense.com/v3/__https://pgp.mit.edu/pks/lookup?search=derek*40chen-becker.org__;JQ!!PbtH5S7Ebw!cY9TyIm1RqAGMkhgyKDjzQcOq6Cy6kzMj_VjvMm40JG9VMm6JgFfH9omG1Spx0UmlkEcGJcFmDtKjcbIGBPzYoayyA$>
>> |
>> | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
>> +---+
>>
>>
>
> --
> Branimir Lambov
> e. branimir.lam...@datastax.com
> w. www.datastax.com
>
>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [Discuss] CASSANDRA-17914: Modernize CQLSH's with argparse for CLI arts

[Derek Chen-Becker]

+1 from me. It sounds like a good opportunity!

Cheers,

Derek

On Thu, Sep 29, 2022, 6:26 AM Brad  wrote:

> 
> The Python standard library introduced argparse a decade ago in Python 2.7
> to replace optparse as described in PEP-0389
>  for command line argument parsing.
> Optparse is no longer maintained, and has been deprecated since Python 3.2,
> although there are no plans to remove it from the std library.
>
> As part of modernizing CQLSH, I have proposed in CASSANDRA-17914 that we
> upgrade from optparse to argparse.  Argparse is part of the Python standard
> library and has been since 2011 so this upgrade involves no new library
> dependencies and should be self-contained and transparent.
>
> The primary benefit is removing dependencies on deprecated classes and
> components.  Consensus seems to be that argparse has more meaningful help
> messages and is more intuitive to use.
>
>
> Regards,
>
> Brad Schoening
>

Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

[Derek Chen-Becker]

Happy to help! Certainly getting Circle CI and ASF CI at parity seems like
a good goal. I'm going to probably have some extra cycles next week while
I'm at ApacheCon, so I'm hoping to make some meaningful progress :)

Cheers,

Derek

On Mon, Sep 26, 2022 at 12:44 PM Ekaterina Dimitrova 
wrote:

> Thanks Josh for bringing this up.
>  I agree with Brandon on both points he made.
> I have also other examples of testing some things only in Jenkins or only
> in CircleCI. If people are fine with that, I can open a ticket and I just
> talked n Slack to Derek who expressed interest to help with that activity.
> Thanks in advance Derek, this will be great support for the project.
>
> On Mon, 26 Sep 2022 at 14:15, Brandon Williams  wrote:
>
>> On Mon, Sep 26, 2022 at 12:20 PM Josh McKenzie 
>> wrote:
>> > For CI on a patch, run the pre-commit suite and also run multiplexer
>> with 250 runs on new, changed, or related tests to ensure not flaky
>>
>> 250 iterations isn't enough; I use 500 as a low water mark.
>>
>> > I think it's worth calling out: relying on circleci test runs along
>> with ASF CI as an option implies we have the same confidence in a green
>> circleci run as we have in a green ASF CI run.
>>
>> This is also assuming that circle and ASF CI run the same tests, which
>> is not entirely true.  We know that circle isn't running some upgrade
>> tests from https://issues.apache.org/jira/browse/CASSANDRA-17912 but I
>> think we need to specifically investigate parity between the two
>> systems for further discrepancies before we can begin to compare
>> confidence in runs between them.
>>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSS] Adding dependency on agrona

[Derek Chen-Becker]

Agrona looks like it has quite a bit more than just buffers, so if we add
this as a dependency for the new memtable, would it potentially open up use
of other parts of Agrona (wittingly or not)? Unless I misunderstood, wasn't
part of the new memtable implementation an interface to allow this to be
pluggable? Could we avoid bringing it in as a full dependency for Cassandra
if the trie memtable were packaged separately as a plugin instead of being
included directly?

Cheers,

Derek

On Wed, Sep 21, 2022 at 6:41 AM Benedict  wrote:

> In principle no, it’s a high quality library. But it might help to briefly
> outline what it’s used for. I assume it is instead of ByteBuffer? In which
> case it could maybe be worthwhile discussing as a project how we foresee
> interaction with existing buffer machinery, and maybe how we expect our
> buffer use to evolve on the project, as we already have several buffers.
>
> That said, I anticipate our buffer use changing significantly with the
> introduction of value types and native memory improvements coming in future
> Java releases, so my personal inclination is just to accept the dependency.
>
> On 21 Sep 2022, at 13:29, Branimir Lambov  wrote:
>
> 
> Hi everyone,
>
> CASSANDRA-17240 (Trie memtable implementation) introduces a dependency on
> the agrona  library (https://github.com/real-logic/agrona).
>
> Does anyone have any objections to adding this dependency?
>
> Regards,
> Branimir
>
>

-- 
+-------+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSS] CEP-20: Dynamic Data Masking

[Derek Chen-Becker]

asking functions
>>>>>> on their own will. I think most dbs allow this pattern of usage, which is
>>>>>> quite straightforward. Obviously, it doesn't allow admins to decide 
>>>>>> enforce
>>>>>> users seeing only masked data. Nevertheless, it's still useful for 
>>>>>> trusted
>>>>>> database users generating masked data that will be consumed by the end
>>>>>> users of the application.
>>>>>>
>>>>>> 2) Masking functions attached to specific columns. This way the same
>>>>>> queries will see different data (masked or not) depending on the
>>>>>> permissions of the user running the query. It has the advantage of not
>>>>>> requiring to change the queries that users with different permissions 
>>>>>> run.
>>>>>> The downside is that users would need to query the schema if they need to
>>>>>> know whether a column is masked, unless we change the names of the 
>>>>>> returned
>>>>>> columns. This is the approach offered by Azure/SQL Server, PostgreSQL, 
>>>>>> IBM
>>>>>> Db2, Oracle, MariaDB/MaxScale and SnowFlake. All these databases support
>>>>>> applying the masking function to columns on the base table, and some of
>>>>>> them also allow to apply masking to views.
>>>>>>
>>>>>> 3) Masking functions as part of projected views. This ways users
>>>>>> might need to query the view appropriate for their permissions instead of
>>>>>> the base table. This might mean changing the queries if the masking 
>>>>>> policy
>>>>>> is changed by the admin. MySQL recommends this approach on a blog entry,
>>>>>> although it's not part of its main documentation for data masking, and 
>>>>>> the
>>>>>> implementation has security issues. Some of the other databases offering
>>>>>> the approach 2) as their main option also support masking on view 
>>>>>> columns.
>>>>>>
>>>>>> Each approach has its own advantages and limitations, and I don't
>>>>>> think we necessarily have to choose. The CEP proposes implementing 1) and
>>>>>> 2), but no one impedes us to also have 3) if we get to have projected
>>>>>> views. However, I think that projected views is a new general-purpose
>>>>>> feature with its own complexities, so it would deserve its own CEP, if
>>>>>> someone is willing to work on the implementation.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, 31 Aug 2022 at 12:03, Claude Warren via dev <
>>>>>> dev@cassandra.apache.org> wrote:
>>>>>>
>>>>>>> Is there enough support here for VIEWS to be the implementation
>>>>>>> strategy for displaying masking functions?
>>>>>>>
>>>>>>> It seems to me the view would have to store the query and apply a
>>>>>>> where clause to it, so the same PK would be in play.
>>>>>>>
>>>>>>> It has data leaking properties.
>>>>>>>
>>>>>>> It has more use cases as it can be used to
>>>>>>>
>>>>>>>- construct views that filter out sensitive columns
>>>>>>>- apply transforms to convert units of measure
>>>>>>>
>>>>>>> Are there more thoughts along this line?
>>>>>>>
>>>>>>
>>>>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISSCUSS] Access to JDK internals only after dev mailing list consensus?

[Derek Chen-Becker]

I think it's fine to state it explicitly rather than making it an
assumption. Are we tracking any usage of internals in the codebase
currently?

Cheers,

Derek

On Fri, Sep 2, 2022 at 6:30 AM Ekaterina Dimitrova 
wrote:

>
>
> “ A quick heads up to the dev list with the jira would be sufficient for
> anybody interested in discussing it further to comment on the jira.”
>
> Agreed, I did’t mean voting but more or less we have the lazy consensus or
> sharing concerns. Discussing them on a ticket should be enough but it needs
> to happen. Also, it shouldn’t be  more often than adding dependencies I
> guess.
>
> JDK team is only closing more and more internals and warning us about
> potential breakages. I want to prevent us from urgent fixing in patch
> releases and to ease the maintenance.
>
> I think ensuring that it is clearly documented why an exception is
> acceptable and what options were considered will be of benefit for
> maintenance. We can revise in time what has changed.
>
> “ . Unless absolutely needed we should avoid accessing the internals.
> Folks on this project should understand why. We can make the dangers of
> this explicit in our contributor documentation. ”
> +1
>
> On Fri, 2 Sep 2022 at 1:26, Dinesh Joshi  wrote:
>
>> Personally not opposed to this. However, this is something that should be
>> vetted closely by the reviewers. Unless absolutely needed we should avoid
>> accessing the internals. Folks on this project should understand why. We
>> can make the dangers of this explicit in our contributor documentation.
>> However, requiring an entire dev list discussion around it seems
>> unnecessary. A quick heads up to the dev list with the jira would be
>> sufficient for anybody interested in discussing it further to comment on
>> the jira. WDYT?
>>
>> Dinesh
>>
>> On Sep 1, 2022, at 8:31 AM, Ekaterina Dimitrova 
>> wrote:
>>
>> Hi everyone,
>>
>>
>> Some time ago we added a note to the project Cassandra Code Style:
>> “New dependencies should not be included without community consensus
>> first being obtained via a [DISCUSS] thread on the
>> dev@cassandra.apache.org mailing list”
>>
>> I would like to suggest also to add a point around accessing JDK
>> internals. Any  patch that suggests accessing internals and/or adding even
>> more add-opens/add-exports to be approved prior commit on the mailing list.
>>
>> It seems to me the project can only benefit of this visibility. If
>> something is accepted as an exception, we need to have the right
>> understanding and visibility of why; in some cases maybe to see for
>> alternatives, to have follow up tickets opened, ownership taken. In my
>> opinion this will be very helpful for maintaining the codebase.
>>
>> If others agree with that I can add a sentence to the Code Style. Please
>> let me know what you think.
>>
>> Best regards,
>> Ekaterina
>>
>>
>>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSS] CEP-20: Dynamic Data Masking

[Derek Chen-Becker]

Yes, I was thinking that simple projection views (essentially a SELECT
statement with application of transform functions) would complement masking
functions, and from the discussion it sounds like this is basically what
some of the other databases do. Projection views seem like they would be
useful in their own right, so would it be proper to write a separate CEP
for that? I would be happy to help drive that document and discussion. I'm
not sure if it's the best name, but I'm trying to distinguish views that
expose a subset of an existing schema vs materialized views, which offer
more complex capabilities.

Cheers,

Derek

On Thu, Aug 25, 2022, 3:11 PM Benedict  wrote:

> I’m inclined to agree that this seems a more straightforward approach that
> makes fewer implied promises.
>
> Perhaps we could deliver simple views backed by virtual tables, and model
> our approach on that of Postgres, MySQL et al?
>
> Views in C* would be very simple, just offering a subset of fields with
> some UDFs applied. It would allow users to define roles with access only to
> the views, or for applications to use the views for presentation purposes.
>
> It feels like a cleaner approach to me, and we’d get two features for the
> price of one. BUT I don’t feel super strongly about this.
>
> On 25 Aug 2022, at 20:16, Derek Chen-Becker  wrote:
>
> 
> To make sure I understand, if I wanted to use a masked column for a
> conditional update, you're saying we would need SELECT_MASKED to use it in
> the IF clause? I worry that this proposal is increasing in complexity; I
> would actually be OK starting with something smaller in scope. Perhaps just
> providing the masking functions and not tying masking to schema would be
> sufficient for an initial goal? That wouldn't preclude additional
> permissions, schema integration, or perhaps just plain Views in the future.
>
> Cheers,
>
> Derek
>
> On Thu, Aug 25, 2022 at 11:12 AM Andrés de la Peña 
> wrote:
>
>> I have modified the proposal adding a new SELECT_MASKED permission. Using
>> masked columns on WHERE/IF clauses would require having SELECT and either
>> UNMASK or SELECT_MASKED permissions. Seeing the unmasked values in the
>> query results would always require both SELECT and UNMASK.
>>
>> This way we can have the best of both worlds, allowing admins to decide
>> whether they trust their immediate users or not. wdyt?
>>
>> On Wed, 24 Aug 2022 at 16:06, Henrik Ingo 
>> wrote:
>>
>>> This is the difference between security and compliance I guess :-D
>>>
>>> The way I see this, the attacker or threat in this concept is not the
>>> developer with access to the database. Rather a feature like this is just a
>>> convenient way to apply some masking rule in a centralized way. The
>>> protection is against an end user of the application, who should not be
>>> able to see the personal data of someone else. Or themselves, even. As long
>>> as the application end user doesn't have access to run arbitrary CQL, then
>>> these frorms of masking prevent accidental unauthorized use/leaking of
>>> personal data.
>>>
>>> henrik
>>>
>>>
>>>
>>> On Wed, Aug 24, 2022 at 10:40 AM Benedict  wrote:
>>>
>>>> Is it typical for a masking feature to make no effort to prevent
>>>> unmasking? I’m just struggling to see the value of this without such
>>>> mechanisms. Otherwise it’s just a default formatter, and we should consider
>>>> renaming the feature IMO
>>>>
>>>> On 23 Aug 2022, at 21:27, Andrés de la Peña 
>>>> wrote:
>>>>
>>>> 
>>>> As mentioned in the CEP document, dynamic data masking doesn't try to
>>>> prevent malicious users with SELECT permissions to indirectly guess the
>>>> real value of the masked value. This can easily be done by just trying
>>>> values on the WHERE clause of SELECT queries. DDM would not be a
>>>> replacement for proper column-level permissions.
>>>>
>>>> The data served by the database is usually consumed by applications
>>>> that present this data to end users. These end users are not necessarily
>>>> the users directly connecting to the database. With DDM, it would be easy
>>>> for applications to mask sensitive data that is going to be consumed by the
>>>> end users. However, the users directly connecting to the database should be
>>>> trusted, provided that they have the right SELECT permissions.
>>>>
>>>> In other words, DDM doesn't directly protect the data, but it eases the
>>>>

Re: [DISCUSS] CEP-20: Dynamic Data Masking

[Derek Chen-Becker]

To make sure I understand, if I wanted to use a masked column for a
conditional update, you're saying we would need SELECT_MASKED to use it in
the IF clause? I worry that this proposal is increasing in complexity; I
would actually be OK starting with something smaller in scope. Perhaps just
providing the masking functions and not tying masking to schema would be
sufficient for an initial goal? That wouldn't preclude additional
permissions, schema integration, or perhaps just plain Views in the future.

Cheers,

Derek

On Thu, Aug 25, 2022 at 11:12 AM Andrés de la Peña 
wrote:

> I have modified the proposal adding a new SELECT_MASKED permission. Using
> masked columns on WHERE/IF clauses would require having SELECT and either
> UNMASK or SELECT_MASKED permissions. Seeing the unmasked values in the
> query results would always require both SELECT and UNMASK.
>
> This way we can have the best of both worlds, allowing admins to decide
> whether they trust their immediate users or not. wdyt?
>
> On Wed, 24 Aug 2022 at 16:06, Henrik Ingo 
> wrote:
>
>> This is the difference between security and compliance I guess :-D
>>
>> The way I see this, the attacker or threat in this concept is not the
>> developer with access to the database. Rather a feature like this is just a
>> convenient way to apply some masking rule in a centralized way. The
>> protection is against an end user of the application, who should not be
>> able to see the personal data of someone else. Or themselves, even. As long
>> as the application end user doesn't have access to run arbitrary CQL, then
>> these frorms of masking prevent accidental unauthorized use/leaking of
>> personal data.
>>
>> henrik
>>
>>
>>
>> On Wed, Aug 24, 2022 at 10:40 AM Benedict  wrote:
>>
>>> Is it typical for a masking feature to make no effort to prevent
>>> unmasking? I’m just struggling to see the value of this without such
>>> mechanisms. Otherwise it’s just a default formatter, and we should consider
>>> renaming the feature IMO
>>>
>>> On 23 Aug 2022, at 21:27, Andrés de la Peña 
>>> wrote:
>>>
>>> 
>>> As mentioned in the CEP document, dynamic data masking doesn't try to
>>> prevent malicious users with SELECT permissions to indirectly guess the
>>> real value of the masked value. This can easily be done by just trying
>>> values on the WHERE clause of SELECT queries. DDM would not be a
>>> replacement for proper column-level permissions.
>>>
>>> The data served by the database is usually consumed by applications that
>>> present this data to end users. These end users are not necessarily the
>>> users directly connecting to the database. With DDM, it would be easy for
>>> applications to mask sensitive data that is going to be consumed by the end
>>> users. However, the users directly connecting to the database should be
>>> trusted, provided that they have the right SELECT permissions.
>>>
>>> In other words, DDM doesn't directly protect the data, but it eases the
>>> production of protected data.
>>>
>>> Said that, we could later go one step ahead and add a way to prevent
>>> untrusted users from inferring the masked data. That could be done adding a
>>> new permission required to use certain columns on WHERE clauses, different
>>> to the current SELECT permission. That would play especially well with
>>> column-level permissions, which is something that we still have pending.
>>>
>>> On Tue, 23 Aug 2022 at 19:13, Aaron Ploetz 
>>> wrote:
>>>
>>>> Applying this should prevent querying on a field, else you could leak
>>>>> its contents, surely?
>>>>>
>>>>
>>>> In theory, yes.  Although I could see folks doing something like this:
>>>>
>>>> SELECT COUNT(*) FROM patients
>>>> WHERE year_of_birth = 2002
>>>> AND date_of_birth >= '2002-04-01'
>>>> AND date_of_birth < '2002-11-01';
>>>>
>>>> In this case, the rows containing the masked key column(s) could be
>>>> filtered on without revealing the actual data.  But again, that's probably
>>>> better for a "phase 2" of the implementation.
>>>>
>>>> Agreed on not being a queryable field. That would also preclude
>>>>> secondary indexing, right?
>>>>
>>>>
>>>> Yes, that's my thought as well.
>>>>
>>>> On Tue, Aug 23, 2022 at 12:42 PM Derek Chen-Becker <
>>>> de...@chen-becker.o

Re: [DISCUSS] CEP-20: Dynamic Data Masking

[Derek Chen-Becker]

Agreed on not being a queryable field. That would also preclude secondary
indexing, right?

On Tue, Aug 23, 2022 at 11:20 AM Benedict  wrote:

> Applying this should prevent querying on a field, else you could leak its
> contents, surely? This pretty much prohibits using it in a clustering key,
> and a partition key with the ordered partitioner - but probably also a
> hashed partitioner since we do not use a cryptographic hash and the hash
> function is well defined.
>
> We probably also need to ensure that any ALLOW FILTERING queries on such a
> field are disabled.
>
> Plausibly the data could be cryptographically jumbled before using it in a
> primary key component (or permitting filtering), but it is probably easier
> and safer to exclude for now…
>
> On 23 Aug 2022, at 18:13, Aaron Ploetz  wrote:
>
> 
> Some thoughts on this one:
>
> In a prior job, we'd give app teams access to a single keyspace, and two
> roles: a read-write role and a read-only role.  In some cases, a
> "privileged" application role was also requested.  Depending on the
> requirements, I could see the UNMASK permission being applied to the RW or
> privileged roles.  But if there's a problem on the table and the operators
> go in to investigate, they will likely use a SUPERUSER account, and they'll
> see that data.
>
> How hard would it be for SUPERUSERs to *not* automatically get the UNMASK
> permission?
>
> I'll also echo the concerns around masking primary key components.  It's
> highly likely that certain personal data properties would be used as a
> partition or clustering key (ex: range query for people born within a
> certain timeframe).  In addition to the "breaks existing" concern, I'm
> curious about the challenges around getting that to work with the current
> primary key implementation.
>
> Does this first implementation only apply to payload (non-key) columns?
> The examples in the CEP currently do not show primary key components being
> masked.
>
> Thanks,
>
> Aaron
>
>
> On Tue, Aug 23, 2022 at 6:44 AM Henrik Ingo 
> wrote:
>
>> On Tue, Aug 23, 2022 at 1:10 PM Andrés de la Peña 
>> wrote:
>>
>>> One thought: The way the CEP is currently written, it is only possible
>>>> to mask a column one way. You can only define one masking function for a
>>>> column, and since you use the original column name, you could only return
>>>> one version of it in the result set, even if you had a way to define
>>>> several functions.
>>>>
>>>
>>> Right, it's one single type of mapping per the column, declared on
>>> CREATE/ALTER TABLE statements. Also, users can manually specify their own
>>> masking function in SELECT statements if they have permissions for seeing
>>> the clear data.
>>>
>>> For those cases where the data is automatically masked for an
>>> unprivileged user, I don't see the use of including different types of
>>> masking for the same column into the same result set. Instead, we might be
>>> interested on having different types of masking associated to different
>>> roles. We could do so with dedicated CREATE/DROP/LIST MASK statements,
>>> instead of using the CREATE/ALTER/DESCRIBE TABLE statements. That CREATE
>>> MASK statement would associate a masking function to a column and role.
>>> However, I'm not sure we need that type of granularity instead of the
>>> simplicity of attaching the masking to the column declaration. wdyt?
>>>
>>>
>>>
>> My gut feeling likewise is that this adds complexity but little value.
>>
>>>
>>>>
>>
>> --
>>
>> Henrik Ingo
>>
>> +358 40 569 7354 <358405697354>
>>
>> [image: Visit us online.] <https://www.datastax.com/>  [image: Visit us
>> on Twitter.] <https://twitter.com/DataStaxEng>  [image: Visit us on
>> YouTube.]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_channel_UCqA6zOSMpQ55vvguq4Y0jAg=DwMFaQ=adz96Xi0w1RHqtPMowiL2g=IFj3MdIKYLLXIUhYdUGB0cTzTlxyCb7_VUmICBaYilU=bmIfaie9O3fWJAu6lESvWj3HajV4VFwgwgVuKmxKZmE=16sY48_kvIb7sRQORknZrr3V8iLTfemFKbMVNZhdwgw=>
>>   [image: Visit my LinkedIn profile.]
>> <https://www.linkedin.com/in/heingo/>
>>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSS] CEP-21: Transactional Cluster Metadata

[Derek Chen-Becker]

On Tue, Aug 23, 2022 at 2:51 AM Sam Tunnicliffe  wrote:

>
> Regular read/write operations should not be halted, even by a total
> failure of the metadata service. There should be no situations where the a
> previously stable database becomes entirely unavailable due to a CMS
> failure. The worst case is where there is some unavailability due to
> permanent failure of multiple nodes where those nodes happen to represent a
> majority of the CMS. In this scenario, the CMS would need to be recovered
> before the down nodes could be replaced, so it's possible it would extend
> the period of unavailabilty, though not necessarily by much.
>

This seems like a reasonable tradeoff. The current approach tries to
achieve better availability at the risk of loss of consistency, but even
then has failure modes that require manual intervention. What I really like
about the proposal is that it's a path toward separating responsibilities
of the cluster into well-defined sub-services.

Cheers,

Derek




>
>
> On 23 Aug 2022, at 05:42, Jeff Jirsa  wrote:
>
> “ The proposed mechanism for dealing with both of these failure types is
> to enable a manual operator override mode. This would allow operators to
> inject metadata changes (potentially overriding the complete metadata
> state) directly on any and all nodes in a cluster. At the most extreme end
> of the spectrum, this could allow an unrecoverably corrupt state to be
> rectified by composing a custom snapshot of cluster metadata and uploading
> it to all nodes in the cluster”
>
> What do you expect this to look like in practice? JSON representation of
> the ring? Would reads and writes have halted? In what situations would the
> database be entirely unavailable?
>
>
>
> On Aug 22, 2022, at 11:15 AM, Derek Chen-Becker 
> wrote:
>
> 
> This looks really interesting; thanks for putting this together! Just so
> I'm clear on CEP nomenclature, having external management of metadata as a
> non-goal doesn't preclude some future use, correct? Coincidentally, I'm
> working on my ApacheCon talk on improving modularity in Cassandra and one
> of the ideas I'm discussing is pluggably (?) replacing gossip with
> something(s) that allow us to externalize some of the complexity of
> maintaining consistency. I need to digest the proposal you've made, but I
> don't see the two ideas being at odds on my first read.
>
> Cheers,
>
> Derek
>
> On Mon, Aug 22, 2022 at 6:45 AM Sam Tunnicliffe  wrote:
>
>> Hi,
>>
>> I'd like to open discussion about this CEP:
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-21%3A+Transactional+Cluster+Metadata
>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-21:+Transactional+Cluster+Metadata>
>>
>> Cluster metadata in Cassandra comprises a number of disparate elements
>> including, but not limited to, distributed schema, topology and token
>> ownership. Following the general design principles of Cassandra, the
>> mechanisms for coordinating updates to cluster state have favoured eventual
>> consistency, with probabilisitic delivery via gossip being a prime example.
>> Undoubtedly, this approach has benefits, not least in terms of resilience,
>> particularly in highly fluid distributed environments. However, this is not
>> the reality of most Cassandra deployments, where the total number of nodes
>> is relatively small (i.e. in the low thousands) and the rate of change
>> tends to be low.
>>
>> Historically, a significant proportion of issues affecting operators and
>> users of Cassandra have been due, at least in part, to a lack of strongly
>> consistent cluster metadata. In response to this, we propose a design which
>> aims to provide linearizability of metadata changes whilst ensuring that
>> the effects of those changes are made visible to all nodes in a strongly
>> consistent manner. At its core, it is also pluggable, enabling
>> Cassandra-derived projects to supply their own implementations if desired.
>> In addition to the CEP document itself, we aim to publish a working
>> prototype of the proposed design. Obviously, this does not implement the
>> entire proposal and there are several parts which remain only partially
>> complete. It does include the core of the system, including a good deal of
>> test infrastructure, so may serve as both illustration of the design and a
>> starting point for real implementation.
>>
>>
>
> --
> +---+
> | Derek Chen-Becker |
> | GPG Key available at https://keybase.io/dchenbecker and   |
> | https://pgp.mit.edu/pks/lookup?search=d

Re: [DISCUSS] CEP-21: Transactional Cluster Metadata

[Derek Chen-Becker]

This looks really interesting; thanks for putting this together! Just so
I'm clear on CEP nomenclature, having external management of metadata as a
non-goal doesn't preclude some future use, correct? Coincidentally, I'm
working on my ApacheCon talk on improving modularity in Cassandra and one
of the ideas I'm discussing is pluggably (?) replacing gossip with
something(s) that allow us to externalize some of the complexity of
maintaining consistency. I need to digest the proposal you've made, but I
don't see the two ideas being at odds on my first read.

Cheers,

Derek

On Mon, Aug 22, 2022 at 6:45 AM Sam Tunnicliffe  wrote:

> Hi,
>
> I'd like to open discussion about this CEP:
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-21%3A+Transactional+Cluster+Metadata
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-21:+Transactional+Cluster+Metadata>
>
> Cluster metadata in Cassandra comprises a number of disparate elements
> including, but not limited to, distributed schema, topology and token
> ownership. Following the general design principles of Cassandra, the
> mechanisms for coordinating updates to cluster state have favoured eventual
> consistency, with probabilisitic delivery via gossip being a prime example.
> Undoubtedly, this approach has benefits, not least in terms of resilience,
> particularly in highly fluid distributed environments. However, this is not
> the reality of most Cassandra deployments, where the total number of nodes
> is relatively small (i.e. in the low thousands) and the rate of change
> tends to be low.
>
> Historically, a significant proportion of issues affecting operators and
> users of Cassandra have been due, at least in part, to a lack of strongly
> consistent cluster metadata. In response to this, we propose a design which
> aims to provide linearizability of metadata changes whilst ensuring that
> the effects of those changes are made visible to all nodes in a strongly
> consistent manner. At its core, it is also pluggable, enabling
> Cassandra-derived projects to supply their own implementations if desired.
> In addition to the CEP document itself, we aim to publish a working
> prototype of the proposed design. Obviously, this does not implement the
> entire proposal and there are several parts which remain only partially
> complete. It does include the core of the system, including a good deal of
> test infrastructure, so may serve as both illustration of the design and a
> starting point for real implementation.
>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Inclusive/exclusive endpoints when compacting token ranges

[Derek Chen-Becker]

+1 to new flags. A released, albeit undocumented, behavior is still a
contract with the end user. Flags (and documentation) seem like the right
path to address the situation.

Cheers,

Derek

On Tue, Jul 26, 2022 at 7:28 AM Benedict Elliott Smith 
wrote:

>
> I think a change like this could be dangerous for a lot of existing
> automation built atop nodetool.
>
> I’m not sure this change is worthwhile. I think it would be better to
> introduce e.g. -ste and -ete for “start token exclusive” and “end token
> exclusive” so that users can opt-in to whichever scheme they prefer for
> their tooling, without breaking existing users.
>
> > On 26 Jul 2022, at 14:22, Brandon Williams  wrote:
> >
> > +1, I think that makes the most sense.
> >
> > Kind Regards,
> > Brandon
> >
> > On Tue, Jul 26, 2022 at 8:19 AM J. D. Jordan 
> wrote:
> >>
> >> I like the third option, especially if it makes it consistent with
> repair, which has supported ranges longer and I would guess most people
> would think the compact ranges work the same as the repair ranges.
> >>
> >> -Jeremiah Jordan
> >>
> >>> On Jul 26, 2022, at 6:49 AM, Andrés de la Peña 
> wrote:
> >>>
> >>> 
> >>> Hi all,
> >>>
> >>> CASSANDRA-17575 has detected that token ranges in nodetool compact are
> interpreted as closed on both sides. For example, the command "nodetool
> compact -st 10 -et 50" will compact the tokens in [10, 50]. This way of
> interpreting token ranges is unusual since token ranges are usually
> half-open, and I think that in the previous example one would expect that
> the compacted tokens would be in (10, 50]. That's for example the way
> nodetool repair works, and indeed the class org.apache.cassandra.dht.Range
> is always half-open.
> >>>
> >>> It's worth mentioning that, differently from nodetool repair, the help
> and doc for nodetool compact doesn't specify whether the supplied start/end
> tokens are inclusive or exclusive.
> >>>
> >>> I think that ideally nodetool compact should interpret the provided
> token ranges as half-open, to be consistent with how token ranges are
> usually interpreted. However, this would change the way the tool has worked
> until now. This change might be problematic for existing users relying on
> the old behaviour. That would be especially severe for the case where the
> begin and end token are the same, because interpreting [x, x] we would
> compact a single token, whereas I think that interpreting (x, x] would
> compact all the tokens. As for compacting ranges including multiple tokens,
> I think the change wouldn't be so bad, since probably the supplied token
> ranges come from tools that are already presenting the ranges as half-open.
> Also, if we are splitting the full ring into smaller ranges, half-open
> intervals would still work and would save us some repetitions.
> >>>
> >>> So my question is: Should we change the behaviour of nodetool compact
> to interpret the token ranges as half-opened, aligning it with the usual
> interpretation of ranges? Or should we just document the current odd
> behaviour to prevent compatibility issues?
> >>>
> >>> A third option would be changing to half-opened ranges and also
> forbidding ranges where the begin and end token are the same, to prevent
> the accidental compaction of the entire ring. Note that nodetool repair
> also forbids this type of token ranges.
> >>>
> >>> What do you think?
>
>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSS] CASSANDRA-17750: Security migration away from Maven Ant Tasks

[Derek Chen-Becker]

I guess dependency management is circular like fashion :) Are the concerns
enumerated in that ticket still valid today? It looks like the makepom
command can take a template for the POM, so that might be a way to deal
with inconsistencies?

Cheers,

Derek

On Tue, Jul 19, 2022 at 2:35 PM Brandon Williams  wrote:

> Ivy is actually how we got to MAT:
> https://issues.apache.org/jira/browse/CASSANDRA-2017
>
> Kind Regards,
> Brandon
>
> On Tue, Jul 19, 2022 at 3:33 PM Derek Chen-Becker 
> wrote:
> >
> > Sorry, I put a comment about this in the PR before seeing this. I think
> if Ivy fits better with Ant, is more compact, and can do everything that we
> were using MAT for, then that's a reasonable path forward. I don't think
> Ivy syntax for dependencies will be foreign to anyone familiar with Maven.
> >
> > Derek
> >
> > On Tue, Jul 19, 2022 at 2:03 PM Mick Semb Wever  wrote:
> >>
> >>
> >>
> >> Rehashing some of the aspects raised by the PR…
> >>
> >>
> >>>
> >>> 1. Is it worth addressing this CVE and retired dependency with changes
> to our build system, or should we suppress it?
> >>
> >>
> >>
> >> If we are not exposed to the CVE then it should be considered
> suppressed.
> >> While this might address (remove) the urgency of the matter, it is not
> an argument against replacing and improving a deprecated and unmaintained
> dependency.
> >>
> >>
> >>
> >>>
> >>> 2. Are there more alternatives to Maven Ant Tasks that should be
> considered, like Ivy?
> >>
> >>
> >>
> >> The question here is… If we are to replace MARAT, then *what*
> dependency framework/format do we want to work with moving forward?
> >>
> >> The choices are:
> >>  - maven
> >>  - ivy
> >>  - gradle
> >>
> >> Note this is ONLY for dependency management, and is only about the
> replacement for this section:
> https://github.com/apache/cassandra/blob/315a1a7/build.xml#L507-L873
> >>
> >> It is a requirement that whatever framework/format we choose it can
> generated into the pom(s) we publish via repository.apache.org
> >> For example maven pom files would be used directly, ivy could use the
> `makepom` command and gradle the `maven-publish` plugin.
> >>
> >> Ivy and Gradle provide more compact dependency declarations, Ivy fits
> in better with Ant, and most are familiar with Maven (and it would avoid
> the generation step).
> >>
> >> What is the best fit for us moving forward?
> >>
> >>
> >>
> >>
> >
> >
> >
> > --
> > +---+
> > | Derek Chen-Becker |
> > | GPG Key available at https://keybase.io/dchenbecker and   |
> > | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
> > | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
> > +---+
> >
>


-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSS] CASSANDRA-17750: Security migration away from Maven Ant Tasks

[Derek Chen-Becker]

Sorry, I put a comment about this in the PR before seeing this. I think if
Ivy fits better with Ant, is more compact, and can do everything that we
were using MAT for, then that's a reasonable path forward. I don't think
Ivy syntax for dependencies will be foreign to anyone familiar with Maven.

Derek

On Tue, Jul 19, 2022 at 2:03 PM Mick Semb Wever  wrote:

>
>
> Rehashing some of the aspects raised by the PR…
>
>
>
>> 1. Is it worth addressing this CVE and retired dependency with changes to
>> our build system, or should we suppress it?
>>
>
>
> If we are not exposed to the CVE then it should be considered suppressed.
> While this might address (remove) the urgency of the matter, it is not an
> argument against replacing and improving a deprecated and unmaintained
> dependency.
>
>
>
>
>> 2. Are there more alternatives to Maven Ant Tasks that should be
>> considered, like Ivy?
>>
>
>
> The question here is… If we are to replace MARAT, then *what* dependency
> framework/format do we want to work with moving forward?
>
> The choices are:
>  - maven
>  - ivy
>  - gradle
>
> Note this is ONLY for dependency management, and is only about the
> replacement for this section:
> https://github.com/apache/cassandra/blob/315a1a7/build.xml#L507-L873
>
> It is a requirement that whatever framework/format we choose it can
> generated into the pom(s) we publish via repository.apache.org
> For example maven pom files would be used directly, ivy could use the
> `makepom` command and gradle the `maven-publish` plugin.
>
> Ivy and Gradle provide more compact dependency declarations, Ivy fits in
> better with Ant, and most are familiar with Maven (and it would avoid the
> generation step).
>
> What is the best fit for us moving forward?
>
>
>
>
>


-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: CEP-15 multi key transaction syntax

[Derek Chen-Becker]

OK, that makes sense. One of the examples in an earlier email had duplicate
"LET miles =" so I was confused. I think failing in the face of ambiguous
identifiers is going to be more friendly by not requiring LET for every
field you might want to use, and we can provide a very clear error message
in that case.

Cheers,

Derek

On Tue, Jun 14, 2022 at 2:40 PM bened...@apache.org 
wrote:

> To be clear, the concerning situation is
>
>
>
> BEGIN TRANSACTION
>
>   LET miles = miles_driven, running=is_running FROM cars WHERE
> model=’pinto’
>
>   IF running THEN
>
> UPDATE cars SET miles_driven = miles + 30 WHERE model='pinto';
>
>   END IF
>
> COMMIT TRANSACTION
>
>
>
> But where there’s some additional column also called *miles* in *cars*
>
>
>
>
>
> *From: *bened...@apache.org 
> *Date: *Tuesday, 14 June 2022 at 21:37
> *To: *dev@cassandra.apache.org 
> *Subject: *Re: CEP-15 multi key transaction syntax
>
> Duplicate declarations are usually rejected by languages, so I think
> that’s fine?
>
>
>
> Option 1 would involve something like
>
>
>
> BEGIN TRANSACTION
>
>   LET car_miles = miles_driven, running=is_running FROM cars WHERE
> model=’pinto’
>
>   LET user_miles = miles_driven FROM users WHERE name=’blake’
>
>   SELECT running, car_miles, user_miles
>
>   IF running THEN
>
> UPDATE users SET miles_driven = user_miles + 30 WHERE name='blake';
> UPDATE cars SET miles_driven = car_miles + 30 WHERE model='pinto';
>
>   END IF
>
> COMMIT TRANSACTION
>
>
>
>
>
>
>
> *From: *Derek Chen-Becker 
> *Date: *Tuesday, 14 June 2022 at 21:27
> *To: *dev@cassandra.apache.org 
> *Subject: *Re: CEP-15 multi key transaction syntax
>
> Just to make sure I'm understanding correctly, I've been thinking of LET
> like a variable declaration and assignment, but is that the right mental
> model? For example, this is a valid statement:
>
>
>
> BEGIN TRANSACTION
>
>   LET miles = miles_driven, running=is_running FROM cars WHERE
> model=’pinto’
>
>   SELECT running, miles   # let the user know if the transaction takes any
> action
>
>   IF running THEN
>
> UPDATE users SET miles_driven = miles_driven + 30 WHERE name='blake';
> UPDATE cars SET miles_driven = miles_driven + 30 WHERE model='pinto';
>
>   END IF
>
> COMMIT TRANSACTION
>
>
>
> But this isn't, because we're trying to bind to "miles" twice
>
>
>
> BEGIN TRANSACTION
>
>   LET miles = miles_driven, running=is_running FROM cars WHERE
> model=’pinto’
>
>   LET miles = miles_driven FROM users WHERE name=’blake’ # duplicate
> binding for "miles"
>
>   SELECT running, miles   # let the user know if the transaction takes any
> action
>
>   IF running THEN
>
> UPDATE users SET miles_driven = miles_driven + 30 WHERE name='blake';
> UPDATE cars SET miles_driven = miles_driven + 30 WHERE model='pinto';
>
>   END IF
>
> COMMIT TRANSACTION
>
>
>
> I think that's option #1, but I'm a little confused now that I'm looking
> at some of the examples.
>
>
>
> Cheers,
>
>
>
> Derek
>
>
>
> On Tue, Jun 14, 2022 at 1:58 PM bened...@apache.org 
> wrote:
>
> It sounds like we’re zeroing in on a solution.
>
>
>
> To draw attention back to Jon’s email, I think the last open question at
> this point is the scope of identifiers declared by LET, and how we handle
> name clashes with table columns in an UPDATE.
>
>
>
> I think we have basically two options:
>
>
>
> 1. Require LET for all input parameters to an assignment in UPDATE
>
> 2. Add some additional syntax to local variables to identify them, e.g.
> 
>
>
>
> Any other ideas?
>
>
>
>
>
>
>
> *From: *Derek Chen-Becker 
> *Date: *Tuesday, 14 June 2022 at 20:31
> *To: *dev@cassandra.apache.org 
> *Subject: *Re: CEP-15 multi key transaction syntax
>
> Sorry, that was in reference to the "Would you require a LIMIT 1 clause if
> the key did not fully specify a row?" question, so I think we're in
> agreement here.
>
>
>
> Cheers,
>
>
>
> Derek
>
>
>
> On Tue, Jun 14, 2022 at 1:27 PM bened...@apache.org 
> wrote:
>
> > It seems like we would want to start with restrictions on number of
> rows, uniqueness, homogeneity of results, etc
>
>
>
> I am not keen on any hard limit on the number of rows, I anticipate a
> configurable guardrail for rejecting queries that are too expensive. I
> think the normal CQL restrictions are likely to apply (must include
> partition key), plus (initially) no range scans, and th

Re: CEP-15 multi key transaction syntax

[Derek Chen-Becker]

Just to make sure I'm understanding correctly, I've been thinking of LET
like a variable declaration and assignment, but is that the right mental
model? For example, this is a valid statement:

BEGIN TRANSACTION

  LET miles = miles_driven, running=is_running FROM cars WHERE model=’pinto’
  SELECT running, miles   # let the user know if the transaction takes any
action

  IF running THEN

UPDATE users SET miles_driven = miles_driven + 30 WHERE name='blake';
UPDATE cars SET miles_driven = miles_driven + 30 WHERE model='pinto';

  END IF

COMMIT TRANSACTION

But this isn't, because we're trying to bind to "miles" twice

BEGIN TRANSACTION

  LET miles = miles_driven, running=is_running FROM cars WHERE model=’pinto’

  LET miles = miles_driven FROM users WHERE name=’blake’ # duplicate
binding for "miles"

  SELECT running, miles   # let the user know if the transaction takes any
action

  IF running THEN

UPDATE users SET miles_driven = miles_driven + 30 WHERE name='blake';
UPDATE cars SET miles_driven = miles_driven + 30 WHERE model='pinto';

  END IF

COMMIT TRANSACTION


I think that's option #1, but I'm a little confused now that I'm looking at
some of the examples.


Cheers,


Derek

On Tue, Jun 14, 2022 at 1:58 PM bened...@apache.org 
wrote:

> It sounds like we’re zeroing in on a solution.
>
>
>
> To draw attention back to Jon’s email, I think the last open question at
> this point is the scope of identifiers declared by LET, and how we handle
> name clashes with table columns in an UPDATE.
>
>
>
> I think we have basically two options:
>
>
>
> 1. Require LET for all input parameters to an assignment in UPDATE
>
> 2. Add some additional syntax to local variables to identify them, e.g.
> 
>
>
>
> Any other ideas?
>
>
>
>
>
>
>
> *From: *Derek Chen-Becker 
> *Date: *Tuesday, 14 June 2022 at 20:31
> *To: *dev@cassandra.apache.org 
> *Subject: *Re: CEP-15 multi key transaction syntax
>
> Sorry, that was in reference to the "Would you require a LIMIT 1 clause if
> the key did not fully specify a row?" question, so I think we're in
> agreement here.
>
>
>
> Cheers,
>
>
>
> Derek
>
>
>
> On Tue, Jun 14, 2022 at 1:27 PM bened...@apache.org 
> wrote:
>
> > It seems like we would want to start with restrictions on number of
> rows, uniqueness, homogeneity of results, etc
>
>
>
> I am not keen on any hard limit on the number of rows, I anticipate a
> configurable guardrail for rejecting queries that are too expensive. I
> think the normal CQL restrictions are likely to apply (must include
> partition key), plus (initially) no range scans, and the aforementioned
> restrictions on what order statements must occur in the transaction.
>
>
>
>
>
> *From: *Derek Chen-Becker 
> *Date: *Tuesday, 14 June 2022 at 18:42
> *To: *dev@cassandra.apache.org 
> *Subject: *Re: CEP-15 multi key transaction syntax
>
> "MIXED" means, "hey, this might not be my standard PGSQL transaction" :)
>
>
>
> I do think that surprise is a meaningful measure, from the perspective of
> an individual developer coming to Cassandra from any arbitrary RDBMS. My
> own experience is that a non-trivial number of developers are essentially
> blindly following guidance given to them by someone else when it comes to
> features like transactions, so making syntax that looks superficially
> similar to SQL transactions but acts subtly different (or uses slightly
> different syntax) is going to be surprising. I think we get diminishing
> marginal returns on "it looks just like SQL!" when we start to venture
> further into territory where even different RDMBSs disagree. I would rather
> use some syntax that is clearly Cassandra-specific, even if the structure
> would be similar to a SQL transaction, just to ensure that developers
> understand that it's different and actually look at the docs.
>
>
>
> I completely agree on focusing on clarity and consistency, and I think
> considering how we think it might evolve is good, but that can't be an
> open-ended exercise. My primary concern is how we can start getting
> incremental improvements into end users' hands more quickly, since the
> alternative right now is to basically roll your own, right?
>
>
>
> Cheers,
>
>
>
> Derek
>
>
>
> On Mon, Jun 13, 2022 at 4:16 PM bened...@apache.org 
> wrote:
>
> What on earth does MIXED mean?
>
>
>
> I agree with the sentiment we should minimise surprise, but everyone is
> surprised differently so it becomes a sort of pointless rubrik, everyone
> claiming it supports their view. I think it is only useful in cases where
> there is clear agreement that so

Re: CEP-15 multi key transaction syntax

[Derek Chen-Becker]

Sorry, that was in reference to the "Would you require a LIMIT 1 clause if
the key did not fully specify a row?" question, so I think we're in
agreement here.

Cheers,

Derek

On Tue, Jun 14, 2022 at 1:27 PM bened...@apache.org 
wrote:

> > It seems like we would want to start with restrictions on number of
> rows, uniqueness, homogeneity of results, etc
>
>
>
> I am not keen on any hard limit on the number of rows, I anticipate a
> configurable guardrail for rejecting queries that are too expensive. I
> think the normal CQL restrictions are likely to apply (must include
> partition key), plus (initially) no range scans, and the aforementioned
> restrictions on what order statements must occur in the transaction.
>
>
>
>
>
> *From: *Derek Chen-Becker 
> *Date: *Tuesday, 14 June 2022 at 18:42
> *To: *dev@cassandra.apache.org 
> *Subject: *Re: CEP-15 multi key transaction syntax
>
> "MIXED" means, "hey, this might not be my standard PGSQL transaction" :)
>
>
>
> I do think that surprise is a meaningful measure, from the perspective of
> an individual developer coming to Cassandra from any arbitrary RDBMS. My
> own experience is that a non-trivial number of developers are essentially
> blindly following guidance given to them by someone else when it comes to
> features like transactions, so making syntax that looks superficially
> similar to SQL transactions but acts subtly different (or uses slightly
> different syntax) is going to be surprising. I think we get diminishing
> marginal returns on "it looks just like SQL!" when we start to venture
> further into territory where even different RDMBSs disagree. I would rather
> use some syntax that is clearly Cassandra-specific, even if the structure
> would be similar to a SQL transaction, just to ensure that developers
> understand that it's different and actually look at the docs.
>
>
>
> I completely agree on focusing on clarity and consistency, and I think
> considering how we think it might evolve is good, but that can't be an
> open-ended exercise. My primary concern is how we can start getting
> incremental improvements into end users' hands more quickly, since the
> alternative right now is to basically roll your own, right?
>
>
>
> Cheers,
>
>
>
> Derek
>
>
>
> On Mon, Jun 13, 2022 at 4:16 PM bened...@apache.org 
> wrote:
>
> What on earth does MIXED mean?
>
>
>
> I agree with the sentiment we should minimise surprise, but everyone is
> surprised differently so it becomes a sort of pointless rubrik, everyone
> claiming it supports their view. I think it is only useful in cases where
> there is clear agreement that something is surprising, but unhelpful when
> choosing between subtle variations on approach.
>
>
>
> The main goal IMO should be clarity and consistency, so that the user can
> reason about the constructs easily, and so we can evolve them.
>
>
>
> For instance, we should be sure to consider how the syntax will look if we
> **do** offer interactive transactions, or JOINs, or anything else we
> might add in future.
>
>
>
>
>
> *From: *Derek Chen-Becker 
> *Date: *Monday, 13 June 2022 at 23:09
> *To: *dev@cassandra.apache.org 
> *Subject: *Re: CEP-15 multi key transaction syntax
>
> On Mon, Jun 13, 2022 at 1:57 PM Blake Eggleston 
> wrote:
>
> I prefer an approach that supports an accurate mental model of what’s
> happening behind the scenes. I think that should be a design priority for
> the syntax. We’ll be able to build things on top of accord, but the core
> multi-key cas operation isn’t going to change too much.
>
>
>
> +1, the principle of least surprise tells me that if this doesn't behave
> exactly like SQL transactions (for whatever SQL actually means), it could
> be more clear to not try and emulate it halfway
>
>
>
> BEGIN MIXED TRANSACTION?
>
>
>
> Derek
>
>
>
>
>
>
>
> On Jun 13, 2022, at 12:14 PM, Blake Eggleston 
> wrote:
>
>
>
> Does the IF <...> ABORT simplify reasoning though? If you restrict it to
> only dealing with the most recent row it would, but referencing the name
> implies you’d be able to include references from other operations, in which
> case you’d have the same problem.
>
> > return instead an exception if the transaction is aborted
>
> Since the txn is not actually interactive, I think it would be better to
> receive values instead of an excetion, to understand why the operation was
> rolled back.
>
>
>
> On Jun 13, 2022, at 10:32 AM, Aaron Ploetz  wrote:
>
>
>
> Benedict,
>
>
>
> I'm really excited about this feature.  I've been observing 

Re: CEP-15 multi key transaction syntax

[Derek Chen-Becker]

"MIXED" means, "hey, this might not be my standard PGSQL transaction" :)

I do think that surprise is a meaningful measure, from the perspective of
an individual developer coming to Cassandra from any arbitrary RDBMS. My
own experience is that a non-trivial number of developers are essentially
blindly following guidance given to them by someone else when it comes to
features like transactions, so making syntax that looks superficially
similar to SQL transactions but acts subtly different (or uses slightly
different syntax) is going to be surprising. I think we get diminishing
marginal returns on "it looks just like SQL!" when we start to venture
further into territory where even different RDMBSs disagree. I would rather
use some syntax that is clearly Cassandra-specific, even if the structure
would be similar to a SQL transaction, just to ensure that developers
understand that it's different and actually look at the docs.

I completely agree on focusing on clarity and consistency, and I think
considering how we think it might evolve is good, but that can't be an
open-ended exercise. My primary concern is how we can start getting
incremental improvements into end users' hands more quickly, since the
alternative right now is to basically roll your own, right?

Cheers,

Derek

On Mon, Jun 13, 2022 at 4:16 PM bened...@apache.org 
wrote:

> What on earth does MIXED mean?
>
>
>
> I agree with the sentiment we should minimise surprise, but everyone is
> surprised differently so it becomes a sort of pointless rubrik, everyone
> claiming it supports their view. I think it is only useful in cases where
> there is clear agreement that something is surprising, but unhelpful when
> choosing between subtle variations on approach.
>
>
>
> The main goal IMO should be clarity and consistency, so that the user can
> reason about the constructs easily, and so we can evolve them.
>
>
>
> For instance, we should be sure to consider how the syntax will look if we
> **do** offer interactive transactions, or JOINs, or anything else we
> might add in future.
>
>
>
>
>
> *From: *Derek Chen-Becker 
> *Date: *Monday, 13 June 2022 at 23:09
> *To: *dev@cassandra.apache.org 
> *Subject: *Re: CEP-15 multi key transaction syntax
>
> On Mon, Jun 13, 2022 at 1:57 PM Blake Eggleston 
> wrote:
>
> I prefer an approach that supports an accurate mental model of what’s
> happening behind the scenes. I think that should be a design priority for
> the syntax. We’ll be able to build things on top of accord, but the core
> multi-key cas operation isn’t going to change too much.
>
>
>
> +1, the principle of least surprise tells me that if this doesn't behave
> exactly like SQL transactions (for whatever SQL actually means), it could
> be more clear to not try and emulate it halfway
>
>
>
> BEGIN MIXED TRANSACTION?
>
>
>
> Derek
>
>
>
>
>
>
>
> On Jun 13, 2022, at 12:14 PM, Blake Eggleston 
> wrote:
>
>
>
> Does the IF <...> ABORT simplify reasoning though? If you restrict it to
> only dealing with the most recent row it would, but referencing the name
> implies you’d be able to include references from other operations, in which
> case you’d have the same problem.
>
> > return instead an exception if the transaction is aborted
>
> Since the txn is not actually interactive, I think it would be better to
> receive values instead of an excetion, to understand why the operation was
> rolled back.
>
>
>
> On Jun 13, 2022, at 10:32 AM, Aaron Ploetz  wrote:
>
>
>
> Benedict,
>
>
>
> I'm really excited about this feature.  I've been observing this
> conversation for a while now, and I"m happy to add some thoughts.
>
>
>
> We must balance the fact we cannot afford to do everything (yet), against
> the need to make sure what we do is reasonably intuitive (to both CQL and
> SQL users) and consistent – including with whatever we do in future.
>
>
>
> I think taking small steps forward, to build a few complete features as
> close to SQL as possible is a good approach.
>
>
>
> question we are currently asking: do we want to have a more LWT-like
> approach... or do we want a more SQL-like approach
>
>
>
> For years now we've been fighting this notion that Cassandra is difficult
> to use.  Coming up with specialized syntax isn't going to bridge that
> divide.  From a (new?) user perspective, the best plan is to stay as
> consistent with SQL as possible.
>
>
>
> I believe that is a MySQL specific concept. This is one problem with
> mimicking SQL – it’s not one thing!
>
>
>
> Right?!?!  As if this needed to be more complex.
>
>
>
> I think we have evidence that it 

Re: CEP-15 multi key transaction syntax

[Derek Chen-Becker]

rences to define them, so you wouldn’t be able to run
>> these.
>>
>> > MVs
>>
>> To prevent being spread too thin, both in syntax design and
>> implementation work, I’d like to limit read and write operations in the
>> initial implementation to vanilla selects, updates, inserts, and deletes.
>> Once we have a solid implementation of multi-key/table transactions
>> supporting foundational operations, we can start figuring out how the more
>> advanced pieces can be best supported. Not a great answer to your question,
>> but a related tangent I should have included in my initial email.
>>
>> > ... RETURNING ...
>>
>> I like the idea of the returning statement, but to echo what Benedict
>> said, I think any scheme for specifying data to be returned should apply
>> the same to select and update statements, since updates can have underlying
>> reads that the user may be interested in. I’d mentioned having an optional
>> RETURN statement in addition to automatically returning selects in my
>> original email.
>>
>> > ... WITH ...
>>
>> I like the idea of defining statement names at the beginning of a
>> statement, since I could imagine mapping names to selects might get
>> difficult if there are a lot of columns in the select or update, but
>> beginning each statement with `WITH ` reduces readability imo. Maybe
>> putting the name after the first term of the statement (ie: `SELECT * AS
>>  WHERE...`, `UPDATE table AS  SET ...`, `INSERT INTO table AS
>>  (...) VALUES (...);`) would be improve finding names without harming
>> overall readability?
>>
>> Benedict,
>>
>> > I agree that SELECT statements should be required to go first.
>>
>> +1
>>
>> > There only remains the issue of conditions imposed upon
>> UPDATE/INSERT/DELETE statements when there are multiple statements that
>> affect the same primary key. I think we can (and should) simply reject such
>> queries for now, as it doesn’t make much sense to have multiple statements
>> for the same primary key in the same transaction.
>>
>> Unfortunately, I think there are use cases for both multiple selects and
>> updates for the same primary key in a txn. Selects aren’t as problematic,
>> but if multiple updates end up touching the same cell, I’d expect the last
>> one to win. This would make dealing with range tombstones a little
>> trickier, since the default behavior of alternating updates and range
>> tombstones affecting the same cells is not intuitive, but I don’t think it
>> would be too bad.
>>
>>
>> Something that’s come up a few times, and that I’ve also been thinking
>> about is whether to return the values that were originally read, or the
>> values written with the update to the client, and there are use cases for
>> both. I don’t remember who suggested it, but I think returning the original
>> values from named select statements, and the post-update values from named
>> update statements is a good way to handle both. Also, while returning the
>> contents of the mutation would be the easiest, implementation wise,
>> swapping cell values from the updates named read would be most
>> useful, since a txn won’t always result in an update, in which case we’d
>> just return the select.
>>
>> Thanks,
>>
>> Blake
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Jun 6, 2022, at 9:41 AM, Henrik Ingo  wrote:
>>
>>
>> On Mon, Jun 6, 2022 at 5:28 PM bened...@apache.org 
>> wrote:
>>
>> > One way to make it obvious is to require the user to explicitly type
>> the SELECTs and then to require that all SELECTs appear before
>> UPDATE/INSERT/DELETE.
>>
>>
>> Yes, I agree that SELECT statements should be required to go first.
>>
>>
>> However, I think this is sufficient and we can retain the shorter format
>> for RETURNING. There only remains the issue of conditions imposed upon
>> UPDATE/INSERT/DELETE statements when there are multiple statements that
>> affect the same primary key. I think we can (and should) simply reject such
>> queries for now, as it doesn’t make much sense to have multiple statements
>> for the same primary key in the same transaction.
>>
>>
>>
>> I guess I was thinking ahead to a future where and UPDATE write set may
>> or may not intersect with a previous update due to allowing WHERE clause to
>> use secondary keys, etc.
>>
>>
>> That said, I'm not saying we SHOULD require explicit SELECT statements
>> for every update. I'm sure that would be annoying more than useful.I was
>> just following a train of thought.
>>
>>
>>
>>
>>
>> > Returning the "result" from an UPDATE presents the question should it
>> be the data at the start of the transaction or end state?
>>
>>
>> I am inclined to only return the new values (as proposed by Alex) for the
>> purpose of returning new auto-increment values etc. If you require the
>> prior value, SELECT is available to express this.
>>
>>
>>
>> That's a great point!
>>
>>
>>
>>
>> > I was thinking the following coordinator-side implementation would
>> allow to use also old drivers
>>
>>
>> I am inclined to return just the first result set to old clients. I think
>> it’s fine to require a client upgrade to get multiple result sets.
>>
>>
>>
>> Possibly. I just wanted to share an idea for consideration. IMO the temp
>> table idea might not be too hard to implement*, but sure the syntax does
>> feel a bit bolted on.
>>
>>
>> *) I'm maybe the wrong person to judge that, of course :-)
>>
>>
>> henrik
>>
>>
>> --
>>
>> Henrik Ingo
>>
>> +358 40 569 7354 <358405697354>
>>
>>
>>
>>
>>
>
>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: CEP-15 multi key transaction syntax

[Derek Chen-Becker]

m.
>
> Patrick, I think Benedict answered your questions? Glad you got the joke :)
>
> Alex,
>
> > 1. Dependant SELECTs
> > 2. Dependant UPDATEs
> > 3. UPDATE from secondary index (or SASI)
> > 5. UPDATE with predicate on non-primary key
>
> The full primary key must be defined as part of the statement, and you
> can’t use column references to define them, so you wouldn’t be able to run
> these.
>
> > MVs
>
> To prevent being spread too thin, both in syntax design and implementation
> work, I’d like to limit read and write operations in the initial
> implementation to vanilla selects, updates, inserts, and deletes. Once we
> have a solid implementation of multi-key/table transactions supporting
> foundational operations, we can start figuring out how the more advanced
> pieces can be best supported. Not a great answer to your question, but a
> related tangent I should have included in my initial email.
>
> > ... RETURNING ...
>
> I like the idea of the returning statement, but to echo what Benedict
> said, I think any scheme for specifying data to be returned should apply
> the same to select and update statements, since updates can have underlying
> reads that the user may be interested in. I’d mentioned having an optional
> RETURN statement in addition to automatically returning selects in my
> original email.
>
> > ... WITH ...
>
> I like the idea of defining statement names at the beginning of a
> statement, since I could imagine mapping names to selects might get
> difficult if there are a lot of columns in the select or update, but
> beginning each statement with `WITH ` reduces readability imo. Maybe
> putting the name after the first term of the statement (ie: `SELECT * AS
>  WHERE...`, `UPDATE table AS  SET ...`, `INSERT INTO table AS
>  (...) VALUES (...);`) would be improve finding names without harming
> overall readability?
>
> Benedict,
>
> > I agree that SELECT statements should be required to go first.
>
> +1
>
> > There only remains the issue of conditions imposed upon
> UPDATE/INSERT/DELETE statements when there are multiple statements that
> affect the same primary key. I think we can (and should) simply reject such
> queries for now, as it doesn’t make much sense to have multiple statements
> for the same primary key in the same transaction.
>
> Unfortunately, I think there are use cases for both multiple selects and
> updates for the same primary key in a txn. Selects aren’t as problematic,
> but if multiple updates end up touching the same cell, I’d expect the last
> one to win. This would make dealing with range tombstones a little
> trickier, since the default behavior of alternating updates and range
> tombstones affecting the same cells is not intuitive, but I don’t think it
> would be too bad.
>
>
> Something that’s come up a few times, and that I’ve also been thinking
> about is whether to return the values that were originally read, or the
> values written with the update to the client, and there are use cases for
> both. I don’t remember who suggested it, but I think returning the original
> values from named select statements, and the post-update values from named
> update statements is a good way to handle both. Also, while returning the
> contents of the mutation would be the easiest, implementation wise,
> swapping cell values from the updates named read would be most
> useful, since a txn won’t always result in an update, in which case we’d
> just return the select.
>
> Thanks,
>
> Blake
>
>
>
>
>
>
>
>
>
>
> On Jun 6, 2022, at 9:41 AM, Henrik Ingo  wrote:
>
>
>
> On Mon, Jun 6, 2022 at 5:28 PM bened...@apache.org 
> wrote:
>
> > One way to make it obvious is to require the user to explicitly type
> the SELECTs and then to require that all SELECTs appear before
> UPDATE/INSERT/DELETE.
>
>
>
> Yes, I agree that SELECT statements should be required to go first.
>
>
>
> However, I think this is sufficient and we can retain the shorter format
> for RETURNING. There only remains the issue of conditions imposed upon
> UPDATE/INSERT/DELETE statements when there are multiple statements that
> affect the same primary key. I think we can (and should) simply reject such
> queries for now, as it doesn’t make much sense to have multiple statements
> for the same primary key in the same transaction.
>
>
>
> I guess I was thinking ahead to a future where and UPDATE write set may or
> may not intersect with a previous update due to allowing WHERE clause to
> use secondary keys, etc.
>
>
>
> That said, I'm not saying we SHOULD require explicit SELECT statements for
> every update. I'm sure that would be annoying more than useful.I was just
> following a train of thought.
>
>
>
>
>
>
>
> > Returning the "result" from an UPDATE presents the question should it
> be the data at the start of the transaction or end state?
>
>
>
> I am inclined to only return the new values (as proposed by Alex) for the
> purpose of returning new auto-increment values etc. If you require the
> prior value, SELECT is available to express this.
>
>
>
> That's a great point!
>
>
>
>
>
> > I was thinking the following coordinator-side implementation would
> allow to use also old drivers
>
>
>
> I am inclined to return just the first result set to old clients. I think
> it’s fine to require a client upgrade to get multiple result sets.
>
>
>
> Possibly. I just wanted to share an idea for consideration. IMO the temp
> table idea might not be too hard to implement*, but sure the syntax does
> feel a bit bolted on.
>
>
>
> *) I'm maybe the wrong person to judge that, of course :-)
>
>
>
> henrik
>
>
>
> --
>
> Henrik Ingo
>
> +358 40 569 7354 <358405697354>
>
>
>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSS] Change code style guide WRT to @Override in subclasses / interface implementations

[Derek Chen-Becker]

+1 to publishing. We should consider it a living document, not something
that we need to necessarily set in stone :)

Cheers,

Derek

On Fri, Jun 3, 2022 at 10:05 AM bened...@apache.org 
wrote:

> I always ask if we’re ready, get a few acks, then one or two new queries
> come out of the woodwork.
>
>
>
> Perhaps I will just publish, and we can start addressing these queries in
> a follow-up process.
>
>
>
> *From: *Dinesh Joshi 
> *Date: *Friday, 3 June 2022 at 16:57
> *To: *dev@cassandra.apache.org 
> *Subject: *Re: [DISCUSS] Change code style guide WRT to @Override in
> subclasses / interface implementations
>
> I don’t think the guide has yet been published to the official website,
> has it? Maybe we should just get it out there.
>
> On Jun 3, 2022, at 8:54 AM, bened...@apache.org wrote:
>
> 
>
> Somebody hasn’t looked at the new style guide*, the conversation for which
> keeps rolling on and so it never quite gets promoted to the wiki. It says:
>
>
>
> Always use @Override annotations when implementing abstract or interface
> methods or overriding a parent method.
>
>
>
> *
> https://docs.google.com/document/d/1sjw0crb0clQin2tMgZLt_ob4hYfLJYaU4lRX722htTo
>
>
>
>
>
> *From: *Josh McKenzie 
> *Date: *Friday, 3 June 2022 at 16:14
> *To: *dev@cassandra.apache.org 
> *Subject: *Re: [DISCUSS] Change code style guide WRT to @Override in
> subclasses / interface implementations
>
> > Avoid redundant @Override annotations when implementing abstract or
> interface methods.
>
> I'd argue they're not redundant. We're humans and infinitely fallible. :)
>
>
>
> +1 to changing this to just always annotate for all the reasons you
> enumerate.
>
>
>
> On Fri, Jun 3, 2022, at 10:16 AM, Alex Petrov wrote:
>
> Right, my thinking matches what David has mentioned:
>
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-16096
>
> https://lists.apache.org/thread/mkskwxn921t5bkfmnog032qvnyjk82t7
>
>
>
> I'll make sure to update the style guide itself, too, since it looks like
> there was a vote, and intellij file is updated, just need to fixup the
> website.
>
>
>
>
>
> On Fri, Jun 3, 2022, at 4:02 PM, Dinesh Joshi wrote:
>
> So your proposal is to always add override annotation? Or are there
> situations where you don’t want to add them?
>
>
>
>
>
> On Jun 3, 2022, at 6:53 AM, Alex Petrov  wrote:
>
> 
>
> Hi everyone,
>
>
>
> In our style guide [1], we have a following statement:
>
>
>
> > Avoid redundant @Override annotations when implementing abstract or
> interface methods.
>
>
>
> I'd like to suggest we change this.
>
>
>
> @Override annotation in subclasses might be annoying when you're writing
> the code for the first time, or reading already familiar code, but when
> you're working on large changes and have complex class hierarchies, or
> multiple overloads for the method, it's easy to overlook methods that were
> not marked as overrides, and leave a wrong method in the code, or
> misinterpret the call chain.
>
>
>
> I think @Override annotations are extremely useful and serve their
> purpose, especially when refactoring: I can change the interface, and will
> not only be pointed to all classes that do not implement the new version
> (which compiler will do anyways), but also will be pointed to the classes
> that, to the human eye, may look like they're overriding the method, but in
> fact they do not.
>
>
>
> More concrete example: there is an abstract class between the interface
> and a concrete implementation: you change the interface, modify the method
> in the abstract class, but then forget to change the signature in the
> overriden implementation of the concrete class, and get a behaviour from
> the abstract class rather then concrete implementation.
>
>
>
> The question is not about taste or code aesthetics, but about making
> maintaining a large codebase that has a lot of complexity and that was
> evolving over many years simpler. If you could provide an example where
> @Override would be counter-productive or overly burdensome, we could
> compare this cost of maintenance with the cost of potential errors.
>
>
>
> Thank you,
>
> --Alex
>
>
>
> [1] https://cassandra.apache.org/_/development/code_style.html
>
>
>
>
>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: [DISCUSS] Change code style guide WRT to @Override in subclasses / interface implementations

[Derek Chen-Becker]

I think we discussed this a couple of weeks ago, but to reiterate my
position: I think we should use annotations where they allow the compiler
and ancillary tooling (e.g. FindBugs) to catch and prevent classes of
errors. @Override seems like a pretty easy one to add, and has concrete
examples of the kinds of errors it can prevent. We discussed @Nullable,
@Threadsafe and others that may have more marginal utility, so we were a
bit less prescriptive there. To sum up, I agree with Alex that @Override
has enough utility to say we should use it by default.

Cheers,

Derek

On Fri, Jun 3, 2022 at 8:04 AM Dinesh Joshi  wrote:

> So your proposal is to always add override annotation? Or are there
> situations where you don’t want to add them?
>
>
> On Jun 3, 2022, at 6:53 AM, Alex Petrov  wrote:
>
> 
> Hi everyone,
>
> In our style guide [1], we have a following statement:
>
> > Avoid redundant @Override annotations when implementing abstract or
> interface methods.
>
> I'd like to suggest we change this.
>
> @Override annotation in subclasses might be annoying when you're writing
> the code for the first time, or reading already familiar code, but when
> you're working on large changes and have complex class hierarchies, or
> multiple overloads for the method, it's easy to overlook methods that were
> not marked as overrides, and leave a wrong method in the code, or
> misinterpret the call chain.
>
> I think @Override annotations are extremely useful and serve their
> purpose, especially when refactoring: I can change the interface, and will
> not only be pointed to all classes that do not implement the new version
> (which compiler will do anyways), but also will be pointed to the classes
> that, to the human eye, may look like they're overriding the method, but in
> fact they do not.
>
> More concrete example: there is an abstract class between the interface
> and a concrete implementation: you change the interface, modify the method
> in the abstract class, but then forget to change the signature in the
> overriden implementation of the concrete class, and get a behaviour from
> the abstract class rather then concrete implementation.
>
> The question is not about taste or code aesthetics, but about making
> maintaining a large codebase that has a lot of complexity and that was
> evolving over many years simpler. If you could provide an example where
> @Override would be counter-productive or overly burdensome, we could
> compare this cost of maintenance with the cost of potential errors.
>
> Thank you,
> --Alex
>
> [1] https://cassandra.apache.org/_/development/code_style.html
>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Updating our Code Contribution/Style Guide

[Derek Chen-Becker]

Looks great!

On Mon, May 30, 2022, 5:37 AM bened...@apache.org 
wrote:

> Any more feedback around this? Everyone happy with the latest proposal?
>
>
>
> *From: *bened...@apache.org 
> *Date: *Sunday, 15 May 2022 at 15:08
> *To: *dev@cassandra.apache.org 
> *Subject: *Re: Updating our Code Contribution/Style Guide
>
> I agree with this sentiment, but I think it will require a bit of time to
> figure out where that balance is.
>
>
>
> I’ve inserted a mention of @Nullable, @ThreadSafe, @NotThreadSafe and
> @Immutable.
>
>
>
> > If we only use one of the two - for example @Nullable - that leaves us
> with "We know the original author expected this to be null at some point in
> its lifecycle and it means something" and "We have no idea if this is
> legacy and nullable or not"
>
>
>
> My inclination is to start building some norms around this, carefully as
> we don’t have enough experience and understanding of the pitfalls and long
> term usage. But, my preferred norms would be that properties should be
> assumed to be @Nonnull and that all nullable parameters and properties
> should be marked as @Nullable. This is how I use these properties today;
> Nonnull always seems superfluous, as it is rare to have a set of properties
> where null is the default, or where it is particularly important that the
> reader or compiler realise this.
>
>
>
> There will be an interim period, in particular for legacy code, where this
> may lead to less clarity. But in the long term this is probably preferable
> to inconsistent usage where some areas of the codebase indicate @Nonnull
> without indicating @Nullable, and vice-versa, or where every variable and
> method ends up marked with one or the other.
>
>
>
> This is probably also most consistent with a future world of cheap
> Optional types (i.e. Valhalla), where Nullable may begin to be replaced
> with Optional, and Nonnull may become very much the default.
>
>
>
> That said, as stated multiple times, the author and reviewer’s
> determinations are final. This document just sets up some basic
> parameters/expectations.
>
>
>
> *From: *Derek Chen-Becker 
> *Date: *Saturday, 14 May 2022 at 20:56
> *To: *dev@cassandra.apache.org 
> *Subject: *Re: Updating our Code Contribution/Style Guide
>
> On Sat, May 14, 2022 at 11:00 AM Josh McKenzie 
> wrote:
>
>
>
> Incidentally, I've found similar value in @ThreadSafe, const, readonly,
> etc - communications of author's intent; being able to signal to future
> maintainers helps them make modifications that are more consistent with and
> safer with regards to the original intention and guarantees of the author.
>
>
>
> Assuming you trust those guarantees that is. :)
>
>
>
> I think author's intent is important, which is why I also think that
> judicious/effective commenting and naming are important (and I'm glad that
> naming is called out in the guidelines explicitly). However, I also think
> that these are also opportunities to help the compiler and tooling help us,
> similar to how Benedict's draft calls out effective use of the type system
> as a way to encode semantics and constraints in the code. These
> annotations, while clunky and verbose, do open the door in some cases to
> static analysis that the Java compiler is incapable of doing. I don't know
> exactly where it is, but I think there's a balance between use of
> annotations to help tooling identify problems while not becoming onerous
> for current and future contributors. I know this is more difficult in Java
> than, say, Rust, but I'm an eternal optimist and I think we can find that
> balance :)
>
>
>
> Cheers,
>
>
>
> Derek
>
>
>
> --
>
> +---+
>
> | Derek Chen-Becker |
>
> | GPG Key available at https://keybase.io/dchenbecker and   |
>
> | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
>
> | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
>
> +---+
>
>
>

Re: [VOTE] Release Apache Cassandra 4.1-alpha1

[Derek Chen-Becker]

+1 nb. Were we going to mention the flaky tests in NEWS.txt or just as part
of the email? Or have those been resolved?

Cheers,

Derek

On Tue, May 24, 2022 at 2:38 AM Mick Semb Wever  wrote:

> Proposing the test build of Cassandra 4.1-alpha1 for release.
>
> sha1: 6f05be447073925a7f3620ddbbd572aa9fcd10ed
> Git:
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/4.1-alpha1-tentative
> Maven Artifacts:
>
> https://repository.apache.org/content/repositories/orgapachecassandra-1273/org/apache/cassandra/cassandra-all/4.1-alpha1/
>
> The Source and Build Artifacts, and the Debian and RPM packages and
> repositories, are available here:
> https://dist.apache.org/repos/dist/dev/cassandra/4.1-alpha1/
>
> The vote will be open for 72 hours (longer if needed). Everyone who
> has tested the build is invited to vote. Votes by PMC members are
> considered binding. A vote passes if there are at least three binding
> +1s and no -1's.
>
> [1]: CHANGES.txt:
>
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/4.1-alpha1-tentative
> [2]: NEWS.txt:
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/4.1-alpha1-tentative
>


-- 
+-------+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Appetite for a 4.1-alpha1 ?

[Derek Chen-Becker]

+1 to a release, but should we be explicit about alphas potentially having
flaky test failures? I think the release lifecycle doc is really good, but
I think when we find implied behavior it's an opportunity to make it more
concrete. I don't want to be dogmatic here, but flaky tests definitely seem
like something we would want users to be aware of as a possible risk of
using alpha.

Cheers,

Derek

On Wed, May 18, 2022 at 3:40 AM Mick Semb Wever  wrote:

> Our release lifecycle docs¹ imply that we can release alphas despite
> flaky test failures, which means we can cut and vote on a 4.1-alpha1
> release today. This is also on the presumption that point (2) on our
> Cassandra CI Process docs² does not apply to pre-beta releases.
>
> Is there an appetite for this?
> Any objections?
> Any tickets about to land folk want us to wait on?
>
> regards,
> Mick
>
>
> 1) https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle
>
> 2)
> https://cwiki.apache.org/confluence/display/CASSANDRA/Cassandra+CI+Process
>


-- 
+-------+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Updating our Code Contribution/Style Guide

[Derek Chen-Becker]

Fair enough. Thanks for the perspective, and the doc is shaping up nicely!

On Sat, May 14, 2022 at 2:28 PM bened...@apache.org 
wrote:

> > having the policy be enums by default as opposed to just recommending
> them
>
>
>
> This might be a stylistic issue. “Prefer an enum to Boolean properties” is
> imperative voice, and is meant to be read as “you should use enums, not
> booleans, unless you have overriding reasons not to” – perhaps the example
> scenarios that follow, in which they are most strongly indicated, weaken
> the effect.
>
>
>
> I’m sure I can tweak the language, but overall I have tried to avoid
> making anything an explicit dictat in this style guide. It’s somewhere
> between a policy and a set of recommendations, as I think it is preferable
> to leave the author and reviewer to make final determinations, and also to
> avoid imbuing documents like this with too much power (and making them too
> contentious).
>
>
>
> I’ll see about tweaking it along with your other suggestions.
>
>
>
> *From: *Derek Chen-Becker 
> *Date: *Saturday, 14 May 2022 at 20:45
> *To: *dev@cassandra.apache.org 
> *Subject: *Re: Updating our Code Contribution/Style Guide
>
>
>
>
>
> On Sat, May 14, 2022 at 8:24 AM bened...@apache.org 
> wrote:
>
> > I'm in favor of codifying the usage of @NotNull and @Nullable
> stylistically. +1
>
>
>
> I’m in favour of the use of _*one*_ of @Nullable and @NotNull, preferably
> the former since we already use it and it’s more reasonable to have a
> default of non-null variables, parameters and properties.
>
>
>
> However, I’m not confident in how to craft guidance for these annotations.
> I don’t think they should be used in every place a variable or property
> might be null, only in places where it is surprising or otherwise
> informative to a reader that they might be null. Annotating every property
> and variable with @NonNull or @Nullable would seriously pollute the screen,
> and probably harm legibility more than help.
>
>
>
> At the very least we should mention @Nullable and invite authors to use it
> where it aids clarity, but if somebody has a good proposal for better
> guidance I’m all ears.
>
>
>
> Yes, unfortunately there's a whole menagerie of these types of
> annotations, and I didn't mean both. If we're already using Nullable (from
> Findbugs) that's the better one anyway because you can specify the when
> parameter. It's also supported by languages like Kotlin for nullable types
> if we were ever considering a language that wouldn't require polluting the
> screen for a bit more safety ;)
>
>
>
> Overall I think that an assumption that all variables are null unless
> explicitly marked is probably a reasonable first step if it's not already
> in place, but it's also a good intention more than a mechanism and I'll put
> some thought into other ways we can improve the situation without impacting
> legibility.
>
>
>
>
>
>
>
> > I think extra clarity and social pressure around "Never catch Exception
> or Throwable unless you explicitly rethrow them" sounds valuable
>
>
>
> We already stipulate that you should always rethrow exceptions, but this
> is very vague. I will try to tidy this up. On the whole, though, we have a
> fail-fast approach to processing commands, so we mostly just propagate,
> with exception handlers existing only for clean-up purposes (except in
> particular circumstances, usually involving checked exceptions like
> InterruptedException). So we mostly *do* catch Throwable (and rethrow), I
> think, which is what informed the current vague formulation.
>
>
>
> Sure, rethrow after cleanup seems reasonable, but I think that should be
> the explicit exception rather than an assumption of our approach to error
> handling.
>
>
>
> > I would recommend that we strengthen the recommendation for using enums
> for Boolean properties for any type that is used in method parameters
>
>
>
> I’m unsure about this. I am not against it per se, but the more enums we
> have the more clashes of enum identifiers we have, and this can cause
> confusion particularly with static imports, and in some cases the Boolean
> property will have a very obvious effect. I prefer to leave some decisions
> to the author, since we have expressed a strong preference here for the
> author to consider. But perhaps a blanket policy would do more good than
> harm. I could endorse it, and am relatively neutral.
>
>
>
>
>
> To be clear, I think there should always be room for (clearly documented)
> exceptions, so I was thinking more of having the policy be enums by default
> as opposed to just recommendin

Re: Updating our Code Contribution/Style Guide

[Derek Chen-Becker]

On Sat, May 14, 2022 at 11:00 AM Josh McKenzie  wrote:

>
> Incidentally, I've found similar value in @ThreadSafe, const, readonly,
> etc - communications of author's intent; being able to signal to future
> maintainers helps them make modifications that are more consistent with and
> safer with regards to the original intention and guarantees of the author.
>
> Assuming you trust those guarantees that is. :)
>

I think author's intent is important, which is why I also think that
judicious/effective commenting and naming are important (and I'm glad that
naming is called out in the guidelines explicitly). However, I also think
that these are also opportunities to help the compiler and tooling help us,
similar to how Benedict's draft calls out effective use of the type system
as a way to encode semantics and constraints in the code. These
annotations, while clunky and verbose, do open the door in some cases to
static analysis that the Java compiler is incapable of doing. I don't know
exactly where it is, but I think there's a balance between use of
annotations to help tooling identify problems while not becoming onerous
for current and future contributors. I know this is more difficult in Java
than, say, Rust, but I'm an eternal optimist and I think we can find that
balance :)

Cheers,

Derek

-- 
+-----------+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Updating our Code Contribution/Style Guide

[Derek Chen-Becker]

On Sat, May 14, 2022 at 8:24 AM bened...@apache.org 
wrote:

> > I'm in favor of codifying the usage of @NotNull and @Nullable
> stylistically. +1
>
>
>
> I’m in favour of the use of _*one*_ of @Nullable and @NotNull, preferably
> the former since we already use it and it’s more reasonable to have a
> default of non-null variables, parameters and properties.
>
>
>
> However, I’m not confident in how to craft guidance for these annotations.
> I don’t think they should be used in every place a variable or property
> might be null, only in places where it is surprising or otherwise
> informative to a reader that they might be null. Annotating every property
> and variable with @NonNull or @Nullable would seriously pollute the screen,
> and probably harm legibility more than help.
>
>
>
> At the very least we should mention @Nullable and invite authors to use it
> where it aids clarity, but if somebody has a good proposal for better
> guidance I’m all ears.
>

Yes, unfortunately there's a whole menagerie of these types of annotations,
and I didn't mean both. If we're already using Nullable (from Findbugs)
that's the better one anyway because you can specify the when parameter.
It's also supported by languages like Kotlin for nullable types if we were
ever considering a language that wouldn't require polluting the screen for
a bit more safety ;)

Overall I think that an assumption that all variables are null unless
explicitly marked is probably a reasonable first step if it's not already
in place, but it's also a good intention more than a mechanism and I'll put
some thought into other ways we can improve the situation without impacting
legibility.



>
>
> > I think extra clarity and social pressure around "Never catch Exception
> or Throwable unless you explicitly rethrow them" sounds valuable
>
>
>
> We already stipulate that you should always rethrow exceptions, but this
> is very vague. I will try to tidy this up. On the whole, though, we have a
> fail-fast approach to processing commands, so we mostly just propagate,
> with exception handlers existing only for clean-up purposes (except in
> particular circumstances, usually involving checked exceptions like
> InterruptedException). So we mostly *do* catch Throwable (and rethrow), I
> think, which is what informed the current vague formulation.
>

Sure, rethrow after cleanup seems reasonable, but I think that should be
the explicit exception rather than an assumption of our approach to error
handling.


> > I would recommend that we strengthen the recommendation for using enums
> for Boolean properties for any type that is used in method parameters
>
>
>
> I’m unsure about this. I am not against it per se, but the more enums we
> have the more clashes of enum identifiers we have, and this can cause
> confusion particularly with static imports, and in some cases the Boolean
> property will have a very obvious effect. I prefer to leave some decisions
> to the author, since we have expressed a strong preference here for the
> author to consider. But perhaps a blanket policy would do more good than
> harm. I could endorse it, and am relatively neutral.
>
>
>
To be clear, I think there should always be room for (clearly documented)
exceptions, so I was thinking more of having the policy be enums by default
as opposed to just recommending them. I've been thinking that as part of
the guidelines it might be good to have some examples of both (here's how
you can use an enum, but here's a case where a boolean was simple and
clear), so let me dig around and see if I can find some code to point to.

Cheers,

Derek



-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+

Re: Resurrection of CASSANDRA-9633 - SSTable encryption

[Derek Chen-Becker]

Thanks, that's really helpful to have some code to look at!

Derek

On Fri, Nov 19, 2021 at 9:35 AM Joseph Lynch  wrote:

> On Fri, Nov 19, 2021 at 9:52 AM Derek Chen-Becker 
> wrote:
> >
> > https://bugs.openjdk.java.net/browse/JDK-7184394 added AES intrinsics in
> > Java 8, in 2012. While it's always possible to have a regression, and
> it's
> > important to understand the performance impact, stories of 2-10x sound
> > apocryphal. If they're all using the same intrinsics, the performance
> > should be roughly the same. I think that the real challenge will be key
> > management, not performance.
> >
> > Derek
>
> > On Fri, Nov 19, 2021 at 7:41 AM Bowen Song  wrote:
> >
> > > On the performance note, I copy & pasted a small piece of Java code to
> > > do AES256-CBC on the stdin and write the result to stdout. I then ran
> > > the following two commands on the same machine (with AES-NI) for
> > > comparison:
> > >
> > > $ dd if=/dev/zero bs=4096 count=$((4*1024*1024)) status=none | time
> > > /usr/lib/jvm/java-11-openjdk/bin/java -jar aes-bench.jar >/dev/null
> > > 36.24s user 5.96s system 100% cpu 41.912 total
> > > $ dd if=/dev/zero bs=4096 count=$((4*1024*1024)) status=none | time
> > > openssl enc -aes-256-cbc -e -K
> > > "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
> > > -iv "0123456789abcdef0123456789abcdef" >/dev/null
> > > 31.09s user 3.92s system 99% cpu 35.043 total
> > >
> > > This is not an accurate test of the AES performance, as the Java test
> > > includes the JVM start up time and the key and IV generation in the
> Java
> > > code. But this gives us a pretty good idea that the total performance
> > > regression is definitely far from the 2x to 10x slower claimed in some
> > > previous emails.
> > >
> > >
>
> I am aware that Java added AES intrinsics support in Java 8, but it is
> still painfully slow doing authenticated AES-GCM and many other forms
> of crypto [1]. Native AES-GCM on my laptop running at 4GHz [2]
> achieves 3.7 GiB/s while Java 8 can manage a mere 289 MiB/s (13x
> slower) and Java 11 manages 768 MiB/s (5x slower) [3]. AWS literally
> funded an entire project [4] to speed up slow Java crypto which has
> sped up basic crypto from 2-10x [5, 6] on real world workloads at
> scale.
>
> I don't think my claims are apocryphal when I and others have spent so
> much time on this project and other JVM projects debugging why they
> are so slow, including most recently determining the root cause to the
> initial serious performance regressions in 4.0's networking code was
> due to native Java 8's TLS stack and specifically AES-GCM
> implementation being painfully slow (the fix we settled on was to use
> tcnative with native AES-GCM) [6] as well as speeding up quorum reads
> by 2x through using faster MD5 crypto [8, 9, 10].
>
> -Joey
>
> [1]
> https://gist.github.com/jolynch/a6db4409ddae8d5163894bef77204934#file-summary-txt
> [2]
> https://gist.github.com/jolynch/a6db4409ddae8d5163894bef77204934#file-benchmarkon-sh
> [3]
> https://gist.github.com/jolynch/a6db4409ddae8d5163894bef77204934#file-authenticated_encryption_perf-txt
> [4] https://github.com/corretto/amazon-corretto-crypto-provider
> [5] https://github.com/corretto/amazon-corretto-crypto-provider/pull/54
> [6] https://github.com/corretto/amazon-corretto-crypto-provider/issues/52
> [7] https://issues.apache.org/jira/browse/CASSANDRA-15175
> [8] https://issues.apache.org/jira/browse/CASSANDRA-14611
> [9] https://issues.apache.org/jira/browse/CASSANDRA-15294
> [10]
> https://github.com/corretto/amazon-corretto-crypto-provider/issues/52#issuecomment-531921577
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
+---+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and   |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
+---+