Re: ssl certificate hot reloading test - cassandra 4.1

[Josh McKenzie]

I think it's all part of the same issue and you're not derailing IMO Abe. For 
the user Pabbireddy here, the unexpected behavior was not closing internode 
connections on that keystore refresh. So ISTM, from a "featureset that would be 
nice to have here" perspective, we could theoretically provide:
 1. An option to disconnect all connections on cert update, disabled by default
 2. An option to drain and recycle connections on a time period, also disabled 
by default
Leave the current behavior in place but allow for these kind of strong 
cert-guarantees if a user needs it in their env.

On Mon, Apr 15, 2024, at 9:51 PM, Abe Ratnofsky wrote:
> Not to derail from the original conversation too far, but wanted to agree 
> that maximum connection establishment time on native transport would be 
> useful. That would provide a maximum duration before an updated client 
> keystore is used for connections, which can be used to safely roll out client 
> keystore updates.
> 
> For example, if the maximum connection establishment time is 12 hours, then 
> you can update the keystore on a canary client, wait 24 hours, confirm that 
> connectivity is maintained, then upgrade keystores across the rest of the 
> fleet.
> 
> With unbounded connection establishment, reconnection isn't tested as often 
> and issues can hide behind long-lived connections.
> 
>> On Apr 15, 2024, at 5:14 PM, Jeff Jirsa  wrote:
>> 
>> It seems like if folks really want the life of a connection to be finite 
>> (either client/server or server/server), adding in an option to quietly 
>> drain and recycle a connection on some period isn’t that difficult.
>> 
>> That type of requirement shows up in a number of environments, usually on 
>> interactive logins (cqlsh, login, walk away, the connection needs to become 
>> invalid in a short and finite period of time), but adding it to internode 
>> could also be done, and may help in some weird situations (if you changed 
>> certs because you believe a key/cert is compromised, having the connection 
>> remain active is decidedly inconvenient, so maybe it does make sense to add 
>> an expiration timer/condition on each connection).
>> 
>> 
>> 
>>> On Apr 15, 2024, at 12:28 PM, Dinesh Joshi  wrote:
>>> 
>>> In addition to what Andy mentioned, I want to point out that for the vast 
>>> majority of use-cases, we would like to _avoid_ interruptions when a 
>>> certificate is updated so it is by design. If you're dealing with a 
>>> situation where you want to ensure that the connections are cycled, you can 
>>> follow Andy's advice. It will require automation outside the database that 
>>> you might already have. If there is demand, we can consider adding a 
>>> feature to slowly cycle the connections so the old SSL context is not used 
>>> anymore.
>>> 
>>> One more thing you should bear in mind is that Cassandra will not load the 
>>> new SSL context if it cannot successfully initialize it. This is again by 
>>> design to prevent an outage when the updated truststore is corrupted or 
>>> could not be read in some way.
>>> 
>>> thanks,
>>> Dinesh
>>> 
>>> On Mon, Apr 15, 2024 at 9:45 AM Tolbert, Andy  
>>> wrote:
 I should mention, when toggling disablebinary/enablebinary between
 instances, you will probably want to give some time between doing this
 so connections can reestablish, and you will want to verify that the
 connections can actually reestablish.  You also need to be mindful of
 this being disruptive to inflight queries (if your client is
 configured for retries it will probably be fine).  Semantically to
 your applications it should look a lot like a rolling cluster bounce.
 
 Thanks,
 Andy
 
 On Mon, Apr 15, 2024 at 11:39 AM pabbireddy avinash
  wrote:
 >
 > Thanks Andy for your reply . We will test the scenario you mentioned.
 >
 > Regards
 > Avinash
 >
 > On Mon, Apr 15, 2024 at 11:28 AM, Tolbert, Andy  
 > wrote:
 >>
 >> Hi Avinash,
 >>
 >> As far as I understand it, if the underlying keystore/trustore(s)
 >> Cassandra is configured for is updated, this *will not* provoke
 >> Cassandra to interrupt existing connections, it's just that the new
 >> stores will be used for future TLS initialization.
 >>
 >> Via: 
 >> https://cassandra.apache.org/doc/4.1/cassandra/operating/security.html#ssl-certificate-hot-reloading
 >>
 >> > When the files are updated, Cassandra will reload them and use them 
 >> > for subsequent connections
 >>
 >> I suppose one could do a rolling disablebinary/enablebinary (if it's
 >> only client connections) after you roll out a keystore/truststore
 >> change as a way of enforcing the existing connections to reestablish.
 >>
 >> Thanks,
 >> Andy
 >>
 >>
 >> On Mon, Apr 15, 2024 at 11:11 AM pabbireddy avinash
 >>  wrote:
 >> >
 >> > Dear Community,
 >> >
 >> > I hope this email 

Re: Cassandra p95 latencies

[Josh McKenzie]

> The queries are rightly designed
Data modeling in Cassandra is 100% gray space; there unfortunately is no right 
or wrong design. You'll need to share basic shapes / contours of your data 
model for other folks to help you; seemingly innocuous things in a data model 
can cause unexpected issues w/C*'s storage engine paradigm thanks to the 
partitioning and data storage happening under the hood.

If you were seeing single digit ms on 3.0.X or 3.11.X and 40ms p95 on 4.0 I'd 
immediately look to the DB as being the culprit. For all other cases, you 
should be seeing single digit ms as queries in C* generally boil down to 
key/value lookups (partition key) to a list of rows you either point query 
(key/value #2) or range scan via clustering keys and pull back out.

There's also paging to take into consideration (whether you're using it or not, 
what your page size is) and the data itself (do you have thousands of columns? 
Multi-MB blobs you're pulling back out? etc). All can play into this.

On Fri, Aug 11, 2023, at 3:40 PM, Jeff Jirsa wrote:
> You’re going to have to help us help you 
> 
> 4.0 is pretty widely deployed. I’m not aware of a perf regression 
> 
> Can you give us a schema (anonymized) and queries and show us a trace ? 
> 
> 
>> On Aug 10, 2023, at 10:18 PM, Shaurya Gupta  wrote:
>> 
>> The queries are rightly designed as I already explained. 40 ms is way too 
>> high as compared to what I seen with other DBs and many a times with 
>> Cassandra 3.x versions.
>> CPU consumed as I mentioned is not high, it is around 20%.
>> 
>> On Thu, Aug 10, 2023 at 5:14 PM MyWorld  wrote:
>>> Hi,
>>> P95 should not be a problem if rightly designed. Levelled compaction 
>>> strategy further reduces this, however it consume some resources. For read, 
>>> caching is also helpful. 
>>> Can you check your cpu iowait as it could be the reason for delay 
>>> 
>>> Regards,
>>> Ashish
>>> 
>>> On Fri, 11 Aug, 2023, 04:58 Shaurya Gupta,  wrote:
 Hi community
 
 What is the expected P95 latency for Cassandra Read and Write queries 
 executed with Local_Quorum over a table with 3 replicas ? The queries are 
 done using the partition + clustering key and row size in bytes is not too 
 much, maybe 1-2 KB maximum.
 Assuming CPU is not a crunch ?
 
 We observe those to be 40 ms P95 Reads and same for Writes. This looks 
 very high as compared to what we expected. We are using Cassandra 4.0.
 
 Any documentation / numbers will be helpful.
 
 Thanks
 --
 Shaurya Gupta
 
>> 
>> 
>> --
>> Shaurya Gupta
>> 

Re: Materialized View inconsistency issue

[Josh McKenzie]

When it comes to denormalization in Cassandra today your options are to either 
do it yourself in your application layer or rely on Materialized Views to do it 
for you at the server layer. Neither are production-ready approaches out of the 
box (which is one of the biggest flaws in the "provide it server side as a 
feature" approach); both implementations will need you as a user to:
 1. Deal with failure cases (data loss in base table, consistency violations 
between base and view due to failures during write / anti-entropy vs. gc_grace, 
etc) and
 2. Manage the storage implications of a given base write and the denormalized 
writes that it spawns. This is arguably worse with MV's as you have less 
visibility into the fanout and they're easier to create; it was common to see 
folks create 5-10 views on a base table when they were first released and lock 
up tables and exhaust storage disks, not realizing the implications.
The current inability to clearly see and rely on the state of consistency 
between a base and a view is a significant limitation that's shared by both the 
MV implementation and a user-hand-rolled version. @regis I'd be super 
interested to hear more about:
> we made a spark script downloading the master table and the MV, and comparing 
> them and fixing data (as said previously we have very few errors and we run 
> it maybe once a year
Given the inclusion of the spark bulk reader and writer in the project 
ecosystem, this could prove to be something really useful for a lot of users.

In a post-Accord world with atomic durable multi-partition transactions, we 
should be able to create a more robust, consistent implementation of MV's. This 
doesn't solve the problem of "complete data loss on a base table leaves you 
with data in a view that's orphaned; you need to rebuild the view." That said, 
a Materialized Views feature that only has that one caveat of "if you lose data 
in the base you need to recreate the views" would be a significant improvement. 
It should also be pretty trivial to augment the upcoming size commands to 
support future MV's as well (CASSANDRA-12367 
)

So yeah. Denormalization is a Hard Problem. MV's were an attempt to take a 
burden off the user but we just didn't have sufficiently robust primitives to 
build on at that time to get it where it needed to go.

I'm personally still on the fence between whether a skilled user should go with 
hand-rolled vs. MV's today, but for the general populace of C* users (i.e. 
people that don't have time to get into the weeds), they're probably best 
avoided still for now.

On Thu, Aug 10, 2023, at 8:19 PM, MyWorld wrote:
> Hi surbhi ,
> There are 2 drawbacks associated with MV.
> 1. Inconsistent view
> 2. The lock it takes on the base table. This gets worse when you have huge 
> number of clustering keys in a specific partition.
> 
> It's better you re-design a seperate table and let your API do a parallel 
> write on both.
> 
> Regards,
> Ashish
> 
> On Fri, 11 Aug, 2023, 02:03 Surbhi Gupta,  wrote:
>> Thanks everyone.
>> 
>> 
>> On Wed, 9 Aug 2023 at 01:00, Regis Le Bretonnic
>>  wrote:
>> >
>> > Hi Surbhi
>> >
>> > We do use cassandra materialized views even if not recommended.
>> > There are known issues you have to make with. Despite of them, we still 
>> > use VM.
>> > What we observe is :
>> > * there are  inconsistency issues but few. Most of them are rows that 
>> > should not exist in the MV...
>> > * we made a spark script downloading the master table and the MV, and 
>> > comparing them and fixing data (as said previously we have very few errors 
>> > and we run it maybe once a year)
>> >
>> > * Things go very very very bad when you add or remove a node ! Limit this 
>> > operation if possible and do it knowing what can happen (we isolate the 
>> > ring/datacenter and fix data before putting it back to production. We did 
>> > this only once in the last 4 years).
>> >
>> > PS : all proposals avoiding MV failed for our project. Basically managing 
>> > a table like a MV (by deleting and inserting rows from code) is worse and 
>> > more corrupted than what MV does...
>> > The worse issue is adding and removing nodes. Maybe cassandra 4 improves 
>> > this point (not tested yet).
>> >
>> > Have fun...
>> >
>> > Le mar. 8 août 2023 à 22:36, Surbhi Gupta  a 
>> > écrit :
>> >>
>> >> Hi,
>> >>
>> >> We get complaints about Materialized View inconsistency issues.
>> >> We are on 3.11.5 and on 3.11.5 Materialized Views were not production 
>> >> ready.
>> >> We are ok to upgrade.
>> >>
>> >> On which version of cassandra MVs doesnt have inconsistency issues?
>> >>
>> >> Thanks
>> >> Surbhi

Re: Repair errors

[Josh McKenzie]

Quick drive-by observation:
> Did not get replies from all endpoints.. Check the 
> logs on the repair participants for further details

> dropping message of type HINT_REQ due to error
> org.apache.cassandra.net.AsyncChannelOutputPlus$FlushException: The
> channel this output stream was writing to has been closed

> Caused by: io.netty.channel.unix.Errors$NativeIoException:
> writeAddress(..) failed: Connection timed out

> java.lang.RuntimeException: Did not get replies from all endpoints.
These all point to the same shaped problem: for whatever reason, the 
coordinator of this repair didn't receive replies from the replicas executing 
it. Could be that they're dead, could be they took too long, could be they 
never got the start message, etc. Distributed operations are tricky like that.

Logs on the replicas doing the actual repairs should give you more insight; 
this is a pretty low level generic set of errors that basically amounts to "we 
didn't hear back from the other participants in time so we timed out."

On Fri, Aug 4, 2023, at 12:02 PM, Surbhi Gupta wrote:
> Can you please try to do nodetool describecluster from every node of the 
> cluster?
> 
> One time I noticed issue when nodetool status shows all nodes UN but 
> describecluster was not.
> 
> Thanks
> Surbhi
> 
> On Fri, Aug 4, 2023 at 8:59 AM Joe Obernberger  
> wrote:
>> Hi All - been using reaper to do repairs, but it has hung.  I tried to run:
>> nodetool repair -pr
>> on each of the nodes, but they all fail with some form of this error:
>> 
>> error: Repair job has failed with the error message: Repair command #521 
>> failed with error Did not get replies from all endpoints.. Check the 
>> logs on the repair participants for further details
>> -- StackTrace --
>> java.lang.RuntimeException: Repair job has failed with the error 
>> message: Repair command #521 failed with error Did not get replies from 
>> all endpoints.. Check the logs on the repair participants for further 
>> details
>>  at 
>> org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:137)
>>  at 
>> org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
>>  at 
>> java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:633)
>>  at 
>> java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:555)
>>  at 
>> java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:474)
>>  at 
>> java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor.lambda$execute$0(ClientNotifForwarder.java:108)
>>  at java.base/java.lang.Thread.run(Thread.java:829)
>> 
>> Using version 4.1.2-1
>> nodetool status
>> Datacenter: datacenter1
>> ===
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address LoadTokens  Owns  Host 
>> ID   Rack
>> UN  172.16.100.45   505.66 GiB  250 ? 
>> 07bccfce-45f1-41a3-a5c4-ee748a7a9b98  rack1
>> UN  172.16.100.251  380.75 GiB  200 ? 
>> 274a6e8d-de37-4e0b-b000-02d221d858a5  rack1
>> UN  172.16.100.35   479.2 GiB   200 ? 
>> 59150c47-274a-46fb-9d5e-bed468d36797  rack1
>> UN  172.16.100.252  248.69 GiB  200 ? 
>> 8f0d392f-0750-44e2-91a5-b30708ade8e4  rack1
>> UN  172.16.100.249  411.53 GiB  200 ? 
>> 49e4f571-7d1c-4e1e-aca7-5bbe076596f7  rack1
>> UN  172.16.100.38   333.26 GiB  200 ? 
>> 0d9509cc-2f23-4117-a883-469a1be54baf  rack1
>> UN  172.16.100.36   405.33 GiB  200 ? 
>> d9702f96-256e-45ae-8e12-69a42712be50  rack1
>> UN  172.16.100.39   437.74 GiB  200 ? 
>> 93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47  rack1
>> UN  172.16.100.248  344.4 GiB   200 ? 
>> 4bbbe57c-6219-41e5-bbac-de92a9594d53  rack1
>> UN  172.16.100.44   409.36 GiB  200 ? 
>> b2e5366e-8386-40ec-a641-27944a5a7cfa  rack1
>> UN  172.16.100.37   236.08 GiB  120 ? 
>> 08a19658-40be-4e55-8709-812b3d4ac750  rack1
>> UN  172.16.20.16975 GiB 500 ? 
>> 1ccd2cc5-3ee5-43c5-a8c3-7065bdc24297  rack1
>> UN  172.16.100.34   340.77 GiB  200 ? 
>> 352fd049-32f8-4be8-9275-68b145ac2832  rack1
>> UN  172.16.100.42   974.86 GiB  500 ? 
>> b088a8e6-42f3-4331-a583-47ef5149598f  rack1
>> 
>> Note: Non-system keyspaces don't have the same replication settings, 
>> effective ownership information is meaningless
>> 
>> Debug log has:
>> 
>> 
>> DEBUG [ScheduledTasks:1] 2023-08-04 11:56:04,955 
>> MigrationCoordinator.java:264 - Pulling unreceived schema versions...
>> INFO  [HintsDispatcher:11344] 2023-08-04 11:56:21,369 
>> HintsDispatchExecutor.java:318 - Finished hinted handoff of file 
>> 1ccd2cc5-3ee5-43c5-a8c3-7065bdc24297-1690426370160-2.hints to endpoint 
>> /172.16.20.16:7000: 1ccd2cc5-3ee5-43c5-a8c3-7065bdc24297, partially
>> WARN 
>> 

Re: Unsubscribe

[Josh McKenzie]

Email user-unsubscr...@cassandra.apache.org to unsub.

https://cassandra.apache.org/_/community.html

See:
User Mailing List

For broad, opinion-based questions, general discussions, ask how to get help, 
or receive announcements, please subscribe to the user mailing list. Security 
issues need to be reported to the Apache Security Team 
.

Before submitting a new question, please search the forums above or the mailing 
list archive to see if it has already been answered.

New to the Mailing List? Read the Archives 
.

SUBSCRIBE 

 
UNSUBSCRIBE 
























On Mon, Jun 19, 2023, at 11:49 PM, Bharat Kul Ratan wrote:
> 

Apache TAC: assistance for travel to Berlin Buzzwords

[Josh McKenzie]

Cassandra Community!

The Travel Assistance Committee with the Apache Foundation is supporting travel 
to Berlin Buzzwords 2023 (https://2023.berlinbuzzwords.de, 18-20 June 2023) for 
up to 6 people. This conference has lined up pretty well with our project in 
the past and would probably be a great opportunity for folks from our community 
to attend: *"Germany’s most exciting conference on storing, processing, 
streaming and searching large amounts of digital data, with a focus on open 
source software projects"*.

Please see the below message from Gavin McDonald w/the TAC:



Hi All,

The ASF Travel Assistance Committee is supporting taking up to six (6)
people to attend Berlin Buzzwords In June this year.

This includes Conference passes, and travel & accommodation as needed.

Please see our website at https://tac.apache.org for more information and
how to apply.

Applications close on 15th April.

Good luck to those that apply.

Gavin McDonald (VP TAC)



~Josh

Re: Filling in the blank To Do sections on the Apache Cassandra web site

[Josh McKenzie]

Might help, organizationally, to put all these efforts under a single
ticket of "Improve web site Documentation" and add these as sub-tasks.
Should be able to do that translation post-creation (i.e. in its current
state) if that's something that makes sense to you.

On Mon, Feb 26, 2018 at 5:24 PM, Kenneth Brotman <
kenbrot...@yahoo.com.invalid> wrote:

> Here are the related JIRA’s.  Please add content even if It’s not well
> formed compositionally.  Myself or someone else will take it from there.
>
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-14274  The
> troubleshooting section of the web site is empty
>
> https://issues.apache.org/jira/browse/CASSANDRA-14273  The Bulk Loading
> web page on the web site is empty
>
> https://issues.apache.org/jira/browse/CASSANDRA-14272  The Backups web
> page on the web site is empty
>
> https://issues.apache.org/jira/browse/CASSANDRA-14271  The Hints web page
> in the web site is empty
>
> https://issues.apache.org/jira/browse/CASSANDRA-14270  The Read repair
> web page is empty
>
> https://issues.apache.org/jira/browse/CASSANDRA-14269  The Data Modeling
> section of the web site is empty
>
> https://issues.apache.org/jira/browse/CASSANDRA-14268  The
> Architecture:Guarantees web page is empty
>
> https://issues.apache.org/jira/browse/CASSANDRA-14267  The Dynamo web
> page on the Apache Cassandra site is missing content
>
> https://issues.apache.org/jira/browse/CASSANDRA-14266  The Architecture
> Overview web page on the Apache Cassandra site is empty
>
>
>
> Thanks for pitching in.
>
>
>
> Kenneth Brotman
>
>
>
> *From:* Kenneth Brotman [mailto:kenbrot...@yahoo.com.INVALID]
> *Sent:* Monday, February 26, 2018 1:54 PM
> *To:* user@cassandra.apache.org
> *Subject:* RE: Filling in the blank To Do sections on the Apache
> Cassandra web site
>
>
>
> Nice!  Thanks for the help Oliver!
>
>
>
> Kenneth Brotman
>
>
>
> *From:* Oliver Ruebenacker [mailto:cur...@gmail.com]
> *Sent:* Sunday, February 25, 2018 7:12 AM
> *To:* user@cassandra.apache.org
> *Cc:* d...@cassandra.apache.org
> *Subject:* Re: Filling in the blank To Do sections on the Apache
> Cassandra web site
>
>
>
>
>
>  Hello,
>
>   I have some slides about Cassandra
> ,
> feel free to borrow.
>
>  Best, Oliver
>
>
>
> On Fri, Feb 23, 2018 at 7:28 PM, Kenneth Brotman <
> kenbrot...@yahoo.com.invalid> wrote:
>
> These nine web pages on the Apache Cassandra web site have blank To Do
> sections.  Most of the web pages are completely blank.  Mind you there is a
> lot of hard work already done on the documentation.  I’ll make JIRA’s for
> any of the blank sections where there is not already a JIRA.  Then it will
> be on to writing up those sections.  *If you have any text to help me get
> started for any of these sections that would be really cool. *
>
>
>
> http://cassandra.apache.org/doc/latest/architecture/overview.html
>
>
>
> http://cassandra.apache.org/doc/latest/architecture/dynamo.html
>
>
>
> http://cassandra.apache.org/doc/latest/architecture/guarantees.html
>
>
>
> http://cassandra.apache.org/doc/latest/data_modeling/index.html
>
>
>
> http://cassandra.apache.org/doc/latest/operating/read_repair.html
>
>
>
> http://cassandra.apache.org/doc/latest/operating/hints.html
>
>
>
> http://cassandra.apache.org/doc/latest/operating/backups.html
>
>
>
> http://cassandra.apache.org/doc/latest/operating/bulk_loading.html
>
>
>
> http://cassandra.apache.org/doc/latest/troubleshooting/index.html
>
>
>
> Kenneth Brotman
>
>
>
>
>
>
> --
>
> Oliver Ruebenacker
>
> Senior Software Engineer, Diabetes Portal
> , Broad Institute
> 
>
>
>

Re: Cassandra Needs to Grow Up by Version Five!

[Josh McKenzie]

There's a disheartening amount of "here's where Cassandra is bad, and
here's what it needs to do for me for free" happening in this thread.

This is open-source software. Everyone is *strongly encouraged* to submit a
patch to move the needle on *any* of these things being complained about in
this thread.

For the Apache Way  to work,
people need to step up and meaningfully contribute to a project to scratch
their own itch instead of just waiting for a random corporation-subsidized
engineer to happen to have interests that align with them and contribute
that to the project.

Beating a dead horse for things everyone on the project knows are serious
pain points is not productive.

On Wed, Feb 21, 2018 at 5:45 AM, Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:

> On Mon, Feb 19, 2018 at 10:01 AM, Kenneth Brotman <
> kenbrot...@yahoo.com.invalid> wrote:
>
> >
> > >> Cluster wide management should be a big theme in any next major
> release.
> > >>
> > >Na. Stability and testing should be a big theme in the next major
> release.
> > >
> >
> > Double Na on that one Jeff.  I think you have a concern there about the
> > need to test sufficiently to ensure the stability of the next major
> > release.  That makes perfect sense.- for every release, especially the
> > major ones.  Continuous improvement is not a phase of development for
> > example.  CI should be in everything, in every phase.  Stability and
> > testing a part of every release not just one.  A major release should be
> a
> > nice step from the previous major release though.
> >
>
> I guess what Jeff refers to is the tick-tock release cycle experiment,
> which has proven to be a complete disaster by popular opinion.
>
> There's also the "materialized views" feature which failed to materialize
> in the end (pun intended) and had to be declared experimental
> retroactively.
>
> Another prominent example is incremental repair which was introduced as the
> default option in 2.2 and now is not recommended to use because of so many
> corner cases where it can fail.  So again experimental as an afterthought.
>
> Not to mention that even if you are aware of the default incremental and go
> with full repair instead, you're still up for a sad surprise:
> anti-compaction will be triggered despite the "full" repair.  Because
> anti-compaction is only disabled in case of sub-range repair (don't ask
> why), so you need to use something advanced like Reaper if you want to
> avoid that.  I don't think you'll ever find this in the documentation.
>
> Honestly, for an eventually-consistent system like Cassandra anti-entropy
> repair is one of the most important pieces to get right.  And Cassandra
> fails really badly on that one: the feature is not really well designed,
> poorly implemented and under-documented.
>
> In a summary, IMO, Cassandra is a poor implementation of some good ideas.
> It is a collection of hacks, not features.  They sometimes play together
> accidentally, and rarely by design.
>
> Regards,
> --
> Alex
>

Re: CDC usability and future development

[Josh McKenzie]

>
> CDC provides only the mutation as opposed to the full column value, which
> tends to be of limited use for us. Applications might want to know the full
> column value, without having to issue a read back. We also see value in
> being able to publish the full column value both before and after the
> update. This is especially true when deleting a column since this stream
> may be joined with others, or consumers may require other fields to
> properly process the delete.


Philosophically, my first pass at the feature prioritized minimizing impact
to node performance first and usability second, punting a lot of the
de-duplication and RbW implications of having full column values, or
materializing stuff off-heap for consumption from a user and flagging as
persisted to disk etc, for future work on the feature. I don't personally
have any time to devote to moving the feature forward now but as Jeff
indicates, Jay and Simon are both active in the space and taking up the
torch.


On Tue, Jan 30, 2018 at 8:35 PM, Jeff Jirsa  wrote:

> Here's a deck of some proposed additions, discussed at one of the NGCC
> sessions last fall:
>
> https://github.com/ngcc/ngcc2017/blob/master/CassandraDataIngestion.pdf
>
>
>
> On Tue, Jan 30, 2018 at 5:10 PM, Andrew Prudhomme  wrote:
>
> > Hi all,
> >
> > We are currently designing a system that allows our Cassandra clusters to
> > produce a stream of data updates. Naturally, we have been evaluating if
> CDC
> > can aid in this endeavor. We have found several challenges in using CDC
> for
> > this purpose.
> >
> > CDC provides only the mutation as opposed to the full column value, which
> > tends to be of limited use for us. Applications might want to know the
> full
> > column value, without having to issue a read back. We also see value in
> > being able to publish the full column value both before and after the
> > update. This is especially true when deleting a column since this stream
> > may be joined with others, or consumers may require other fields to
> > properly process the delete.
> >
> > Additionally, there is some difficulty with processing CDC itself such
> as:
> > - Updates not being immediately available (addressed by CASSANDRA-12148)
> > - Each node providing an independent streams of updates that must be
> > unified and deduplicated
> >
> > Our question is, what is the vision for CDC development? The current
> > implementation could work for some use cases, but is a ways from a
> general
> > streaming solution. I understand that the nature of Cassandra makes this
> > quite complicated, but are there any thoughts or desires on the future
> > direction of CDC?
> >
> > Thanks
> >
> >
>

Re: [RELEASE] Apache Cassandra 3.1 released

[Josh McKenzie]

Kai,


> The most stable version will be 3.1 because it includes the critical fixes
> in 3.0.1 and some additional bug fixes

3.0.1 and 3.1 are identical. This is a unique overlap specific to 3.0.1 and
3.1.

To summarize, the most stable version should be x.Max(2n+1).z.

Going forward, you can expect the following:
3.2: new features
3.3: stabilization (built on top of 3.2)
3.4: new features
3.5: stabilization (built on top of 3.4)

And in parallel (for the 3.x major version / transition to tick-tock
transition period only):
3.0.2: bugfixes only
3.0.3: bugfixes only
3.0.4: bugfixes only
etc

*Any bugfix that goes into 3.0.X will be in the 3.X line, however not all
bugfixes in 3.X will be in 3.0.X* (bugfixes for new features introduced in
3.2, 3.4, etc will obviously not be back-ported to 3.0.X).

So, for the 3.x line:

   - If you absolutely must have the most stable version of C* and don't
   care at all about the new features introduced in even versions of 3.x, you
   want the 3.0.N release.
   - If you want access to the new features introduced in even release
   versions of 3.x (3.2, 3.4, 3.6), you'll want to run the latest odd version
   (3.3, 3.5, 3.7, etc) after the release containing the feature you want
   access to (so, if the feature's introduced in 3.4 and we haven't dropped
   3.5 yet, obviously you'd need to run 3.4).


This is only going to be the case during the transition phase from old
release cycles to tick-tock. We're targeting changes to CI and quality
focus going forward to greatly increase the stability of the odd releases
of major branches (3.1, 3.3, etc) so, for the 4.X releases, our
recommendation would be to run the highest # odd release for greatest
stability.

Hope that helps clarify.

On Thu, Dec 10, 2015 at 10:34 AM, Kai Wang  wrote:

> Paulo,
>
> Thank you for the examples.
>
> So if I go to download page and see 3.0.1, 3.1 and 3.2. The most stable
> version will be 3.1 because it includes the critical fixes in 3.0.1 and
> some additional bug fixes while doesn't have any new features introduced in
> 3.2. In that sense 3.0.1 becomes obsolete as soon as 3.1 comes out.
>
> To summarize, the most stable version should be x.Max(2n+1).z.
>
> Am I correct?
>
>
> On Thu, Dec 10, 2015 at 6:22 AM, Paulo Motta 
> wrote:
>
>> > Will 3.2 contain the bugfixes that are in 3.0.2 as well?
>>
>> If the bugfix affects both 3.2 and 3.0.2, yes. Otherwise it will only go
>> in the affected version.
>>
>> > Is 3.x.y just 3.0.x plus new stuff? Where most of the time y is 0,
>> unless there's a really serious issue that needs fixing?
>>
>> You can't really compare 3.0.y with 3.x(.y) because they're two different
>> versioning schemes.  To make it a bit clearer:
>>
>> Old model:
>> * x.y.z, where:
>>   * x.y represents the "major" version (eg: 2.1, 2.2)
>>   * z represents the "minor" version (eg: 2.1.1, 2.2.2)
>>
>> New model:
>> * a.b(.c), where:
>>   * a represents the "major" version (3, 4, 5)
>>   * b represents the "minor" version (3.1, 3.2, 4.1, etc), where:
>> * if b is even, it' a tick release, meaning it can contain both
>> bugfixes and new features.
>> * if b is odd, it's a tock release, meaning it can only contain
>> bugfixes.
>>   * c is a "subminor" optional version, which will only happen in
>> emergency situations, for example, if a critical/blocker bug is discovered
>> before the next release is out. So we probably won't have a 3.1.1, unless a
>> critical bug is discovered in 3.1 and needs urgent fix before 3.2.
>>
>> The 3.0.x series is an interim stabilization release using the old
>> versioning scheme, and will only receive bug fixes that affects it.
>>
>> 2015-12-09 18:21 GMT-08:00 Maciek Sakrejda :
>>
>>> I'm still confused, even after reading the blog post twice (and reading
>>> the linked Intel post). I understand what you are doing conceptually, but
>>> I'm having a hard time mapping that to actual planned release numbers.
>>>
>>> > The 3.0.2 will only contain bugfixes, while 3.2 will introduce new
>>> features.
>>>
>>>
>>>
>>
>

Re: Cassandra 2.2.1 stuck at 100% on Windows

[Josh McKenzie]

One option: use process explorer to find out the TID's of the java process (
instructions
),
screen cap that, then also run jstack against the running cassandra process
out to a file a few times (instructions
).

We should be able to at least link up the TID to the hex thread # in the
jstack output to figure out who/what is spinning on there.

On Fri, Oct 16, 2015 at 1:28 PM, Michael Shuler 
wrote:

> On 10/16/2015 12:02 PM, Alaa Zubaidi (PDF) wrote:
>
>> No OOM in any of the log files, and NO long GC at that time.
>> I attached the last 2 minutes before it hangs until we restart cassandra
>> after hour an half.
>>
>
> Your logs show gossip issues with some seed nodes. `nodetool gossipinfo`
> on all nodes might be an interesting place to start.
>
> --
> Michael
>

Apache Cassandra 2.2.0-rc1: calling all Windows users

[Josh McKenzie]

With the upcoming release of Cassandra-2.2 Windows is finally an officially
supported operating system. While many months of JIRA tickets
https://issues.apache.org/jira/issues/?filter=-1jql=project%20%3D%20cassandra%20and%20labels%20%3D%20Windows,
bug fixes, and contributions have gone into making Cassandra on Windows as
seamless of an experience as possible, we need your help as the community
to further kick the tires and let us know if you run into any problems.

Please let us know https://issues.apache.org/jira/browse/CASSANDRA/ if
you find anything that's not working on the platform so we can keep
Cassandra on Windows running strong!

-- 
Joshua McKenzie

Windows-aware Cassandra

[Josh McKenzie]

With the release of Cassandra 2.1.0-rc3, the Cassandra team would like to
open the doors to widespread testing of Cassandra on Windows.  As of this
release we have most of the platform-specific kinks ironed out and would
like to get this into the hands of more developers and users and collect
your feedback.

Please take Cassandra for a spin on the platform, kick the tires, and let
us know of any problems or frustrations you run into.  Some highlights of
what we've been working on as well as some known issues with the 2.X line
of releases on Windows:

*2.1.0:*

   - Launch-script platform parity - re-written in PowerShell
  - Heap size determination based on system memory
  - Runtime override of cassandra-env.ps1 available
  - True background daemon start-up option
  - Graceful shutdown sending ctrl+c to running process via
  stop-server.bat
  - Heap dump and error file output redirection
  - Variety of tuning parameters available in conf/cassandra-env.ps1
  - Based on PowerShell 2.0 - works on both Win7/2008 and Win8/2012
   - Tightened up and constrained some features that were causing
   inconsistent file-system behavior on Windows
   - Cleaned up platform-specific issues uncovered by unit tests

*Known issues:*

   - Snapshot-based repair is disabled until version 3.0 due to java file
   creation flags pre-nio.2 (java bug reference
   http://bugs.java.com/view_bug.do?bug_id=6607535) (CASSANDRA-6907
   https://issues.apache.org/jira/browse/CASSANDRA-6907)
   - Attempts to delete snapshots will throw IOExceptions if SSTableReaders
   have segments of the original SSTables open

*Features deferred to 3.0:*

   - nio.2-based File I/O for FILE_SHARE_DELETE flag on files (
   CASSANDRA-4050 https://issues.apache.org/jira/browse/CASSANDRA-4050)
   - Snapshot-based repair (CASSANDRA-6907
   https://issues.apache.org/jira/browse/CASSANDRA-6907)
   - Pre-emptive opening of compaction results (CASSANDRA-6916
   https://issues.apache.org/jira/browse/CASSANDRA-6916 / CASSANDRA-7365
   https://issues.apache.org/jira/browse/CASSANDRA-7365)


*JIRA Reference*:
Open Windows tickets
https://issues.apache.org/jira/issues/?filter=-1jql=project%20%3D%20cassandra%20and%20resolution%20%3D%20Unresolved%20and%20labels%20%3D%20Windows%20order%20by%20fixversion%20asc
Completed Windows JIRA tickets up to release 2.1.0-rc3
https://issues.apache.org/jira/issues/?filter=-1jql=project%20%3D%20cassandra%20and%20status%20%3D%20resolved%20and%20fixversion%20%3C%203.0%20and%20labels%20%3D%20Windows%20ORDER%20BY%20updated%20desc
Completed Windows JIRA tickets release 3.0
https://issues.apache.org/jira/issues/?filter=-1jql=project%20%3D%20cassandra%20and%20status%20%3D%20resolved%20and%20fixversion%20%3E%3D%203.0%20and%20labels%20%3D%20Windows%20ORDER%20BY%20updated%20desc

-- 
Joshua McKenzie
DataStax -- The Apache Cassandra Company

Re: Windows uname -o not supported.

[Josh McKenzie]

That uname call is in there specifically to check if you're running from
cygwin and force a pidfile to be written for stop-server.bat if so; ctrl+c
functionality in mintty is inconsistent with its signal trapping across
both cygwin and Windows releases.

I wrote the PowerShell start-up scripts with the expectation that they
would be called from the .bat files so please give it a shot running the
batch files from a command-prompt or mintty terminal (or whatever your
terminal of choice is).  I'll look into what's broken when running directly
from PowerShell as this may be a more common use-case than I realized.

Along with that - are you running a 32-bit JVM or 64-bit here?

Xms3072M -Xmx3072M -Xmn768M

would lead to that kind of failure on a 32-bit JVM, though CASSANDRA-7353
https://issues.apache.org/jira/browse/CASSANDRA-7353 should have resolved
that potential problem.  Might be related to the environment in PowerShell
rather than a command-prompt as well.


On Mon, Jun 30, 2014 at 5:11 AM, Lars Schouw schou...@yahoo.com wrote:

 How do I start cassandra on Windows? And what does my environment have to
 look like?

 I am getting an error when starting cassandra on Windows... uname -o not
 supported.
 I am using uname (GNU sh-utils) 2.0
 I am not running in cygwin but just a pure powershell Window.

 My Cassandra version comes directly from git

 Here is what I tried:

 D:\dev\3rdparty\cassandra\cassandra [trunk +1 ~0 -0 !]
 .\bin\cassandra.bat -v -f
 Detected powershell execution permissions.  Running with enhanced startup
 scripts.
 Sourcing cassandra config file:
 D:/dev/3rdparty/cassandra/cassandra/conf/cassandra-env.ps1
 Setting up Cassandra environment
 Starting cassandra server
 Running cassandra with: [java.exe
  -javaagent:D:\dev\3rdparty\cassandra\cassandra\lib\jamm-0.2.6.jar -ea
 -Dlog4j.defa
 ultInitOverride=true -XX:+CMSClassUnloadingEnabled
 -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms3072M -Xmx3
 072M -Xmn768M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
 -XX:StringTableSize=103 -XX:+UseParNewGC -XX:+UseConcMarkSwe
 epGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
  -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB
 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=71
 99 -Dcom.sun.management.jmxremote.ssl=false
 -Dcom.sun.management.jmxremote.authenticate=false  -Dlog4j.configuration=lo
 g4j-server.properties -Dcassandra -Dlogback.configurationFile=logback.xml
 -Dcassandra.logdir=D:\dev\3rdparty\cassandra
 \cassandra/logs
 -Dcassandra.storagedir=D:\dev\3rdparty\cassandra\cassandra/data -cp
 D:\dev\3rdparty\cassandra\cassa

 ndra\conf;D:/dev/3rdparty/cassandra/cassandra/lib/airline-0.6.jar;D:/dev/3rdparty/cassandra/cassandra/lib/antlr-run

 time-3.5.2.jar;D:/dev/3rdparty/cassandra/cassandra/lib/commons-cli-1.1.jar;D:/dev/3rdparty/cassandra/cassandra/lib/

 commons-codec-1.2.jar;D:/dev/3rdparty/cassandra/cassandra/lib/commons-lang3-3.1.jar;D:/dev/3rdparty/cassandra/cassa

 ndra/lib/commons-math3-3.2.jar;D:/dev/3rdparty/cassandra/cassandra/lib/compress-lzf-0.8.4.jar;D:/dev/3rdparty/cassa

 ndra/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar;D:/dev/3rdparty/cassandra/cassandra/lib/disruptor-3.0.1.jar;

 D:/dev/3rdparty/cassandra/cassandra/lib/guava-16.0.jar;D:/dev/3rdparty/cassandra/cassandra/lib/high-scale-lib-1.0.6.j

 ar;D:/dev/3rdparty/cassandra/cassandra/lib/jackson-core-asl-1.9.2.jar;D:/dev/3rdparty/cassandra/cassandra/lib/jacks

 on-mapper-asl-1.9.2.jar;D:/dev/3rdparty/cassandra/cassandra/lib/jamm-0.2.6.jar;D:/dev/3rdparty/cassandra/cassandra/

 lib/javax.inject.jar;D:/dev/3rdparty/cassandra/cassandra/lib/jbcrypt-0.3m.jar;D:/dev/3rdparty/cassandra/cassandra/l

 ib/jline-1.0.jar;D:/dev/3rdparty/cassandra/cassandra/lib/jna-4.0.0.jar;D:/dev/3rdparty/cassandra/cassandra/lib/json

 -simple-1.1.jar;D:/dev/3rdparty/cassandra/cassandra/lib/libthrift-0.9.1.jar;D:/dev/3rdparty/cassandra/cassandra/lib

 /logback-classic-1.1.2.jar;D:/dev/3rdparty/cassandra/cassandra/lib/logback-core-1.1.2.jar;D:/dev/3rdparty/cassandra

 /cassandra/lib/lz4-1.2.0.jar;D:/dev/3rdparty/cassandra/cassandra/lib/metrics-core-2.2.0.jar;D:/dev/3rdparty/cassand

 ra/cassandra/lib/netty-all-4.0.20.Final.jar;D:/dev/3rdparty/cassandra/cassandra/lib/reporter-config-2.1.0.jar;D:/de

 v/3rdparty/cassandra/cassandra/lib/slf4j-api-1.7.2.jar;D:/dev/3rdparty/cassandra/cassandra/lib/snakeyaml-1.11.jar;D

 :/dev/3rdparty/cassandra/cassandra/lib/snappy-java-1.0.5.1.jar;D:/dev/3rdparty/cassandra/cassandra/lib/stream-2.5.2.j

 ar;D:/dev/3rdparty/cassandra/cassandra/lib/stringtemplate-4.0.2.jar;D:/dev/3rdparty/cassandra/cassandra/lib/super-c

 sv-2.1.0.jar;D:/dev/3rdparty/cassandra/cassandra/lib/thrift-server-0.3.5.jar;D:\dev\3rdparty\cassandra\cassandra\bu
 ild\classes\main;D:\dev\3rdparty\cassandra\cassandra\build\classes\thrift
 org.apache.cassandra.service.CassandraDae
 mon]
 

ccm support for Windows

[Josh McKenzie]

As of today, ccm https://github.com/pcmanus/ccmsupports Windows.  It
should work in both cygwin and the general command-prompt though there are
some known issues right now which are documented in the README.

If any Windows users are so inclined to test or tinker I'd be happy to
field questions / concerns / fix bugs; feel free to email me directly about
it.

-- 
Joshua McKenzie
DataStax -- The Apache Cassandra Company

Re: ccm support for Windows

[Josh McKenzie]

The windows dtests take another pull request - this one incredibly minor -
to fix some of the odd pathing in Windows.  I'll get that in today.


On Fri, Mar 14, 2014 at 1:29 PM, Robert Coli rc...@eventbrite.com wrote:

 On Fri, Mar 14, 2014 at 11:23 AM, Josh McKenzie 
 josh.mcken...@datastax.com wrote:

 As of today, ccm https://github.com/pcmanus/ccmsupports Windows.  It
 should work in both cygwin and the general command-prompt though there are
 some known issues right now which are documented in the README.

 If any Windows users are so inclined to test or tinker I'd be happy to
 field questions / concerns / fix bugs; feel free to email me directly about
 it.


 As this enables Windows dtests, I find this quite exciting!
 Congratulations and thanks!

 =Rob




-- 
Joshua McKenzie
DataStax -- The Apache Cassandra Company

Re: ccm support for Windows

[Josh McKenzie]

dtest changes are merged in.  Have fun with that Rob.  ;)

The few I poked at weren't looking clean on Windows - there may be some
timing / wait issues in ccm that aren't playing nice but it's a step in the
right direction.  I'm hoping to overhaul the Windows launching process
before moving on to tightening up ccm and then fixing unit tests and dtests.


On Fri, Mar 14, 2014 at 2:03 PM, Josh McKenzie
josh.mcken...@datastax.comwrote:

 The windows dtests take another pull request - this one incredibly minor -
 to fix some of the odd pathing in Windows.  I'll get that in today.


 On Fri, Mar 14, 2014 at 1:29 PM, Robert Coli rc...@eventbrite.com wrote:

 On Fri, Mar 14, 2014 at 11:23 AM, Josh McKenzie 
 josh.mcken...@datastax.com wrote:

 As of today, ccm https://github.com/pcmanus/ccmsupports Windows.  It
 should work in both cygwin and the general command-prompt though there are
 some known issues right now which are documented in the README.

 If any Windows users are so inclined to test or tinker I'd be happy to
 field questions / concerns / fix bugs; feel free to email me directly about
 it.


 As this enables Windows dtests, I find this quite exciting!
 Congratulations and thanks!

 =Rob




 --
 Joshua McKenzie
 DataStax -- The Apache Cassandra Company




-- 
Joshua McKenzie
DataStax -- The Apache Cassandra Company