Re: Cassandra on RocksDB experiment result

2017-04-19 Thread Jon Haddad
I have no clue what it would take to accomplish a pluggable storage engine, but 
I love this idea.  

Obviously the devil is in the details, & a simple K/V is very different from 
supporting partitions, collections, etc, but this is very cool & seems crazy 
not to explore further.  Will you be open sourcing this work?

Jon


> On Apr 19, 2017, at 9:21 AM, Dikang Gu  wrote:
> 
> Hi Cassandra developers,
> 
> This is Dikang from Instagram, I'd like to share you some experiment
> results we did recently, to use RocksDB as Cassandra's storage engine. In
> the experiment, I built a prototype to integrate Cassandra 3.0.12 and
> RocksDB on single column (key-value) use case, shadowed one of our
> production use case, and saw about 4-6X P99 read latency drop during peak
> time, compared to 3.0.12. Also, the P99 latency became more predictable as
> well.
> 
> Here is detailed note with more metrics:
> 
> https://docs.google.com/document/d/1Ztqcu8Jzh4USKoWBgDJQw82DBurQm
> sV-PmfiJYvu_Dc/edit?usp=sharing
> 
> Please take a look and let me know your thoughts. I think the biggest
> latency win comes from we get rid of most Java garbages created by current
> read/write path and compactions, which reduces the JVM overhead and makes
> the latency to be more predictable.
> 
> We are very excited about the potential performance gain. As the next step,
> I propose to make the Cassandra storage engine to be pluggable (like Mysql
> and MongoDB), and we are very interested in providing RocksDB as one
> storage option with more predictable performance, together with community.
> 
> Thanks.
> 
> -- 
> Dikang



Re: CASSANDRA-9472 Reintroduce off heap memtables - patch to 3.0

2017-07-31 Thread Jon Haddad
+1.  IMO there’s very little reason to use 3.0 at this point.  If someone wants 
to back port and make a 3.0 patch publicly available, cool, but merging it into 
3.0 after 2 years doesn’t make much sense to me.

> On Jul 31, 2017, at 9:26 AM, Jeremiah D Jordan  
> wrote:
> 
> 
>> On Jul 31, 2017, at 12:17 PM, Jeff Jirsa  wrote:
>> On 2017-07-29 10:02 (-0700), Jay Zhuang  
>> wrote: 
>>> Should we consider back-porting it to 3.0 for the community? I think
>>> this is a performance regression instead of new feature. And we have the
>>> feature in 2.1, 2.2.
>>> 
>> 
>> Personally / individually, I'd much rather see 3.0 stabilize.
> 
> +1.  The feature is there in 3.11.x if you are running one of the use cases 
> where this helps, and for most existing things 3.0 and 3.11 are about the 
> same stability, so you can go to 3.11.x if you want to keep using the off 
> heap stuff.
> 
> -Jeremiah
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



Re: [VOTE] Release Apache Cassandra 3.11.1

2017-10-02 Thread Jon Haddad
Same boat as Jeff, +1 since it’s not a regression.

> On Oct 2, 2017, at 2:04 PM, Steinmaurer, Thomas 
> <thomas.steinmau...@dynatrace.com> wrote:
> 
> Jeff of course, not Jon. Sorry :-)
> 
> 
> -Original Message-
> From: Steinmaurer, Thomas [mailto:thomas.steinmau...@dynatrace.com]
> Sent: Montag, 02. Oktober 2017 22:58
> To: dev@cassandra.apache.org
> Subject: RE: [VOTE] Release Apache Cassandra 3.11.1
> 
> Jon,
> 
> yes I also did see it with 3.11.0.
> 
> Thomas
> 
> 
> -Original Message-
> From: Jeff Jirsa [mailto:jji...@gmail.com]
> Sent: Montag, 02. Oktober 2017 22:47
> To: Cassandra DEV <dev@cassandra.apache.org>
> Subject: Re: [VOTE] Release Apache Cassandra 3.11.1
> 
> Thomas, did you see this on 3.11.0 as well, or have you not tried 3.11.0 (I 
> know you probably want fixes from 3.11.1, but let's just clarify that this is 
> or is not a regression).
> 
> If it's not a regression, we should ship this and then hopefully we'll spin a 
> 3.11.2 as soon as this is fixed.
> 
> If it is a regression, I'll flip my vote to -1.
> 
> 
> 
> On Mon, Oct 2, 2017 at 1:29 PM, Steinmaurer, Thomas < 
> thomas.steinmau...@dynatrace.com> wrote:
> 
>> Jon,
>> 
>> please see my latest comment + attached screen from our monitoring here:
>> https://issues.apache.org/jira/browse/CASSANDRA-13754?
>> focusedCommentId=16188758=com.atlassian.jira.
>> plugin.system.issuetabpanels:comment-tabpanel#comment-16188758
>> 
>> Thanks,
>> Thomas
>> 
>> -Original Message-
>> From: Jon Haddad [mailto:jonathan.had...@gmail.com] On Behalf Of Jon
>> Haddad
>> Sent: Montag, 02. Oktober 2017 22:09
>> To: dev@cassandra.apache.org
>> Subject: Re: [VOTE] Release Apache Cassandra 3.11.1
>> 
>> You’re saying the same memory leak happens under 3.11?
>> 
>>> On Oct 2, 2017, at 1:04 PM, Aleksey Yeshchenko <alek...@apple.com>
>> wrote:
>>> 
>>> Thomas,
>>> 
>>> I would maybe agree with waiting for a while because of it, if we
>>> had a
>> proper fix at least under review - or in progress by someone.
>>> 
>>> But this is not a regression, and there’s been a lot of fixes
>> accumulated and not released yet. Arguable worse to hold them back :\
>>> 
>>> —
>>> AY
>>> 
>>> On 2 October 2017 at 20:54:38, Steinmaurer, Thomas (
>> thomas.steinmau...@dynatrace.com) wrote:
>>> 
>>> Jeff,
>>> 
>>> even if it is not a strict regression, this currently forces us to
>>> do a
>> rolling restart every ~ 72hrs to be on the safe-side with -Xmx8G.
>> Luckily this is just a loadtest environment. We don't have 3.11 in 
>> production yet.
>>> 
>>> I can offer to immediately deploy a snapshot build into our loadtest
>> environment, in case this issue gets attention and a fix needs
>> verification at constant load.
>>> 
>>> Thanks,
>>> Thomas
>>> 
>>> -Original Message-
>>> From: Jeff Jirsa [mailto:jji...@gmail.com]
>>> Sent: Montag, 02. Oktober 2017 20:04
>>> To: Cassandra DEV <dev@cassandra.apache.org>
>>> Subject: Re: [VOTE] Release Apache Cassandra 3.11.1
>>> 
>>> +1
>>> 
>>> ( Somewhat concerned that
>>> https://issues.apache.org/jira/browse/CASSANDRA-13754 may not be
>>> fixed,
>> but it's not a strict regression ? )
>>> 
>>> 
>>> 
>>> On Mon, Oct 2, 2017 at 10:58 AM, Michael Shuler
>>> <mich...@pbandjelly.org>
>>> wrote:
>>> 
>>>> I propose the following artifacts for release as 3.11.1.
>>>> 
>>>> sha1: 983c72a84ab6628e09a78ead9e20a0c323a005af
>>>> Git:
>>>> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
>>>> shortlog;h=refs/tags/3.11.1-tentative
>>>> Artifacts:
>>>> https://repository.apache.org/content/repositories/
>>>> orgapachecassandra-1151/org/apache/cassandra/apache-cassandra/3.11.
>>>> 1/
>>>> Staging repository:
>>>> https://repository.apache.org/content/repositories/
>>>> orgapachecassandra-1151/
>>>> 
>>>> The Debian packages are available here:
>>>> http://people.apache.org/~mshuler
>>>> 
>>>> The vote will be open for 72 hours (longer if needed).
>>>> 
>>>> [1]: (CHANGES.txt) https://goo.gl/dZCRk8
>>>> [2]: (NEWS.txt) https

Re: Proposal to retroactively mark materialized views experimental

2017-10-02 Thread Jon Haddad
Developers are also not the only people that are able to make decisions.  
Keeping it in the YAML means an operator can disable it vs a developer *maybe* 
seeing the warning.  Keep in mind not everyone creates tables through CQLSH.

> On Oct 2, 2017, at 2:05 PM, Blake Eggleston <beggles...@apple.com> wrote:
> 
> The message isn't materially different, but it will reach fewer people, 
> later. People typically aren't as attentive to logs as they should be. 
> Developers finding out about new warnings in the logs later than they could 
> have, sometimes even after it's been deployed, is not uncommon. It's happened 
> to me. Requiring a flag will reach everyone trying to use MVs as soon as they 
> start developing against MVs. Logging a warning will reach a subset of users 
> at some point, hopefully. The only downside I can think of for the flag is 
> that it's not as polite.
> 
> On October 2, 2017 at 1:16:10 PM, Josh McKenzie (jmcken...@apache.org) wrote:
> 
> "Nobody is talking about removing MVs."  
> Not precisely true for this email thread:  
> 
> "but should there be some point in the  
> future where we consider removing them from the code base unless they have  
> gotten significant improvement as well?"  
> 
> IMO a .yaml change requirement isn't materially different than barfing a  
> warning on someone's screen during the dev process when they use the DDL  
> for MV's. At the end of the day, it's just a question of how forceful you  
> want that messaging to be. If the cqlsh client prints 'THIS FEATURE IS NOT  
> READY' in big bold letters, that's not going to miscommunicate to a user  
> that 'feature X is ready' when it's not.  
> 
> Much like w/SASI, this is something that's in the code-base that for  
> certain use-cases apparently works just fine. Might be worth considering  
> the approach of making boundaries around those use-cases more rigid instead  
> of throwing the baby out with the bathwater.  
> 
> On Mon, Oct 2, 2017 at 3:32 PM, DuyHai Doan <doanduy...@gmail.com> wrote:  
> 
>> Ok so IF there is a flag to enable MV (à-la UDA/UDF in cassandra.yaml) then  
>> I'm fine with it. I initially understood that we wanted to disable it  
>> definitively. Maybe we should then add an explicit error message when MV is  
>> disabled and someone tries to use it, something like:  
>> 
>> "MV has been disabled, to enable it, turn on the flag  in  
>> cassandra.yaml" so users don't spend 3h searching around  
>> 
>> 
>> On Mon, Oct 2, 2017 at 9:07 PM, Jon Haddad <j...@jonhaddad.com> wrote:  
>> 
>>> There’s a big difference between removal of a protocol that every single  
>>> C* user had to use and disabling a feature which is objectively broken  
>> and  
>>> almost nobody is using. Nobody is talking about removing MVs. If you  
>> want  
>>> to use them you can enable them very trivially, but it should be an  
>>> explicit option because they really aren’t ready for general use.  
>>> 
>>> Claiming disabling by default == removal is not helpful to the  
>>> conversation and is very misleading.  
>>> 
>>> Let’s be practical here. The people that are most likely to put MVs in  
>>> production right now are people new to Cassandra that don’t know any  
>>> better. The people that *should* be using MVs are the contributors to  
>> the  
>>> project. People that actually wrote Cassandra code that can do a patch  
>> and  
>>> push it into prod, and get it submitted upstream when they fix something.  
>>> Yes, a lot of this stuff requires production usage to shake out the bugs,  
>>> that’s fine, but we shouldn’t lie to people and say “feature X is ready”  
>>> when it’s not. That’s a great way to get a reputation as “unstable” or  
>>> “not fit for production."  
>>> 
>>> Jon  
>>> 
>>> 
>>>> On Oct 2, 2017, at 11:54 AM, DuyHai Doan <doanduy...@gmail.com> wrote:  
>>>> 
>>>> "I would (in a patch release) disable MV CREATE statements, and emit  
>>>> warnings for ALTER statements and on schema load if they’re not  
>>> explicitly  
>>>> enabled"  
>>>> 
>>>> --> I find this pretty extreme. Now we have an existing feature sitting  
>>>> there in the base code but forbidden from version xxx onward.  
>>>> 
>>>> Since when do we start removing feature in a patch release ?  
>> (forbidding  
>>> to  
>>>> create new MV == removing the feature, defacto)  
>>>> 
>>>> Eve

Re: [VOTE] Release Apache Cassandra 3.11.1

2017-10-02 Thread Jon Haddad
You’re saying the same memory leak happens under 3.11?  

> On Oct 2, 2017, at 1:04 PM, Aleksey Yeshchenko  wrote:
> 
> Thomas,
> 
> I would maybe agree with waiting for a while because of it, if we had a 
> proper fix at least under review - or in progress by someone.
> 
> But this is not a regression, and there’s been a lot of fixes accumulated and 
> not released yet. Arguable worse to hold them back :\
> 
> —
> AY
> 
> On 2 October 2017 at 20:54:38, Steinmaurer, Thomas 
> (thomas.steinmau...@dynatrace.com) wrote:
> 
> Jeff,  
> 
> even if it is not a strict regression, this currently forces us to do a 
> rolling restart every ~ 72hrs to be on the safe-side with -Xmx8G. Luckily 
> this is just a loadtest environment. We don't have 3.11 in production yet.  
> 
> I can offer to immediately deploy a snapshot build into our loadtest 
> environment, in case this issue gets attention and a fix needs verification 
> at constant load.  
> 
> Thanks,  
> Thomas  
> 
> -Original Message-  
> From: Jeff Jirsa [mailto:jji...@gmail.com]  
> Sent: Montag, 02. Oktober 2017 20:04  
> To: Cassandra DEV   
> Subject: Re: [VOTE] Release Apache Cassandra 3.11.1  
> 
> +1  
> 
> ( Somewhat concerned that  
> https://issues.apache.org/jira/browse/CASSANDRA-13754 may not be fixed, but 
> it's not a strict regression ? )  
> 
> 
> 
> On Mon, Oct 2, 2017 at 10:58 AM, Michael Shuler   
> wrote:  
> 
>> I propose the following artifacts for release as 3.11.1.  
>> 
>> sha1: 983c72a84ab6628e09a78ead9e20a0c323a005af  
>> Git:  
>> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=  
>> shortlog;h=refs/tags/3.11.1-tentative  
>> Artifacts:  
>> https://repository.apache.org/content/repositories/  
>> orgapachecassandra-1151/org/apache/cassandra/apache-cassandra/3.11.1/  
>> Staging repository:  
>> https://repository.apache.org/content/repositories/  
>> orgapachecassandra-1151/  
>> 
>> The Debian packages are available here:  
>> http://people.apache.org/~mshuler  
>> 
>> The vote will be open for 72 hours (longer if needed).  
>> 
>> [1]: (CHANGES.txt) https://goo.gl/dZCRk8  
>> [2]: (NEWS.txt) https://goo.gl/rh24MX  
>> 
>> -  
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org  
>> For additional commands, e-mail: dev-h...@cassandra.apache.org  
>> 
>> 
> The contents of this e-mail are intended for the named addressee only. It 
> contains information that may be confidential. Unless you are the named 
> addressee or an authorized designee, you may not copy or use it, or disclose 
> it to anyone else. If you received it in error please notify us immediately 
> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) 
> is a company registered in Linz whose registered office is at 4040 Linz, 
> Austria, Freistädterstraße 313  
> 
> -  
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org  
> For additional commands, e-mail: dev-h...@cassandra.apache.org  
> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



Re: Weekly Cassandra Wrap-Up: Oct 16 Edition

2017-10-16 Thread Jon Haddad
Regarding the stress tests, if you’re willing to share, I’m starting a repo 
where we can keep a bunch of different stress profiles.  I’d like to start 
running them on releases before we agree to push them out.  If anyone has a 
stress test they are willing to share, please get in touch with me!



> On Oct 16, 2017, at 8:37 AM, Jeff Jirsa  wrote:
> 
> I got some feedback last week that I should try this on Monday morning, so
> let's see if we can nudge a few people into action this week.
> 
> 3.0.15 and 3.11.1 are released. This is a dev list, so that shouldn't be a
> surprise to anyone here - you should have seen the votes and release
> notifications. The people working directly ON Cassandra every day are
> probably very aware of the number and nature of fixes in those versions -
> if you're not aware, the Change lists are HUGE, and some of the fixes are
> VERY IMPORTANT. So this week's wrap-up is really a reflection on the size
> of those two release changelogs.
> 
> One of the advantages of the Cassandra project is the size of the user base
> - I don't know if we have accurate counts (and some of the "surveys" are
> laughable), but we know it's on the order of thousands (probably tens of
> thousands) of companies, and some huge number of instances (not willing to
> speculate here, we know it's at least in the hundreds of thousands, may be
> well into the millions). Historically, the best stabilizer of a release was
> people upgrading their unusual use cases, finding bugs that the developers
> hadn't anticipated (and therefore tests didn't exist for those edge cases),
> reporting them, and the next release would be slightly better than the one
> before it. The chicken/egg problem here is pretty obvious, and while a lot
> of us are spending a lot of time making things better, I want to use this
> email to ask a favor (in 3 parts):
> 
> 1) If you haven't tried 3.0 or 3.11 yet, please spin it up on a test
> cluster. 3.11 would be better, 3.0 is ok too. It doesn't need to be a
> thousand node cluster, most of the weird stuff we've seen in the post-3.0
> world deals with data, not cluster size. Grab some of your prod data if you
> can, throw it into a test cluster, add a node/remove a node, tell us if it
> doesn't work.
> 2) Please run a stress workload against that test cluster, even if it's
> 5-10 minutes. Purpose here is two-fold: like #1, it'll help us find some
> edge cases we haven't seen before, but it'll also help us identify holes in
> stress coverage. We have some tickets to add UDTs to stress (
> https://issues.apache.org/jira/browse/CASSANDRA-13260 ) and LWT (
> https://issues.apache.org/jira/browse/CASSANDRA-7960 ). Ideally your stress
> profile should be more than "80% reads 20% writes" - try to actually model
> your schema and query behavior. Do you use static columns? Do you use
> collections?  If you're unable to model your use case because of a
> deficiency in stress, open a JIRA. If things break, open a JIRA. If it
> works perfectly, I'm interested in seeing your stress yaml and results
> (please send it to me privately, don't spam the list).
> 3) If you're somehow not able to run stress because you don't have hardware
> for a spare cluster, profiling your live cluster is also incredibly useful.
> TLP has some notes on how to generate flame graphs -
> https://github.com/thelastpickle/lightweight-java-profiler - I saw one
> example from a cluster that really surprised me. There are versions and use
> cases that we know have been heavily profiled, but there are probably
> versions and use cases where nobody's ever run much in the way of
> profiling. If you're running openjdk in prod, and you're able to SAFELY
> attach a profiler to generate some flame graphs, please send those to me
> (again, privately please, I don't think the whole list needs a copy).
> 
> My hope in all of this is to build up a corpus of real world use cases (and
> real current state via profiling) that we can leverage to make testing and
> performance better going forward. If I get much in the way of response to
> either of these, I'll try to send out a summary in next week's email).
> 
> - Jeff


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



Re: [VOTE] Release Apache Cassandra 3.11.1

2017-10-02 Thread Jon Haddad
The comment at the end of CASSANDRA-13754 
 is a bit concerning, as 
it was from yesterday and the user is seeing heap issues.  It would be 
unfortunate to have to pull the release if it’s suffering from a major memory 
leak.

> On Oct 2, 2017, at 11:01 AM, Aleksey Yeshchenko  wrote:
> 
> +1
> 
> —
> AY
> 
> On 2 October 2017 at 18:58:57, Michael Shuler (mich...@pbandjelly.org) wrote:
> 
> I propose the following artifacts for release as 3.11.1.  
> 
> sha1: 983c72a84ab6628e09a78ead9e20a0c323a005af  
> Git:  
> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/3.11.1-tentative
>   
> Artifacts:  
> https://repository.apache.org/content/repositories/orgapachecassandra-1151/org/apache/cassandra/apache-cassandra/3.11.1/
>   
> Staging repository:  
> https://repository.apache.org/content/repositories/orgapachecassandra-1151/  
> 
> The Debian packages are available here: http://people.apache.org/~mshuler  
> 
> The vote will be open for 72 hours (longer if needed).  
> 
> [1]: (CHANGES.txt) https://goo.gl/dZCRk8  
> [2]: (NEWS.txt) https://goo.gl/rh24MX  
> 
> -  
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org  
> For additional commands, e-mail: dev-h...@cassandra.apache.org  
> 



Re: [VOTE] Release Apache Cassandra 3.0.15

2017-10-02 Thread Jon Haddad
+1

> On Oct 2, 2017, at 11:00 AM, Brandon Williams  wrote:
> 
> +1
> 
> On Oct 2, 2017 12:58 PM, "Aleksey Yeshchenko"  wrote:
> 
>> +1
>> 
>> —
>> AY
>> 
>> On 2 October 2017 at 18:18:26, Michael Shuler (mich...@pbandjelly.org)
>> wrote:
>> 
>> I propose the following artifacts for release as 3.0.15.
>> 
>> sha1: b32a9e6452c78e6ad08e371314bf1ab7492d0773
>> Git:
>> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
>> shortlog;h=refs/tags/3.0.15-tentative
>> Artifacts:
>> https://repository.apache.org/content/repositories/
>> orgapachecassandra-1150/org/apache/cassandra/apache-cassandra/3.0.15/
>> Staging repository:
>> https://repository.apache.org/content/repositories/
>> orgapachecassandra-1150/
>> 
>> The Debian packages are available here: http://people.apache.org/~mshuler
>> 
>> The vote will be open for 72 hours (longer if needed).
>> 
>> [1]: (CHANGES.txt) https://goo.gl/RyuPpw
>> [2]: (NEWS.txt) https://goo.gl/qxwUti
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



Re: Proposal to retroactively mark materialized views experimental

2017-10-02 Thread Jon Haddad
Having now helped a few folks that have put MVs into prod without realizing 
what they got themselves into, I’m +1 on a flag disabling the feature by 
default.  A WARN message would not have helped them.


> On Oct 2, 2017, at 10:56 AM, Blake Eggleston  wrote:
> 
> Yeah I’m not sure that just emitting a warning is enough. The point is to be 
> super explicit that bad things will happen if you use MVs. I would (in a 
> patch release) disable MV CREATE statements, and emit warnings for ALTER 
> statements and on schema load if they’re not explicitly enabled. Only 
> emitting a warning really reduces visibility where we need it: in the 
> development process.
> 
> By only emitting warning, we're just protecting users that don't run even 
> rudimentary tests before upgrading their clusters. If an operator is going to 
> blindly deploy a database update to prod without testing, they’re going to 
> poke their eye out on something anyway. Whether it’s an MV flag or something 
> else. If we make this change clear in NEWS.txt, and the user@ list, I think 
> that’s the best thing to do.
> 
> 
> On October 2, 2017 at 10:18:52 AM, Jeremiah D Jordan 
> (jeremiah.jor...@gmail.com) wrote:
> 
> Hindsight is 20/20. For 8099 this is the reason we cut the 2.2 release before 
> 8099 got merged.  
> 
> But moving forward with where we are now, if we are going to start adding 
> some experimental flags to things, then I would definitely put SASI on this 
> list as well.  
> 
> For both SASI and MV I don’t know that adding a flags in the cassandra.yaml 
> which prevents their use is the right way to go. I would propose that we emit 
> WARN from the native protocol mechanism when a user does an ALTER/CREATE what 
> ever that tries to use an experiment feature, and probably in the system.log 
> as well.  So someone who is starting new development using them will get a 
> warning showing up in cqlsh “hey the thing you just used is experimental, 
> proceed with caution” and also in their logs.  
> 
> These things are live on clusters right now, and I would not want someone to 
> upgrade their cluster to a new *patch* release and suddenly something that 
> may have been working for them now does not function. Anyway, we need to be 
> careful about how this gets put into practice if we are going to do it 
> retroactively.  
> 
> -Jeremiah  
> 
> 
>> On Oct 1, 2017, at 5:36 PM, Josh McKenzie  wrote:  
>> 
>>> 
>>> I think committing 8099, or at the very least, parts of it, behind an  
>>> experimental flag would have been the right thing to do.  
>> 
>> With a major refactor like that, it's a staggering amount of extra work to  
>> have a parallel re-write of core components of a storage engine accessible  
>> in parallel to the major based on an experimental flag in the same branch.  
>> I think the complexity in the code-base of having two such channels in  
>> parallel would be an altogether different kind of burden along with making  
>> the work take considerably longer. The argument of modularizing a change  
>> like that, however, is something I can get behind as a matter of general  
>> principle. As we discussed at NGCC, the amount of static state in the C*  
>> code-base makes this an aspirational goal rather than a reality all too  
>> often, unfortunately.  
>> 
>> Not looking to get into the discussion of the appropriateness of 8099 and  
>> other major refactors like it (nio MessagingService for instance) - but  
>> there's a difference between building out new features and shielding the  
>> code-base and users from their complexity and reliability and refactoring  
>> core components of the code-base to keep it relevant.  
>> 
>> On Sun, Oct 1, 2017 at 5:01 PM, Dave Brosius  wrote:  
>> 
>>> triggers  
>>> 
>>> 
>>> On 10/01/2017 11:25 AM, Jeff Jirsa wrote:  
>>> 
 Historical examples are anything that you wouldn’t bet your job on for  
 the first release:  
 
 Udf/uda in 2.2  
 Incremental repair - would have yanked the flag following 9143  
 SASI - probably still experimental  
 Counters - all sorts of correctness issues originally, no longer true  
 since the rewrite in 2.1  
 Vnodes - or at least shuffle  
 CDC - is the API going to change or is it good as-is?  
 CQL - we’re on v3, what’s that say about v1?  
 
 Basically anything where we can’t definitively say “this feature is going  
 to work for you, build your product on it” because companies around the  
 world are trying to make that determination on their own, and they don’t  
 have the same insight that the active committers have.  
 
 The transition out we could define as a fixed number of releases or a dev@ 
  
 vote, I don’t think you’ll find something that applies to all experimental 
  
 features, so being flexible is probably the best bet there  
 
 
 
>>> 
>>> 

Re: Proposal to retroactively mark materialized views experimental

2017-10-02 Thread Jon Haddad
There’s a big difference between removal of a protocol that every single C* 
user had to use and disabling a feature which is objectively broken and almost 
nobody is using.  Nobody is talking about removing MVs.  If you want to use 
them you can enable them very trivially, but it should be an explicit option 
because they really aren’t ready for general use.

Claiming disabling by default == removal is not helpful to the conversation and 
is very misleading.  

Let’s be practical here.  The people that are most likely to put MVs in 
production right now are people new to Cassandra that don’t know any better.  
The people that *should* be using MVs are the contributors to the project.  
People that actually wrote Cassandra code that can do a patch and push it into 
prod, and get it submitted upstream when they fix something.  Yes, a lot of 
this stuff requires production usage to shake out the bugs, that’s fine, but we 
shouldn’t lie to people and say “feature X is ready” when it’s not.  That’s a 
great way to get a reputation as “unstable” or “not fit for production."

Jon


> On Oct 2, 2017, at 11:54 AM, DuyHai Doan  wrote:
> 
> "I would (in a patch release) disable MV CREATE statements, and emit
> warnings for ALTER statements and on schema load if they’re not explicitly
> enabled"
> 
> --> I find this pretty extreme. Now we have an existing feature sitting
> there in the base code but forbidden from version xxx onward.
> 
> Since when do we start removing feature in a patch release ? (forbidding to
> create new MV == removing the feature, defacto)
> 
> Even the Thrift protocol has gone through a long process of deprecation and
> will be removed on 4.0
> 
> And if we start opening the Pandora box like this, what's next ? Forbidding
> to create SASI index too ? Removing Vnodes ?
> 
> 
> 
> 
> On Mon, Oct 2, 2017 at 8:16 PM, Jeremiah D Jordan > wrote:
> 
>>> Only emitting a warning really reduces visibility where we need it: in
>> the development process.
>> 
>> How does emitting a native protocol warning reduce visibility during the
>> development process?  If you run CREATE MV and cqlsh then prints out a
>> giant warning statement about how it is an experimental feature I think
>> that is pretty visible during development?
>> 
>> I guess I can see just blocking new ones without a flag set, but we need
>> to be careful here.  We need to make sure we don’t cause a problem for
>> someone that is using them currently, even with all the edge cases issues
>> they have now.
>> 
>> -Jeremiah
>> 
>> 
>>> On Oct 2, 2017, at 2:01 PM, Blake Eggleston 
>> wrote:
>>> 
>>> Yeah, I'm not proposing that we disable MVs in existing clusters.
>>> 
>>> 
>>> On October 2, 2017 at 10:58:11 AM, Aleksey Yeshchenko (alek...@apple.com)
>> wrote:
>>> 
>>> The idea is to check the flag in CreateViewStatement, so creation of new
>> MVs doesn’t succeed without that flag flipped.
>>> 
>>> Obviously, just disabling existing MVs working in a minor would be silly.
>>> 
>>> As for the warning - yes, that should also be emitted. Unconditionally.
>>> 
>>> —
>>> AY
>>> 
>>> On 2 October 2017 at 18:18:52, Jeremiah D Jordan (
>> jeremiah.jor...@gmail.com) wrote:
>>> 
>>> These things are live on clusters right now, and I would not want
>> someone to upgrade their cluster to a new *patch* release and suddenly
>> something that may have been working for them now does not function.
>> Anyway, we need to be careful about how this gets put into practice if we
>> are going to do it retroactively.
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



Re: Proposal to retroactively mark materialized views experimental

2017-09-29 Thread Jon Haddad
I’m very much +1 on this, and to new features in general.  

I think having a clear line in which we classify something as production ready 
would be nice.  It would be great if committers were using the feature in prod 
and could vouch for it’s stability.

> On Sep 29, 2017, at 1:09 PM, Blake Eggleston  wrote:
> 
> Hi dev@,
> 
> I’d like to propose that we retroactively classify materialized views as an 
> experimental feature, disable them by default, and require users to enable 
> them through a config setting before using.
> 
> Materialized views have several issues that make them (effectively) unusable 
> in production. Some of the issues aren’t just implementation problems, but 
> problems with the design that aren’t easily fixed. It’s unfair of us to make 
> features available to users in this state without providing a clear warning 
> that bad or unexpected things are likely to happen if they use it.
> 
> Obviously, this isn’t great news for users that have already adopted MVs, and 
> I don’t have a great answer for that. I think that’s sort of a sunk cost at 
> this point. If they have any MV related problems, they’ll have them whether 
> they’re marked experimental or not. I would expect this to reduce the number 
> of users adopting MVs in the future though, and if they do, it would be 
> opt-in.
> 
> Once MVs reach a point where they’re usable in production, we can remove the 
> flag. Specifics of how the experimental flag would work can be hammered out 
> in a forthcoming JIRA, but I’d imagine it would just prevent users from 
> creating new MVs, and maybe log warnings on startup for existing MVs if the 
> flag isn’t enabled.
> 
> Let me know what you think.
> 
> Thanks,
> 
> Blake


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



Re: Proposal to retroactively mark materialized views experimental

2017-10-04 Thread Jon Haddad
The default part I was referring to incremental repair.

SASI still has a pretty fatal issue where nodes OOM: 
https://issues.apache.org/jira/browse/CASSANDRA-12662 
<https://issues.apache.org/jira/browse/CASSANDRA-12662> 



> On Oct 4, 2017, at 12:21 PM, Pavel Yaskevich <pove...@gmail.com> wrote:
> 
> On Wed, Oct 4, 2017 at 12:09 PM, Jon Haddad <j...@jonhaddad.com 
> <mailto:j...@jonhaddad.com>> wrote:
> 
>> MVs work fine for *some use cases*, not the general use case.  That’s why
>> there should be a flag.  To opt into the feature when the behavior is only
>> known to be correct under a certain set of circumstances.  Nobody is saying
>> the flag should be “enable_terrible_feature_nobody_tested_and_we_all_hate”,
>> or something ridiculous like that.  It’s not an attack against the work
>> done by anyone, the level of effort put in, or minimizing the complexity of
>> the problem.  “enable_materialized_views” would be just fine.
>> 
>> We should be honest to people about what they’re getting into.  You may
>> not be aware of this, but a lot of people still believe Cassandra isn’t a
>> DB that you should put in prod.  It’s because features like SASI, MVs,  or
>> incremental repair get merged in prematurely (or even made the default),
>> without having been thoroughly tested, understood and vetted by trusted
>> community members.  New users hit the snags because they deploy the
>> bleeding edge code and hit the bugs.
>> 
> 
> I beg to differ in case of SASI, it has been tested and vetted and ported
> to different versions. I'm pretty sure it still has better test coverage
> then most of the project does, it's not a "default" and you actually have
> to opt-in to it by creating a custom index, how is that premature or
> misleading to users?
> 
> 
>> 
>> That’s not how the process should work.
>> 
>> Ideally, we’d follow a process that looks a lot more like this:
>> 
>> 1. New feature is built with an opt in flag.  Unknowns are documented, the
>> risk of using the feature is known to the end user.
>> 2. People test and use the feature that know what they’re doing.  They are
>> able to read the code, submit patches, and help flush out the issues.  They
>> do so in low risk environments.  In the case of MVs, they can afford to
>> drop and rebuild the view over a week, or rebuild the cluster altogether.
>> We may not even need to worry as much about backwards compatibility.
>> 3. The feature matures.  More tests are written.  More people become aware
>> of how to contribute to the feature’s stability.
>> 4. After a while, we vote on removing the feature flag and declare it
>> stable for general usage.
>> 
>> If nobody actually cares about a feature (why it was it written in the
>> first place?), then it would never get to 2, 3, 4.  It would take a while
>> for big features like MVs to be marked stable, and that’s fine, because it
>> takes a long time to actually stabilize them.  I think we can all agree
>> they are really, really hard problems to solve, and maybe it takes a while.
>> 
>> Jon
>> 
>> 
>> 
>>> On Oct 4, 2017, at 11:44 AM, Josh McKenzie <jmcken...@apache.org> wrote:
>>> 
>>>> 
>>>> So you’d rather continue to lie to users about the stability of the
>>>> feature rather than admitting it was merged in prematurely?
>>> 
>>> 
>>> Much like w/SASI, this is something that's in the code-base that for
>>>> certain use-cases apparently works just fine.
>>> 
>>> I don't know of any outstanding issues with the feature,
>>> 
>>> There appear to be varying levels of understanding of the implementation
>>>> details of MV's (that seem to directly correlate with faith in the
>>>> feature's correctness for the use-cases recommended)
>>> 
>>> We have users in the wild relying on MV's with apparent success (same
>> holds
>>>> true of all the other punching bags that have come up in this thread)
>>> 
>>> You're right, Jon. That's clearly exactly what I'm saying.
>>> 
>>> 
>>> On Wed, Oct 4, 2017 at 2:39 PM, Jon Haddad <j...@jonhaddad.com> wrote:
>>> 
>>>> So you’d rather continue to lie to users about the stability of the
>>>> feature rather than admitting it was merged in prematurely?  I’d rather
>>>> come clean and avoid future problems, and give people the opportunity to
>>>> stop using MVs rather than let them keep taking risks they’re unaware
>> of.
>>>> This is incredib

Re: Proposal to retroactively mark materialized views experimental

2017-10-04 Thread Jon Haddad
So you’d rather continue to lie to users about the stability of the feature 
rather than admitting it was merged in prematurely?  I’d rather come clean and 
avoid future problems, and give people the opportunity to stop using MVs rather 
than let them keep taking risks they’re unaware of.  This is incredibly 
irresponsible in my opinion.  

> On Oct 4, 2017, at 11:26 AM, Josh McKenzie  wrote:
> 
>> 
>> Oh, come on. You're being disingenuous.
> 
> Not my intent. MV's (and SASI, for example) are fairly well isolated; we
> have a history of other changes that are much more broadly and higher
> impact risk-wise across the code-base.
> 
> If I were an operator and built a critical part of my business on a
> released feature that developers then decided to default-disable as
> 'experimental' post-hoc, I'd think long and hard about using any new
> features in that project in the future (and revisit my confidence in all
> other features I relied on, and the software as a whole). We have users in
> the wild relying on MV's with apparent success (same holds true of all the
> other punching bags that have come up in this thread) and I'd hate to see
> us alienate them by being over-aggressive in the way we handle this.
> 
> I'd much rather we continue to aggressively improve and continue to analyze
> MV's stability before a 4.0 release and then use the experimental flag in
> the future, if at all possible.
> 
> On Wed, Oct 4, 2017 at 2:01 PM, Benedict Elliott Smith 
> <_...@belliottsmith.com>
> wrote:
> 
>> Can't we promote these behavioural flags to keyspace properties (with
>> suitable permissions to edit necessary)?
>> 
>> I agree that enabling/disabling features shouldn't require a rolling
>> restart, and nor should switching their consistency safety level.
>> 
>> I think this would be the most suitable equivalent to ALLOW FILTERING for
>> MVs.
>> 
>> 
>> 
>>> On 4 Oct 2017, at 12:31, Jeremy Hanna 
>> wrote:
>>> 
>>> Not to detract from the discussion about whether or not to classify X or
>> Y as experimental but https://issues.apache.org/jira/browse/CASSANDRA-8303
>>  was originally
>> about operators preventing users from abusing features (e.g. allow
>> filtering).  Could that concept be extended to features like MVs or SASI or
>> anything else?  On the one hand it is nice to be able to set those things
>> dynamically without a rolling restart as well as by user.  On the other
>> it’s less clear about defaults.  There could be a property file or just in
>> the yaml, the operator could specify the default features that are enabled
>> for users and then it could be overridden within that framework.
>>> 
 On Oct 4, 2017, at 10:24 AM, Aleksey Yeshchenko 
>> wrote:
 
 We already have those for UDFs and CDC.
 
 We should have more: for triggers, SASI, and MVs, at least. Operators
>> need a way to disable features they haven’t validated.
 
 We already have sufficient consensus to introduce the flags, and we
>> should. There also seems to be sufficient consensus on emitting warnings.
 
 The debate is now on their defaults for MVs in 3.0, 3.11, and 4.0. I
>> agree with Sylvain that flipping the default in a minor would be invasive.
>> We shouldn’t do that.
 
 For trunk, though, I think we should default to off. When it comes to
>> releasing 4.0 we can collectively decide if there is sufficient trust in
>> MVs at the time to warrant flipping the default to true. Ultimately we can
>> decide this in a PMC vote. If I misread the consensus regarding the default
>> for 4.0, then we might as well vote on that. What I see is sufficient
>> distrust coming from core committers, including the author of the v1
>> design, to warrant opt-in for MVs.
 
 If we don’t trust in them as developers, we shouldn’t be cavalier with
>> the users, either. Not until that trust is gained/regained.
 
 —
 AY
 
 On 4 October 2017 at 13:26:10, Stefan Podkowinski (s...@apache.org)
>> wrote:
 
 Introducing feature flags for enabling or disabling different code paths
 is not sustainable in the long run. It's hard enough to keep up with
 integration testing with the couple of Jenkins jobs that we have.
 Running jobs for all permutations of flags that we keep around, would
 turn out impractical. But if we don't, I'm pretty sure something will
 fall off the radar and it won't take long until someone reports that
 enabling feature X after the latest upgrade will simply not work
>> anymore.
 
 There may also be some more subtle assumptions and cross dependencies
 between features that may cause side effects by disabling a feature (or
 parts of it), even if it's just e.g. a metric value that suddenly won't
 get updated anymore, but is used somewhere else. We'll also have to
 consider migration paths for turning a 

Re: Proposal to retroactively mark materialized views experimental

2017-10-04 Thread Jon Haddad
MVs work fine for *some use cases*, not the general use case.  That’s why there 
should be a flag.  To opt into the feature when the behavior is only known to 
be correct under a certain set of circumstances.  Nobody is saying the flag 
should be “enable_terrible_feature_nobody_tested_and_we_all_hate”, or something 
ridiculous like that.  It’s not an attack against the work done by anyone, the 
level of effort put in, or minimizing the complexity of the problem.  
“enable_materialized_views” would be just fine.

We should be honest to people about what they’re getting into.  You may not be 
aware of this, but a lot of people still believe Cassandra isn’t a DB that you 
should put in prod.  It’s because features like SASI, MVs,  or incremental 
repair get merged in prematurely (or even made the default), without having 
been thoroughly tested, understood and vetted by trusted community members.  
New users hit the snags because they deploy the bleeding edge code and hit the 
bugs. 

That’s not how the process should work.  

Ideally, we’d follow a process that looks a lot more like this:

1. New feature is built with an opt in flag.  Unknowns are documented, the risk 
of using the feature is known to the end user.  
2. People test and use the feature that know what they’re doing.  They are able 
to read the code, submit patches, and help flush out the issues.  They do so in 
low risk environments.  In the case of MVs, they can afford to drop and rebuild 
the view over a week, or rebuild the cluster altogether.  We may not even need 
to worry as much about backwards compatibility.
3. The feature matures.  More tests are written.  More people become aware of 
how to contribute to the feature’s stability.
4. After a while, we vote on removing the feature flag and declare it stable 
for general usage.

If nobody actually cares about a feature (why it was it written in the first 
place?), then it would never get to 2, 3, 4.  It would take a while for big 
features like MVs to be marked stable, and that’s fine, because it takes a long 
time to actually stabilize them.  I think we can all agree they are really, 
really hard problems to solve, and maybe it takes a while.

Jon



> On Oct 4, 2017, at 11:44 AM, Josh McKenzie <jmcken...@apache.org> wrote:
> 
>> 
>> So you’d rather continue to lie to users about the stability of the
>> feature rather than admitting it was merged in prematurely?
> 
> 
> Much like w/SASI, this is something that's in the code-base that for
>> certain use-cases apparently works just fine.
> 
> I don't know of any outstanding issues with the feature,
> 
> There appear to be varying levels of understanding of the implementation
>> details of MV's (that seem to directly correlate with faith in the
>> feature's correctness for the use-cases recommended)
> 
> We have users in the wild relying on MV's with apparent success (same holds
>> true of all the other punching bags that have come up in this thread)
> 
> You're right, Jon. That's clearly exactly what I'm saying.
> 
> 
> On Wed, Oct 4, 2017 at 2:39 PM, Jon Haddad <j...@jonhaddad.com> wrote:
> 
>> So you’d rather continue to lie to users about the stability of the
>> feature rather than admitting it was merged in prematurely?  I’d rather
>> come clean and avoid future problems, and give people the opportunity to
>> stop using MVs rather than let them keep taking risks they’re unaware of.
>> This is incredibly irresponsible in my opinion.
>> 
>>> On Oct 4, 2017, at 11:26 AM, Josh McKenzie <jmcken...@apache.org> wrote:
>>> 
>>>> 
>>>> Oh, come on. You're being disingenuous.
>>> 
>>> Not my intent. MV's (and SASI, for example) are fairly well isolated; we
>>> have a history of other changes that are much more broadly and higher
>>> impact risk-wise across the code-base.
>>> 
>>> If I were an operator and built a critical part of my business on a
>>> released feature that developers then decided to default-disable as
>>> 'experimental' post-hoc, I'd think long and hard about using any new
>>> features in that project in the future (and revisit my confidence in all
>>> other features I relied on, and the software as a whole). We have users
>> in
>>> the wild relying on MV's with apparent success (same holds true of all
>> the
>>> other punching bags that have come up in this thread) and I'd hate to see
>>> us alienate them by being over-aggressive in the way we handle this.
>>> 
>>> I'd much rather we continue to aggressively improve and continue to
>> analyze
>>> MV's stability before a 4.0 release and then use the experimental flag in
>>> the future, if at all possible.
>>> 
&g

Re: [PROPOSAL] Migrate to pytest from nosetests for dtests

2017-11-28 Thread Jon Haddad
+1

I stopped using nose a long time ago in favor of py.test.  It’s a significant 
improvement.

> On Nov 28, 2017, at 10:49 AM, Michael Kjellman  wrote:
> 
> I'd like to propose we move from nosetest to pytest for the dtests. It looks 
> like nosetests is basically abandoned, the python community doesn't like it, 
> it hasn't been updated since 2015, and pytest even has nosetests support 
> which would help us greatly during migration 
> (https://docs.pytest.org/en/latest/nose.html).
> 
> Thoughts?
> 
> best,
> kjellman


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



duration based config settings

2017-12-04 Thread Jon Haddad
I ways back I had entire CASSANDRA-13976 out of sheer annoyance to change the 
hint time to be in minutes instead of ms.  Millisecond based resolution is a 
bit absurd for things like hints.  I figured minutes would be better, but after 
some back and forth realized durations (3h, 30m, etc) would be a lot easier to 
work with, and would probably be appropriate across the board.

I’ve dealt with quite a few clusters in the last year, and I’ve seen a handful 
of fat fingered config files, or non-standard times that make me bust out a 
calculator to be sure I’ve got things sorted out right, hence the original 
issue.

Jeff Jirsa suggested migrating to duration types would result in migration 
pain, and I’m not disagreeing with him.  I think we if were to move to duration 
types, we’d want something like the following:

1. add a blah_blah for every blah_blah_ms setting which accepts a duration
2. convert every setting to use blah_blah 
3. if blah_blah_ms is present, use that for blah_blah and set the duration to ms
4. internally everything converts to ms
5. make all node tool commands use durations 
6. for ever setting that’s switch to blah_blah, leave a note that says the 
setting it’s replacing
7. put a warning when people use blah_blah_ms and suggest the conversation to 
the new config field 
8. *sometime* in the future remove _ms.  Maybe as far as a year or two down the 
line.

This seems to me like a significant change and I wanted to get some more 
opinions on the topic before pushing forward.  Thoughts?

Jon
-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



Re: duration based config settings

2017-12-04 Thread Jon Haddad
Sure, I’m fine w/ letting the _ms settings work indefinitely.  Can revisit 
retiring them if there’s ever a real need, am OK if that never happens.

I’ll create the JIRA.

> On Dec 4, 2017, at 5:19 PM, Nate McCall  wrote:
> 
>> I'd be in favour of never retiring the _ms format though - it's almost
>> free, there's no backward compatibility problems, and it's fairly intuitive
>> so long as we're consistent about it.
>> 
> 
> Agreed on this point. Overall, this will be excellent for usability.
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



Re: custom validation before replication

2017-11-16 Thread Jon Haddad
Looks like you’ve got this thread going on the user & dev ML.  This list is the 
dev one, and is meant for discussion of the Cassandra project.  Would everyone 
mind replying to the thread of the same name on the user list instead?

> On Nov 16, 2017, at 1:36 PM, Abdelkrim Fitouri  wrote:
> 
> ok please find bellow an example:
> 
> Lets suppose that i have a cassandra cluster of 4 nodes / one DC /
> replication factor = 4, So in this architecture i have on full copy of the
> data on each node.
> 
> Imagine now that one node have been hacked and in some way with full access
> to cqlsh session, if data is changed on that node, data will be changed on
> the three other, am i right ?
> 
> imagine now that i am able to know (using cryptographic bases) if one
> column was modified by my API ( => normal way) or not ( => suspicious way),
> and i want to execute this check function just before any replication of a
> keyspace to avoid that all the replica will be affected by that and so a
> rollback will be not easy and the integrity of all the system will be down,
> the check will for example kill the local cassandra service ...
> 
> Hope that my question is more clear now.
> 
> Many thanks for any help.
> 
> 2017-11-16 21:59 GMT+01:00 Nate McCall :
> 
>> On Fri, Nov 17, 2017 at 9:11 AM, Abdelkrim Fitouri 
>> wrote:
>>> Trigger does not resolve my problem because it is not a format validation
>>> issue but an integrity constraint ...
>>> 
>>> My purpose is to check data integrity before replication, by returning an
>>> error and killing the service, so i am killing the node that is supposed
>> to
>>> replicate data after a write action ...
>> 
>> I'm a little confused. Can you provide some specific examples of your
>> requirements?
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> 
> 
> 
> -- 
> 
> Cordialement / Best Regards.
> 
> *Abdelkarim FITOURI*
> 
> LPIC/CEH/ITIL
> 
> System And Security Engineer


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



Re: Apache Cassandra Wiki access

2017-12-07 Thread Jon Haddad
The wiki is effectively dead.

Please contribute to the in tree docs section: 
https://github.com/apache/cassandra/tree/trunk/doc 


I recently merged in an improvement that uses Docker to generate the docs.  The 
short version:

cd ./doc

# build the Docker image
docker-compose build build-docs

# build the documentation
docker-compose run build-docs

Jon

> On Dec 7, 2017, at 11:25 AM, Russell Bateman  wrote:
> 
> It appears that deeper access to the wiki is available for the asking? 
> https://wiki.apache.org/cassandra/FrontPage states that, "most of the 
> information on this Wiki is being deprecated." Is this already done? Please 
> advise.
> 
> If so, please grant this to me. I don't know that I have a "wiki username". 
> If I need one, and need to give it to you, please choose from:
> 
>   my e-mail address
>   russell.bateman
>   windofkeltia
> 
> 
> Note: I'm specifically looking to write a custom/secondary index plug-in, 
> similar to Stratio's Lucene index.
> 
> Thanks,
> 
> Russ



Re: CASSANDRA-8527

2018-01-05 Thread Jon Haddad
I think it’s reasonable to count the number of range tombstones towards the 
total tombstone count / threshold.  

I agree the number of rows shadowed by the tombstones should be tracked 
separately, and we should probably think a bit more about how to add 
configuration / limits around this without burdening the user with a bunch of 
new flags they have to think about.  I would prefer to avoid any more 
configuration settings as complex as back_pressure_strategy going forward.

> On Jan 5, 2018, at 9:36 AM, Alexander Dejanovski <a...@thelastpickle.com> 
> wrote:
> 
> Hi Aleksey,
> 
> ok we'll split the work and only deal with row level tombstones in
> CASSANDRA-8527 <https://issues.apache.org/jira/browse/CASSANDRA-8527> and
> create a follow up ticket to work on the separate counts you're suggesting.
> My understanding of what you say is that you would not include range
> tombstones in the warn/fail threshold, but row level tombstones have
> somewhat an impact that is similar to cell tombstones. They will be
> retained in memory and will be sent to replicas.
> If we don't count them in the thresholds (at least for warnings), people
> will miss the fact that they may be reading a lot of tombstones.
> Are you ok with including those row tombstones as part of the thresholds ?
> This was the original intent for creating this ticket, which was a
> follow-up to CASSANDRA-8477
> <https://issues.apache.org/jira/browse/CASSANDRA-8477>.
> 
> For the follow up ticket, we may want to move the discussion in JIRA once
> we've create the ticket, but a merge listener seems like the right way to
> detect rows shadowed by range tombstones. That would force to change the
> UnfilteredRowIterator interface to include the tombstones/rows/cells stats
> as this is what is returned from the lower levels of the read path.
> Is there any easier way to achieve this that you can think of, as that
> interface is used in many parts of the code ?
> 
> On Wed, Dec 27, 2017 at 1:35 PM Aleksey Yeshchenko <alek...@apple.com>
> wrote:
> 
>> As Jeff says, the number of actual tombstones is no less relevant than the
>> number of
>> cells and rows shadowed by them. And the way row and cell tombstones
>> affect a read
>> query can be very different from the way a big range tombstone might: you
>> potentially
>> have to retain every (regular) tombstone in memory and have replicas ship
>> them to
>> a coordinator, while you can discard everything shadowed by a big RT and
>> only serialize
>> a tiny bit of the data between the replicas and the coordinator.
>> 
>> So a mixed metric that just mixes up rows and cells shadowed by all three
>> kinds of tombstones
>> without any differentiation, while better than not having that visibility
>> at all, is worse than having
>> a detailed rundown. If possible, I’d love to see proper separate counts
>> tracked: range, row, and single-cell tombstones encountered, and # of rows
>> or cells obsoleted by each type.
>> 
>> I know that this is a non-trivial change, however, but hey, it doesn’t
>> have to be a trivial patch if it’s going into 4.0.
>> 
>> In the meantime I think it’d be helpful to report that single count. But I
>> don’t like the idea of redefining what
>> tombstone_warn_threshold and tombstone_failure_threshold mean, even in a
>> major release, as RTs are
>> qualitatively different from other tombstones, and have a much lower
>> impact per dead row.
>> 
>> —
>> AY
>> 
>> On 22 December 2017 at 03:53:47, kurt greaves (k...@instaclustr.com)
>> wrote:
>> 
>> I think that's a good idea for 4.x, but not so for current branches. I
>> think as long as we document it well for 4.0 upgrades it's not so much of a
>> problem. Obviously there will be cases of queries failing that were
>> previously succeeding but we can already use
>> tombstone_failure|warn_threshold to tune around that already. I don't think
>> we need another yaml setting to enable/disable counting deleted rows for
>> these thresholds, especially because it's going into 4.0. It *might* be a
>> good idea to bump the tombstone failure threshold default as a safety net
>> though (as well as put something in the NEWS.txt).
>> 
>> On 21 December 2017 at 20:11, Jon Haddad <j...@jonhaddad.com> wrote:
>> 
>>> I had suggested to Alex we kick this discussion over to the ML because
>> the
>>> change will have a significant impact on the behavior of Cassandra when
>>> doing reads with range tombstones that cover a lot of rows. The behavior
>>> now is a little weird, a single tombstone could shadow hundreds of

Re: CASSANDRA-8527

2017-12-21 Thread Jon Haddad
The question isn’t so much about reporting them (we should), it’s about the 
behavior of tombstone_warn_threshold and tombstone_failure_threshold.  The 
patch changes the behavior to include the number of rows that are passed over 
due to the range tombstones.  We’re interested in feedback on if it makes sense 
to change the current behavior.  I’m a +.5 on the change, it makes sense to me, 
but am wondering if there’s a case we haven’t considered.  At the very least 
we’d need a NEWS entry since it is a behavior change.


> On Dec 21, 2017, at 12:33 PM, DuyHai Doan <doanduy...@gmail.com> wrote:
> 
> +1 to report range tombstones. This one is quite tricky indeed to track
> 
> +1 to Mockito too, with the reserve that it should be used wisely
> 
> On Thu, Dec 21, 2017 at 9:11 PM, Jon Haddad <j...@jonhaddad.com> wrote:
> 
>> I had suggested to Alex we kick this discussion over to the ML because the
>> change will have a significant impact on the behavior of Cassandra when
>> doing reads with range tombstones that cover a lot of rows.  The behavior
>> now is a little weird, a single tombstone could shadow hundreds of
>> thousands or even millions of rows, and the query would probably just time
>> out.  Personally, I’m in favor of the change in behavior of this patch but
>> I wanted to get some more feedback before committing to it.  Are there any
>> objections to what Alex described?
>> 
>> Regarding Mockito, I’ve been meaning to bring this up for a while, and I’m
>> a solid +1 on introducing it to help with testing.  In an ideal world we’d
>> have no singletons and could test everything in isolation, but
>> realistically that’s a multi year process and we just aren’t there.
>> 
>> 
>>> On Dec 19, 2017, at 11:07 PM, Alexander Dejanovski <
>> a...@thelastpickle.com> wrote:
>>> 
>>> Hi folks,
>>> 
>>> I'm working on CASSANDRA-8527
>>> <https://issues.apache.org/jira/browse/CASSANDRA-8527> and would need to
>>> discuss a few things.
>>> 
>>> The ticket makes it visible in tracing and metrics that rows shadowed by
>>> range tombstones were scanned during reads.
>>> Currently, scanned range tombstones aren't reported anywhere which hides
>>> the cause of performance issues during reads when the users perform
>> primary
>>> key deletes.
>>> As such, they do not count in the warn and failure thresholds.
>>> 
>>> While the number of live rows and tombstone cells is counted in the
>>> ReadCommand class, it is currently not possible to count the number of
>>> range tombstones there as they are merged with the rows they shadow
>> before
>>> reaching the class.
>>> Instead, we can count the number of deleted rows that were read , which
>>> already improves diagnosis and show that range tombstones were scanned :
>>> 
>>> if (row.hasLiveData(ReadCommand.this.nowInSec(), enforceStrictLiveness))
>>>   ++liveRows;
>>> else if (!row.primaryKeyLivenessInfo().isLive(ReadCommand.this.
>> nowInSec()))
>>> {
>>>   // We want to detect primary key deletions only.
>>>   // If all cells have expired they will count as tombstones.
>>>  ++deletedRows;
>>> }
>>> 
>>> Deleted rows would be part of the warning threshold so that we can spot
>> the
>>> range tombstone scans in the logs and tracing would look like this :
>>> 
>>> WARN  [ReadStage-2] 2017-12-18 18:22:31,352 ReadCommand.java:491 -
>>> Read 2086 live rows, 2440 deleted rows and 0 tombstone cells for
>>> query..
>>> 
>>> 
>>> Are there any caveats to that approach ?
>>> Should we include the number of deleted rows in the failure threshold or
>>> make it optional, knowing that it could make some queries fail while they
>>> were passing before ?
>>> 
>>> On a side note, is it ok to bring in Mockito in order to make it easier
>>> writing tests ? I would like to use a Spy in order to write some tests
>>> without starting the whole stack.
>>> 
>>> Thanks,
>>> 
>>> 
>>> --
>>> -
>>> Alexander Dejanovski
>>> France
>>> @alexanderdeja
>>> 
>>> Consultant
>>> Apache Cassandra Consulting
>>> http://www.thelastpickle.com
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



Re: [VOTE] Release Apache Cassandra 2.1.20

2018-02-12 Thread Jon Haddad
+1

> On Feb 12, 2018, at 12:30 PM, Michael Shuler  wrote:
> 
> I propose the following artifacts for release as 2.1.20.
> 
> sha1: b2949439ec62077128103540e42570238520f4ee
> Git:
> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/2.1.20-tentative
> Artifacts:
> https://repository.apache.org/content/repositories/orgapachecassandra-1152/org/apache/cassandra/apache-cassandra/2.1.20/
> Staging repository:
> https://repository.apache.org/content/repositories/orgapachecassandra-1152/
> 
> Debian and RPM packages are available here:
> http://people.apache.org/~mshuler
> 
> *** This release addresses an important fix for CASSANDRA-14092 ***
>"Max ttl of 20 years will overflow localDeletionTime"
>https://issues.apache.org/jira/browse/CASSANDRA-14092
> 
> The vote will be open for 72 hours (longer if needed).
> 
> [1]: (CHANGES.txt) https://goo.gl/5i2nw9
> [2]: (NEWS.txt) https://goo.gl/i9Fg2u
> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



Re: [VOTE] Release Apache Cassandra 3.11.2

2018-02-12 Thread Jon Haddad
I’m a +1 on getting this into 3.11.2.  

> On Feb 12, 2018, at 1:11 PM, mck  wrote:
> 
>> I propose the following artifacts for release as 3.11.2.
>> …
>> The vote will be open for 72 hours (longer if needed).
> 
> 
> I just pushed the back-port of CASSANDRA-13080 (under CASSANDRA-14212).
> This is the improvement "Use new token allocation for non bootstrap case as 
> well".
> We've seen that this effects 3.11.1 users and that it would be positive to 
> see it in 3.11.2.
> 
> Any chance we could recut 3.11.2 ?
> 
> regards,
> Mick
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



Re: [VOTE] Release Apache Cassandra 2.2.12

2018-02-13 Thread Jon Haddad
+1


> On Feb 13, 2018, at 7:21 AM, Marcus Eriksson  wrote:
> 
> +1
> 
> On Tue, Feb 13, 2018 at 4:18 PM, Gary Dusbabek  wrote:
> 
>> +1
>> 
>> On Mon, Feb 12, 2018 at 2:30 PM, Michael Shuler 
>> wrote:
>> 
>>> I propose the following artifacts for release as 2.2.12.
>>> 
>>> sha1: 1602e606348959aead18531cb8027afb15f276e7
>>> Git:
>>> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
>>> shortlog;h=refs/tags/2.2.12-tentative
>>> Artifacts:
>>> https://repository.apache.org/content/repositories/
>>> orgapachecassandra-1153/org/apache/cassandra/apache-cassandra/2.2.12/
>>> Staging repository:
>>> https://repository.apache.org/content/repositories/
>>> orgapachecassandra-1153/
>>> 
>>> Debian and RPM packages are available here:
>>> http://people.apache.org/~mshuler
>>> 
>>> *** This release addresses an important fix for CASSANDRA-14092 ***
>>>"Max ttl of 20 years will overflow localDeletionTime"
>>>https://issues.apache.org/jira/browse/CASSANDRA-14092
>>> 
>>> The vote will be open for 72 hours (longer if needed).
>>> 
>>> [1]: (CHANGES.txt) https://goo.gl/QkJeXH
>>> [2]: (NEWS.txt) https://goo.gl/A4iKFb
>>> 
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



Re: [VOTE] Release Apache Cassandra 3.0.16

2018-02-13 Thread Jon Haddad
+1

> On Feb 13, 2018, at 10:52 AM, Josh McKenzie  wrote:
> 
> +1
> 
> On Feb 13, 2018 9:20 AM, "Marcus Eriksson"  wrote:
> 
>> +1
>> 
>> On Tue, Feb 13, 2018 at 1:29 PM, Aleksey Yeshchenko 
>> wrote:
>> 
>>> +1
>>> 
>>> —
>>> AY
>>> 
>>> On 12 February 2018 at 20:31:23, Michael Shuler (mich...@pbandjelly.org)
>>> wrote:
>>> 
>>> I propose the following artifacts for release as 3.0.16.
>>> 
>>> sha1: 91e83c72de109521074b14a8eeae1309c3b1f215
>>> Git:
>>> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
>>> shortlog;h=refs/tags/3.0.16-tentative
>>> Artifacts:
>>> https://repository.apache.org/content/repositories/
>>> orgapachecassandra-1154/org/apache/cassandra/apache-cassandra/3.0.16/
>>> Staging repository:
>>> https://repository.apache.org/content/repositories/
>>> orgapachecassandra-1154/
>>> 
>>> Debian and RPM packages are available here:
>>> http://people.apache.org/~mshuler
>>> 
>>> *** This release addresses an important fix for CASSANDRA-14092 ***
>>> "Max ttl of 20 years will overflow localDeletionTime"
>>> https://issues.apache.org/jira/browse/CASSANDRA-14092
>>> 
>>> The vote will be open for 72 hours (longer if needed).
>>> 
>>> [1]: (CHANGES.txt) https://goo.gl/rLj59Z
>>> [2]: (NEWS.txt) https://goo.gl/EkrT4G
>>> 
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



Re: [VOTE] (Take 3) Release Apache Cassandra 3.11.2

2018-02-14 Thread Jon Haddad
+1

> On Feb 14, 2018, at 1:21 PM, Brandon Williams  wrote:
> 
> +1
> 
> On Wed, Feb 14, 2018 at 3:09 PM, Michael Shuler 
> wrote:
> 
>> I propose the following artifacts for release as 3.11.2.
>> 
>> sha1: 1d506f9d09c880ff2b2693e3e27fa58c02ecf398
>> Git:
>> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
>> shortlog;h=refs/tags/3.11.2-tentative
>> Artifacts:
>> https://repository.apache.org/content/repositories/
>> orgapachecassandra-1158/org/apache/cassandra/apache-cassandra/3.11.2/
>> Staging repository:
>> https://repository.apache.org/content/repositories/
>> orgapachecassandra-1158/
>> 
>> Debian and RPM packages are available here:
>> http://people.apache.org/~mshuler
>> 
>> *** This release addresses an important fix for CASSANDRA-14092 ***
>>"Max ttl of 20 years will overflow localDeletionTime"
>>https://issues.apache.org/jira/browse/CASSANDRA-14092
>> 
>> The vote will be open for 72 hours (longer if needed).
>> 
>> [1]: (CHANGES.txt) https://goo.gl/RLZLrR
>> [2]: (NEWS.txt) https://goo.gl/kpnVHp
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



Re: Why isn't there a separate JVM per table?

2018-02-22 Thread Jon Haddad
Yeah, I’m in the compaction on it’s own JVM camp, in an ideal world where we’re 
isolating crazy GC churning parts of the DB.  It would mean reworking how tasks 
are created and removal of all shared state in favor of messaging + a smarter 
manager, which imo would be a good idea regardless. 

It might be a better use of time (especially for 4.0) to do some GC performance 
profiling and cut down on the allocations, since that doesn’t involve a massive 
effort.  

I’ve been meaning to do a little benchmarking and profiling for a while now, 
and it seems like a few others have the same inclination as well, maybe now is 
a good time to coordinate that.  A nice perf bump for 4.0 would be very 
rewarding.

Jon

> On Feb 22, 2018, at 2:00 PM, Nate McCall  wrote:
> 
> I've heard a couple of folks pontificate on compaction in its own
> process as well, given it has such a high impact on GC. Not sure about
> the value of individual tables. Interesting idea though.
> 
> On Fri, Feb 23, 2018 at 10:45 AM, Gary Dusbabek  wrote:
>> I've given it some thought in the past. In the end, I usually talk myself
>> out of it because I think it increases the surface area for failure. That
>> is, managing N processes is more difficult that managing one process. But
>> if the additional failure modes are addressed, there are some interesting
>> possibilities.
>> 
>> For example, having gossip in its own process would decrease the odds that
>> a node is marked dead because STW GC is happening in the storage JVM. On
>> the flipside, you'd need checks to make sure that the gossip process can
>> recognize when the storage process has died vs just running a long GC.
>> 
>> I don't know that I'd go so far as to have separate processes for
>> keyspaces, etc.
>> 
>> There is probably some interesting work that could be done to support the
>> orgs who run multiple cassandra instances on the same node (multiple
>> gossipers in that case is at least a little wasteful).
>> 
>> I've also played around with using domain sockets for IPC inside of
>> cassandra. I never ran a proper benchmark, but there were some throughput
>> advantages to this approach.
>> 
>> Cheers,
>> 
>> Gary.
>> 
>> 
>> On Thu, Feb 22, 2018 at 8:39 PM, Carl Mueller 
>> wrote:
>> 
>>> GC pauses may have been improved in newer releases, since we are in 2.1.x,
>>> but I was wondering why cassandra uses one jvm for all tables and
>>> keyspaces, intermingling the heap for on-JVM objects.
>>> 
>>> ... so why doesn't cassandra spin off a jvm per table so each jvm can be
>>> tuned per table and gc tuned and gc impacts not impact other tables? It
>>> would probably increase the number of endpoints if we avoid having an
>>> overarching query router.
>>> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



Re: Cassandra Needs to Grow Up by Version Five!

2018-02-21 Thread Jon Haddad
Ken,

Maybe it’s not clear how open source projects work, so let me try to explain.  
There’s a bunch of us who either get paid by someone or volunteer on our free 
time.  The folks that get paid, (yay!) usually take direction on what the 
priorities are, and work on projects that directly affect our jobs.  That means 
that someone needs to care enough about the features you want to work on them, 
if you’re not going to do it yourself. 

Now as others have said already, please put your list of demands in JIRA, if 
someone is interested, they will work on it.  You may need to contribute a 
little more than you’ve done already, be prepared to get involved if you 
actually want to to see something get done.  Perhaps learning a little more 
about Cassandra’s internals and the people involved will reveal some of the 
design decisions and priorities of the project.  

Third, you seem to be a little obsessed with market share.  While market share 
is fun to talk about, *most* of us that are working on and contributing to 
Cassandra do so because it does actually solve a problem we have, and solves it 
reasonably well.  If some magic open source DB appears out of no where and does 
everything you want Cassandra to, and is bug free, keeps your data consistent, 
automatically does backups, comes with really nice cert management, ad hoc 
querying, amazing materialized views that are perfect, no caveats to secondary 
indexes, and somehow still gives you linear scalability without any mental 
overhead whatsoever then sure, people might start using it.  And that’s 
actually OK, because if that happens we’ll all be incredibly pumped out of our 
minds because we won’t have to work as hard.  If on the slim chance that 
doesn’t manifest, those of us that use Cassandra and are part of the community 
will keep working on the things we care about, iterating, and improving things. 
 Maybe someone will even take a look at your JIRA issues.  

Further filling the mailing list with your grievances will likely not help you 
progress towards your goal of a Cassandra that’s easier to use, so I encourage 
you to try to be a little more productive and try to help rather than just 
complain, which is not constructive.  I did a quick search for your name on the 
mailing list, and I’ve seen very little from you, so to everyone’s who’s been 
around for a while and trying to help you it looks like you’re just some random 
dude asking for people to work for free on the things you’re asking for, 
without offering anything back in return.

Jon


> On Feb 21, 2018, at 11:56 AM, Kenneth Brotman  
> wrote:
> 
> Josh, 
> 
> To say nothing is indifference.  If you care about your community, sometimes 
> don't you have to bring up a subject even though you know it's also 
> temporarily adding some discomfort?  
> 
> As to opening a JIRA, I've got a very specific topic to try in mind now.  An 
> easy one I'll work on and then announce.  Someone else will have to do the 
> coding.  A year from now I would probably just knock it out to make sure it's 
> as easy as I expect it to be but to be honest, as I've been saying, I'm not 
> set up to do that right now.  I've barely looked at any Cassandra code; for 
> one; everyone on this list probably codes more than I do, secondly; and 
> lastly, it's a good one for someone that wants an easy one to start with: 
> vNodes.  I've already seen too many people seeking assistance with the vNode 
> setting.
> 
> And you can expect as others have been mentioning that there should be 
> similar ones on compaction, repair and backup. 
> 
> Microsoft knows poor usability gives them an easy market to take over. And 
> they make it easy to switch.
> 
> Beginning at 4:17 in the video, it says the following:
> 
>   "You don't need to worry about replica sets, quorum or read repair.  
> You can focus on writing correct application logic."
> 
> At 4:42, it says:
>   "Hopefully this gives you a quick idea of how seamlessly you can bring 
> your existing Cassandra applications to Azure Cosmos DB.  No code changes are 
> required.  It works with your favorite Cassandra tools and drivers including 
> for example native Cassandra driver for Spark. And it takes seconds to get 
> going, and it's elastically and globally scalable."
> 
> More to come,
> 
> Kenneth Brotman
> 
> -Original Message-
> From: Josh McKenzie [mailto:jmcken...@apache.org] 
> Sent: Wednesday, February 21, 2018 8:28 AM
> To: dev@cassandra.apache.org
> Cc: User
> Subject: Re: Cassandra Needs to Grow Up by Version Five!
> 
> There's a disheartening amount of "here's where Cassandra is bad, and here's 
> what it needs to do for me for free" happening in this thread.
> 
> This is open-source software. Everyone is *strongly encouraged* to submit a 
> patch to move the needle on *any* of these things being complained about in 
> this thread.
> 
> For the Apache Way  to work, 

Re: CASSANDRA-8527

2017-12-21 Thread Jon Haddad
I had suggested to Alex we kick this discussion over to the ML because the 
change will have a significant impact on the behavior of Cassandra when doing 
reads with range tombstones that cover a lot of rows.  The behavior now is a 
little weird, a single tombstone could shadow hundreds of thousands or even 
millions of rows, and the query would probably just time out.  Personally, I’m 
in favor of the change in behavior of this patch but I wanted to get some more 
feedback before committing to it.  Are there any objections to what Alex 
described?  

Regarding Mockito, I’ve been meaning to bring this up for a while, and I’m a 
solid +1 on introducing it to help with testing.  In an ideal world we’d have 
no singletons and could test everything in isolation, but realistically that’s 
a multi year process and we just aren’t there.  


> On Dec 19, 2017, at 11:07 PM, Alexander Dejanovski  
> wrote:
> 
> Hi folks,
> 
> I'm working on CASSANDRA-8527
>  and would need to
> discuss a few things.
> 
> The ticket makes it visible in tracing and metrics that rows shadowed by
> range tombstones were scanned during reads.
> Currently, scanned range tombstones aren't reported anywhere which hides
> the cause of performance issues during reads when the users perform primary
> key deletes.
> As such, they do not count in the warn and failure thresholds.
> 
> While the number of live rows and tombstone cells is counted in the
> ReadCommand class, it is currently not possible to count the number of
> range tombstones there as they are merged with the rows they shadow before
> reaching the class.
> Instead, we can count the number of deleted rows that were read , which
> already improves diagnosis and show that range tombstones were scanned :
> 
> if (row.hasLiveData(ReadCommand.this.nowInSec(), enforceStrictLiveness))
>++liveRows;
> else if (!row.primaryKeyLivenessInfo().isLive(ReadCommand.this.nowInSec()))
> {
>// We want to detect primary key deletions only.
>// If all cells have expired they will count as tombstones.
>   ++deletedRows;
> }
> 
> Deleted rows would be part of the warning threshold so that we can spot the
> range tombstone scans in the logs and tracing would look like this :
> 
> WARN  [ReadStage-2] 2017-12-18 18:22:31,352 ReadCommand.java:491 -
> Read 2086 live rows, 2440 deleted rows and 0 tombstone cells for
> query..
> 
> 
> Are there any caveats to that approach ?
> Should we include the number of deleted rows in the failure threshold or
> make it optional, knowing that it could make some queries fail while they
> were passing before ?
> 
> On a side note, is it ok to bring in Mockito in order to make it easier
> writing tests ? I would like to use a Spy in order to write some tests
> without starting the whole stack.
> 
> Thanks,
> 
> 
> -- 
> -
> Alexander Dejanovski
> France
> @alexanderdeja
> 
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



Re: A JIRA proposing a seperate repository for the online documentation

2018-03-15 Thread Jon Haddad
Murukesh is correct on a very useable, pretty standard process of 
multi-versioned docs.

I’ll put my thoughts in a JIRA epic tonight.  I’ll be a multi-phase process.  
Also correct in that I’d like us to move to Hugo for the site, I’d like us to 
have a unified system between the site & the docs, and hugo has been excellent. 
We run the reaper site & docs off hugo, it works well.  We just don’t do 
multi-versions (because we don’t support multiple): 
https://github.com/thelastpickle/cassandra-reaper/tree/master/src/docs 
. 

Jon

> On Mar 15, 2018, at 8:57 AM, Murukesh Mohanan  
> wrote:
> 
> On Fri, Mar 16, 2018 at 0:19 Kenneth Brotman  >
> wrote:
> 
>> Help me out here.  I could have had a website with support for more than
>> one version done several different ways by now.
>> 
>> A website with several versions of documentation is going to have
>> sub-directories for each version of documentation obviously.  I've offered
>> to create those sub-directories under the "doc" folder of the current
>> repository; and I've offered to move the online documentation to a separate
>> repository and have the sub-directories there.  Both were shot down.  Is
>> there a third way?  If so please just spill the beans.
>> 
> 
> There is. Note that the website is an independent repository. So to host
> docs for multiple versions, only the website's repository (or rather, the
> final built contents) needs multiple directories. You can just checkout
> each branch or tag, generate the docs, make a directory for that branch or
> tag in the website repo, and copy the generated docs there with appropriate
> modifications.
> 
> I do this on a smaller scale using GitHub Pages (repo:
> https://github.com/murukeshm/cassandra 
>  site:
> https://murukeshm.github.io/cassandra/ 
> 
>  > ). The method is a bit
> hacky as I noted in CASSANDRA-13907. A daily cronjobs updated the repo if
> docs are updated. 3.9+ versions are available.
> 
> 
> 
> 
>> Also, no offense to anyone at Sphinx but for a project our size it's not
>> suitable.  We need to move off it now.  It's a problem.
>> 
>> Can we go past this and on to the documenting!  Please help resolve this.
>> 
>> How are we going to:
>> Make the submission of code changes include required changes to
>> documentation including the online documentation?
>> Allow, encourage the online documentation to publish multiple versions of
>> documentation concurrently including all officially supported versions?
> 
> 
> Only on this point: we'll need to start by improving the website build
> process. Michael's comment on 13907 (
> https://issues.apache.org/jira/browse/CASSANDRA-13907?focusedCommentId=16211365=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16211365
>  
> 
> )
> shows it's a painful, fiddly process. That seems to be the main blocker. I
> think Jon has shown interest in moving to Hugo from the current Jekyll
> setup.
> 
> 
> 
>> Move our project onto a more suitable program than Sphinx for our needs?
>> 
>> Kenneth Brotman
>> 
>> -Original Message-
>> From: Eric Evans [mailto:john.eric.ev...@gmail.com]
>> Sent: Thursday, March 15, 2018 7:50 AM
>> To: dev@cassandra.apache.org
>> Subject: Re: A JIRA proposing a seperate repository for the online
>> documentation
>> 
>> On Thu, Mar 15, 2018 at 4:58 AM, Rahul Singh 
>> wrote:
>>> 
>>> I don’t understand why it’s so complicated. In tree docs are as good as
>> any. All the old docs are there in the version control system.
>>> 
>>> All we need to is a) generate docs for old versions b) improve user
>> experience on the site by having it clearly laid out what is latest vs. old
>> docs and c) have some semblance of a search maybe using something like
>> Algolia or whatever.
>> 
>> This.
>> 
>> Keeping the docs in-tree is a huge win, because they can move in lock-step
>> with changes occurring in that branch/version.  I don't think we've been
>> enforcing this, but code-changes that alter documented behavior should be
>> accompanied by corresponding changes to the documentation, or be rejected.
>> Code-changes that correspond with undocumented behavior are an opportunity
>> to include some docs (not grounds to reject a code-review IMO, but
>> certainly an opportunity to politely ask/suggest).
>> 
>> Publishing more than one version (as generated from the respective
>> branches/tags), is a solvable problem.
>> 
 On Thu, Mar 15, 2018 at 1:22 Kenneth Brotman
 

Re: Roadmap for 4.0

2018-04-04 Thread Jon Haddad
+1, well said Scott.

> On Apr 4, 2018, at 5:13 PM, Jonathan Ellis  wrote:
> 
> On Wed, Apr 4, 2018, 7:06 PM Nate McCall  wrote:
> 
>> Top-posting as I think this summary is on point - thanks, Scott! (And
>> great to have you back, btw).
>> 
>> It feels to me like we are coalescing on two points:
>> 1. June 1 as a freeze for alpha
>> 2. "Stable" is the new "Exciting" (and the testing and dogfooding
>> implied by such before a GA)
>> 
>> How do folks feel about the above points?
>> 
> 
> +1
> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



Re: Roadmap for 4.0

2018-04-04 Thread Jon Haddad
Agreed with Josh.  There’s nothing set in stone after we release 4.0, trying to 
extrapolate what we do here for the rest of humanity’s timeline isn’t going to 
be a useful exercise.

Regarding building a big list - it’s of no value.  In fact, if we’re already 
talking about releasing 4.0 we should really only be merging in small features 
that enhance user experience like improving nodetool output or reasonable 
optimizations.  Merging in big features at the very end of the merge window is 
a really great idea to have dozens of follow up bug fix releases that nobody 
considers stable, where the Coli conjecture always wins.  IMO, it would be 
better / more responsible to merge them into trunk *after* we branch for 4.0. 
Yes, that makes the next release less exciting, but I really don’t think 
“exciting” is what we’re shooting for.  I’m more in favor of stable.

Regarding supporting 3.0 / 3.11, since we’re talking about feature freezing 4.0 
2 months from now, and releasing it *sometime* after that, then add 6 months, 
we’re talking about close to an extra year of 3.0 support.  People are, of 
course, free to continue patching 3.0, back porting fixes, etc, but I am 
completely OK with saying there’s only 9 more months of support starting today.

I’m also in the go straight to 3.11 camp.  I see no reason to upgrade to only 
3.0 if you’re on 2.x.  

Jon

> On Apr 4, 2018, at 6:29 AM, Josh McKenzie  wrote:
> 
>> 
>> This discussion was always about the release strategy. There is no
>> separation between the release strategy for 4.0 and the release strategy
>> for the project, they are the same thing and what is intended to be
>> discussed here.
> 
> Not trying to be pedantic here, but the email thread is titled "Roadmap for
> 4.0" and has been concerned with how we get 4.0 out the door. I don't think
> it's implicit that whatever strategy we settle on for 4.0 is intended to
> apply to subsequent releases, since the 3.0.X to 3.X to 4.0
> relationship/delta is different than a 4.0 to 5.0 can be expected to be.
> 
> 
>> sidenote: 3.10 was released in January 2017, and while the changes list for
>> 4.0 is getting quite large there's not much there that's going to win over
>> users. It's mostly refactorings and improvements that affect developers
>> more so than users.
> 
> If you assume most 3. users are on 3.10, this argument makes sense. I
> believe a majority are on 3.0.X or 2.1/2.2, which leaves a minority looking
> at the small delta from 3.10 to 4.0 in the current form.
> 
> 
> 
> On Wed, Apr 4, 2018 at 8:25 AM, kurt greaves  wrote:
> 
>>> 
>>> I'm also a bit sad that we seem to be getting back to our old demons of
>>> trying
>>> to shove as much as we possibly can in the next major as if having a
>>> feature
>>> miss it means it will never happen.
>> 
>> That wasn't the intention of this thread, but that's the response I got.
>> Thought I made it pretty clear that this was about compiling a list of
>> things that people are currently working on and can commit to getting
>> finished soon (which should be a relatively small list considering the
>> limited number of full time contributors).
>> 
>> Of course, we should probably (re-(re-(re-)))start a discussion on release
>>> "strategy" in parallel because it doesn't seem we have one right now, but
>>> that's imo a discussion we should keep separate.
>> 
>> This discussion was always about the release strategy. There is no
>> separation between the release strategy for 4.0 and the release strategy
>> for the project, they are the same thing and what is intended to be
>> discussed here. I don't think it's possible to have a separate discussion
>> on these two things as the release strategy has a pretty big influence on
>> how 4.0 is released.
>> 
>> I'm all for a feature freeze and KISS, but I feel that this really needs a
>> bit more thought before we just jump in and set another precedent for
>> future releases. IMO the Cassandra project has had a seriously bad track
>> record of releasing major versions in the past, and we should probably work
>> at resolving that properly, rather than just continuing the current "let's
>> just try something new every time without really thinking about it".
>> 
>> Some points:
>> 
>>   1.  This strategy means that we don't care about what improvements
>>   actually make it into any given major version. This means that we will
>> have
>>   major releases with nothing/very little desirable for users, and thus
>>   little reason to upgrade other than to stay on a supported version (from
>>   experience this isn't terribly important to users of a database). I
>> think
>>   this inevitably leads to supporting more versions than necessary, and in
>>   general a pretty poor experience for users as we spend more time
>> fighting
>>   bugs in production rather than before we do a release (purely because of
>>   increased frequency of releases).
>>   2. We'll always be driven by feature 

Re: Repair scheduling tools

2018-04-04 Thread Jon Haddad
Implementation details aside, I’m firmly in the “it would be nice of C* could 
take care of it” camp.  Reaper is pretty damn easy to use and people *still* 
don’t put it in prod.  


> On Apr 4, 2018, at 4:16 AM, Rahul Singh  wrote:
> 
> I understand the merits of both approaches. In working with other DBs In the 
> “old country” of SQL, we often had to write indexing sequences manually for 
> important tables. It was “built into the product” but in order to leverage 
> the maximum benefits of indices we had to have different indices other than 
> the clustered (physical index). The process still sucked. It’s never perfect.
> 
> The JVM is already fraught with GC issues and putting another process being 
> managed in the same heapspace is what I’m worried about. Technically the 
> process could be in the same binary but started as a side Car or in the same 
> main process.
> 
> Consider a process called “cassandra-agent” that’s sitting around with a 
> scheduler based on config or a Cassandra table. Distributed in the same 
> release. Shell / service scripts would start it. The end user knows it only 
> by examining the .sh files. This opens possibilities of including a GUI 
> hosted in the same process without cluttering the core coolness of Cassandra.
> 
> Best,
> 
> --
> Rahul Singh
> rahul.si...@anant.us
> 
> Anant Corporation
> 
> On Apr 4, 2018, 2:50 AM -0400, Dor Laor , wrote:
>> We at Scylla, implemented repair in a similar way to the Cassandra reaper.
>> We do
>> that using an external application, written in go that manages repair for
>> multiple clusters
>> and saves the data in an external Scylla cluster. The logic resembles the
>> reaper one with
>> some specific internal sharding optimizations and uses the Scylla rest api.
>> 
>> However, I have doubts it's the ideal way. After playing a bit with
>> CockroachDB, I realized
>> it's super nice to have a single binary that repairs itself, provides a GUI
>> and is the core DB.
>> 
>> Even while distributed, you can elect a leader node to manage the repair in
>> a consistent
>> way so the complexity can be reduced to a minimum. Repair can write its
>> status to the
>> system tables and to provide an api for progress, rate control, etc.
>> 
>> The big advantage for repair to embedded in the core is that there is no
>> need to expose
>> internal state to the repair logic. So an external program doesn't need to
>> deal with different
>> version of Cassandra, different repair capabilities of the core (such as
>> incremental on/off)
>> and so forth. A good database should schedule its own repair, it knows
>> whether the shreshold
>> of hintedhandoff was cross or not, it knows whether nodes where replaced,
>> etc,
>> 
>> My 2 cents. Dor
>> 
>> On Tue, Apr 3, 2018 at 11:13 PM, Dinesh Joshi <
>> dinesh.jo...@yahoo.com.invalid> wrote:
>> 
>>> Simon,
>>> You could still do load aware repair outside of the main process by
>>> reading Cassandra's metrics.
>>> In general, I don't think the maintenance tasks necessarily need to live
>>> in the main process. They could negatively impact the read / write path.
>>> Unless strictly required by the serving path, it could live in a sidecar
>>> process. There are multiple benefits including isolation, faster iteration,
>>> loose coupling. For example - this would mean that the maintenance tasks
>>> can have a different gc profile than the main process and it would be ok.
>>> Today that is not the case.
>>> The only issue I see is that the project does not provide an official
>>> sidecar. Perhaps there should be one. We probably would've not had to have
>>> this discussion ;)
>>> Dinesh
>>> 
>>> On Tuesday, April 3, 2018, 10:12:56 PM PDT, Qingcun Zhou <
>>> zhouqing...@gmail.com> wrote:
>>> 
>>> Repair has been a problem for us at Uber. In general I'm in favor of
>>> including the scheduling logic in Cassandra daemon. It has the benefit of
>>> introducing something like load-aware repair, eg, only schedule repair
>>> while no ongoing compaction or traffic is low, etc. As proposed by others,
>>> we can expose keyspace/table-level configurations so that users can opt-in.
>>> Regarding the risk, yes there will be problems at the beginning but in the
>>> long run, users will appreciate that repair works out of the box, just like
>>> compaction. We have large Cassandra deployments and can work with Netflix
>>> folks for intensive testing to boost user confidence.
>>> 
>>> On the other hand, have we looked into how other NoSQL databases do repair?
>>> Is there a side car process?
>>> 
>>> 
>>> On Tue, Apr 3, 2018 at 9:21 PM, sankalp kohli >> wrote:
>>> 
 Repair is critical for running C* and I agree with Roopa that it needs to
 be part of the offering. I think we should make it easy for new users to
 run C*.
 
 Can we have a side car process which we can add to Apache Cassandra
 offering and we can put this repair their? I am 

Re: Roadmap for 4.0

2018-04-12 Thread Jon Haddad
Sept works for me too.  I’ll be involved in the validation process before the 
cutoff date.  


> On Apr 12, 2018, at 3:17 PM, Carlos Rolo  wrote:
> 
> I will commit time to test (not a full validation, but at least go through
> operations) regardless of the date. Both seems fine to me.
> 
> Regards,
> 
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
> 
> Pythian - Love your data
> 
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
> *linkedin.com/in/carlosjuzarterolo
> *
> Mobile: +351 918 918 100
> www.pythian.com
> 
> On Thu, Apr 12, 2018 at 11:00 PM, Joseph Lynch 
> wrote:
> 
>> The Netflix team prefers September as well. We don't have time before that
>> to do a full certification (e2e and performance testing), but can probably
>> work it into end of Q3 / start of Q4.
>> 
>> I personally hope that the extra time gives us as a community a chance to
>> come up with a compelling user story for why users would want to upgrade. I
>> don't feel we have one right now.
>> 
>> -Joey
>> 
>> 
>> On Thu, Apr 12, 2018 at 2:51 PM, Ariel Weisberg  wrote:
>> 
>>> Hi,
>>> 
>>> +1 to September 1st. I know I will have much better availability then.
>>> 
>>> Ariel
>>> On Thu, Apr 12, 2018, at 5:15 PM, Sankalp Kohli wrote:
 +1 with Sept 1st as I am seeing willingness for people to test it after
>>> it
 
> On Apr 12, 2018, at 13:59, Ben Bromhead  wrote:
> 
> While I would prefer earlier, if Sept 1 gets better buy-in and we can
>>> have
> broader commitment to testing. I'm super happy with that. As Nate
>> said,
> having a solid line to work towards is going to help massively.
> 
> On Thu, Apr 12, 2018 at 4:07 PM Nate McCall 
>>> wrote:
> 
>>> If we push it to Sept 1 freeze, I'll personally spend a lot of time
>> testing.
>>> 
>>> What can I do to help convince the Jun1 folks that Sept1 is
>>> acceptable?
>> 
>> I can come around to that. At this point, I really just want us to
>> have a date we can start talking to/planning around.
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> --
> Ben Bromhead
> CTO | Instaclustr 
> +1 650 284 9692
> Reliability at Scale
> Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
 For additional commands, e-mail: dev-h...@cassandra.apache.org
 
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>>> 
>> 
> 
> -- 
> 
> 
> --
> 
> 
> 
> 
> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



Re: Roadmap for 4.0

2018-04-03 Thread Jon Haddad
I’d prefer to time box it as well.  I like Sylvain’s suggestion, although I’d 
also be comfortable with setting a more aggressive cutoff date for features 
(maybe a month), given all the stuff that’s already in.

If we plan on a follow up (4.1/5.0) in 6 months I *hope* there will be less of 
a desire to do a bunch of last minute feature merges, maybe I’m too optimistic.

Jon

 

> On Apr 3, 2018, at 9:48 AM, Ben Bromhead  wrote:
> 
> +1
> 
> Even though I suggested clearing blockers, I'm equally happy with a
> time-boxed event to draw the line in the sand. As long as its something
> clear to work towards with appropriate commitment from folk.
> 
> On Tue, Apr 3, 2018 at 8:10 AM Sylvain Lebresne  >
> wrote:
> 
>> For what it's worth (and based on the project experience), I think the
>> strategy
>> of "let's agree on a list of tickets everyone would love to get in before
>> we
>> freeze 4.0" doesn't work very well (it's largely useless, expect for making
>> us
>> feel good about not releasing anything). Those lists always end up being
>> too
>> big especially given we have no control on people's ability to contribute
>> (some stuffs will always lag for a very long time, even when they sound
>> really cool on paper).
>> 
>> I'm also a bit sad that we seem to be getting back to our old demons of
>> trying
>> to shove as much as we possibly can in the next major as if having a
>> feature
>> miss it means it will never happen. The 4.0 changelog is big already and we
>> haven't made a release with new features in almost a year now, so I
>> personally
>> think we should start being a bit more aggressive with it and learn to get
>> comfortable letting feature slip if they are not ready.
>> 
>> My concrete proposal would be to declare a feature freeze for 4.0 in 2
>> months,
>> so say June 1th. That leave some time for finishing features that are in
>> progress, but not too much to get derailed. And let's be strict on that
>> freeze.
>> After that, we'll see how quickly we can get stuffs to stabilize but I'd
>> suggest aiming for an alpha 3-4 weeks after that.
>> 
>> Of course, we should probably (re-(re-(re-)))start a discussion on release
>> "strategy" in parallel because it doesn't seem we have one right now, but
>> that's imo a discussion we should keep separate.
>> 
>> --
>> Sylvain
>> 
>> 
>> On Mon, Apr 2, 2018 at 4:54 PM DuyHai Doan  wrote:
>> 
>>> My wish list:
>>> 
>>> * Add support for arithmetic operators (CASSANDRA-11935)
>>> * Allow IN restrictions on column families with collections
>>> (CASSANDRA-12654)
>>> * Add support for + and - operations on dates (CASSANDRA-11936)
>>> * Add the currentTimestamp, currentDate, currentTime and currentTimeUUID
>>> functions (CASSANDRA-13132)
>>> * Allow selecting Map values and Set elements (CASSANDRA-7396)
>>> 
>>> Those are mostly useful for timeseries data models and I guess has no
>>> significant impact on the internals and operations so the risk of
>>> regression is low
>>> 
>>> On Mon, Apr 2, 2018 at 4:33 PM, Jeff Jirsa  wrote:
>>> 
 9608 (java9)
 
 --
 Jeff Jirsa
 
 
> On Apr 2, 2018, at 3:45 AM, Jason Brown 
>> wrote:
> 
> The only additional tickets I'd like to mention are:
> 
> https://issues.apache.org/jira/browse/CASSANDRA-13971 - Automatic
> certificate management using Vault
> - Stefan's Vault integration work. A sub-ticket, CASSANDRA-14102,
 addresses
> encryption at-rest, subsumes CASSANDRA-9633 (SSTable encryption) -
>>> which
 I
> doubt I would be able to get to any time this year. It would
>> definitely
 be
> nice to have a clarified encryption/security story for 4.0.
> 
> https://issues.apache.org/jira/browse/CASSANDRA-11990 - Address rows
 rather
> than partitions in SASI
> - a nice update for SASI, but not critical.
> 
> -Jason
> 
>> On Sat, Mar 31, 2018 at 6:53 PM, Ben Bromhead 
 wrote:
>> 
>> Apologies all, I didn't realize I was responding to this discussion
 only on
>> the @user list. One of the perils of responding to a thread that is
>> on
 both
>> user and dev...
>> 
>> For context, I have included my response to Kurt's previous
>> discussion
 on
>> this topic as it only ended up on the user list.
>> 
>> *After some further discussions with folks offline, I'd like to
>> revive
 this
>> discussion. *
>> 
>> *As Kurt mentioned, to keep it simple I if we can simply build
>>> consensus
>> around what is in for 4.0 and what is out. We can then start the
 process of
>> working off a 4.0 branch towards betas and release candidates. Again
>>> as
>> Kurt mentioned, assigning a timeline to it right now is difficult,
>> but
>> having a firm line in the sand around what features/patches are in,

Re: Paying off tech debt and correctly naming things

2018-03-22 Thread Jon Haddad
Cool.  I think there’s general agreement that doing this in as small bites as 
possible is going to be the best approach.  I have no interest in mega patches. 
 

>  The combined approach takes a
> change that's already non-trivially dealing with complex subsystem
> changes and injects a bunch of trivial renaming noise across unrelated
> subsystems into the signal of an actual logic refactor.

I agree.  This is why I like the idea of proactively working to improve the 
readability of the codebase as a specific goal, rather than being wrapped into 
some other unrelated patch.  Keeping the scope in check is the challenge.  
Simple class and method renames, as several have pointed out, is easy enough 
with IDEA.  

I’ll start with class renames, as individual patches for each of them.  I’ll be 
sure to call it out on the ML.  First one will be ColumnFamilyStore -> 
TableStore.  

Jon

> On Mar 22, 2018, at 7:13 AM, Jason Brown <jasedbr...@gmail.com> wrote:
> 
> Jon,
> 
> Thanks for bringing up this topic. I'll admit that I've been around this
> code base for long enough, and have enough accumulated history, that I
> probably can't fully appreciate the impact for a newcomer wrt naming.
> However, as Josh points out, this situation probably happens to "every
> non-trivially aged code-base ever".
> 
> One thing I'd like to add is that with these types of large refactoring
> changes, the review effort is non-trivial. This is because the review still
> has to ensure that correctness is preserved and it's easy to overlook a
> seemingly innocuous change.
> 
> That being said, I am supportive of this effort. However, I believe it's
> going to be best, for contributor and reviewer, to break it up into
> smaller, more digestible pieces. I'd also like to request that we not go
> whole hog and try to do everything in a compressed time frame; reviewer
> availability is already stretched thin and I'm afraid of deepening the
> review queue, especially mine :)
> 
> Thanks,
> 
> -Jason
> 
> 
> 
> 
> On Thu, Mar 22, 2018 at 6:41 AM, Josh McKenzie <jmcken...@apache.org> wrote:
> 
>>> Some of us have big patches in flight, things that actually
>>> pay off some technical debt, and dealing with such renames is rebase
>> hell :\
>> For sure, but with a code-base this old / organically grown, I expect
>> this will always be the case. If we're talking something as simple as
>> an intellij rename refactor, while menial, couldn't someone with a
>> giant patch just do the same thing on their side and spend half an
>> hour of their life clicking next? ;)
>> 
>>> That said, there is good time for such renames - it’s during
>>> those major refactors and rewrites. When you are
>>> changing a subsystem, might as well do the appropriate renames.
>> Does that hold true for a code-base with as much static state and
>> abstraction leaking / bad factoring as we have? (i.e. every
>> non-trivially aged code-base ever) The combined approach takes a
>> change that's already non-trivially dealing with complex subsystem
>> changes and injects a bunch of trivial renaming noise across unrelated
>> subsystems into the signal of an actual logic refactor.
>> 
>> On Thu, Mar 22, 2018 at 9:31 AM, Aleksey Yeshchenko <alek...@apple.com>
>> wrote:
>>> Poor and out-of-date naming of things is probably the least serious part
>> of our technical debt. Bad factoring, and straight-up
>>> poorly written components is where it’s really at.
>>> 
>>> Doing a big rename for rename sake alone does more harm than it is good,
>> sometimes. Some of us have big patches
>>> in flight, things that actually pay off some technical debt, and dealing
>> with such renames is rebase hell :\
>>> 
>>> That said, there is good time for such renames - it’s during those major
>> refactors and rewrites. When you are
>>> changing a subsystem, might as well do the appropriate renames.
>>> 
>>> —
>>> AY
>>> 
>>> On 20 March 2018 at 22:04:48, Jon Haddad (j...@jonhaddad.com) wrote:
>>> 
>>> Whenever I hop around in the codebase, one thing that always manages to
>> slow me down is needing to understand the context of the variable names
>> that I’m looking at. We’ve now removed thrift the transport, but the
>> variables, classes and comments still remain. Personally, I’d like to go in
>> and pay off as much technical debt as possible by refactoring the code to
>> be as close to CQL as possible. Rows should be rows, not partitions, I’d
>> love to see the term column family removed forever in favor of always using
>> tables. Tha

Paying off tech debt and correctly naming things

2018-03-20 Thread Jon Haddad
Whenever I hop around in the codebase, one thing that always manages to slow me 
down is needing to understand the context of the variable names that I’m 
looking at.  We’ve now removed thrift the transport, but the variables, classes 
and comments still remain.  Personally, I’d like to go in and pay off as much 
technical debt as possible by refactoring the code to be as close to CQL as 
possible.  Rows should be rows, not partitions, I’d love to see the term column 
family removed forever in favor of always using tables.  That said, it’s a big 
task.  I did a quick refactor in a branch, simply changing the 
ColumnFamilyStore class to TableStore, and pushed it up to GitHub. [1]

Didn’t click on the link?  That’s ok.  The TL;DR is that it’s almost 2K LOC 
changed across 275 files.  I’ll note that my branch doesn’t change any of the 
almost 1000 search results of “columnfamilystore” found in the codebase and 
hundreds of tests failed on my branch in CircleCI, so that 2K LOC change would 
probably be quite a bit bigger.  There is, of course, a lot more than just 
renaming this one class, there’s thousands of variable names using any manner 
of “cf”, “cfs”, “columnfamily”, names plus comments and who knows what else.  
There’s lots of references in probably every file that would have to get 
updated.

What are people’s thoughts on this?  We should be honest with ourselves and 
know this isn’t going to get any easier over time.  It’s only going to get more 
confusing for new people to the project, and having to figure out “what kind of 
row am i even looking at” is a waste of time.  There’s obviously a much bigger 
impact than just renaming a bunch of files, there’s any number of patches and 
branches that would become outdated, plus anyone pulling in Cassandra as a 
dependency would be affected.  I don’t really have a solution for the 
disruption other than “leave it in place”, but in my mind that’s not a great 
(or even good) solution.

Anyways, enough out of me.  My concern for ergonomics and naming might be 
significantly higher than the rest of the folks working in the code, and I 
wanted to put a feeler out there before I decided to dig into this in a more 
serious manner. 

Jon

[1] 
https://github.com/apache/cassandra/compare/trunk...rustyrazorblade:refactor_column_family_store?expand=1
 


Re: Audit logging to tables.

2019-04-03 Thread Jon Haddad
; > > > >> > > > > > >>> wrote:
> >  >> > > > >> > > > > > >>>
> >  >> > > > >> > > > > > >>>> I strongly echo Josh’s sentiment. Imagine
> > losing
> >  >> > audit
> >  >> > > > >> entries
> >  >> > > > >> > > > > > because C*
> >  >> > > > >> > > > > > >>>> is overloaded? It’s fine if you don’t care
> > about
> >  >> > losing
> >  >> > > > >> audit
> >  >> > > > >> > > > > entries.
> >  >> > > > >> > > > > > >>>>
> >  >> > > > >> > > > > > >>>> Dinesh
> >  >> > > > >> > > > > > >>>>
> >  >> > > > >> > > > > > >>>>> On Feb 28, 2019, at 6:41 AM, Joshua McKenzie <
> >  >> > > > >> > > > jmcken...@apache.org
> >  >> > > > >> > > > > >
> >  >> > > > >> > > > > > >>>> wrote:
> >  >> > > > >> > > > > > >>>>>
> >  >> > > > >> > > > > > >>>>> One of the things we've run into
> > historically, on
> >  >> a
> >  >> > > > *lot*
> >  >> > > > >> of
> >  >> > > > >> > > > axes,
> >  >> > > > >> > > > > is
> >  >> > > > >> > > > > > >>>> that
> >  >> > > > >> > > > > > >>>>> "just put it in C*" for various functionality
> >  >> looks
> >  >> > > > great
> >  >> > > > >> > from
> >  >> > > > >> > > a
> >  >> > > > >> > > > > user
> >  >> > > > >> > > > > > >>> and
> >  >> > > > >> > > > > > >>>>> usability perspective, and proves to be
> > something
> >  >> > of a
> >  >> > > > >> > > nightmare
> >  >> > > > >> > > > > from
> >  >> > > > >> > > > > > >>> an
> >  >> > > > >> > > > > > >>>>> admin / cluster behavior perspective.
> >  >> > > > >> > > > > > >>>>>
> >  >> > > > >> > > > > > >>>>> i.e. - cluster suffering so you're writing
> > hints?
> >  >> > > Write
> >  >> > > > >> them
> >  >> > > > >> > to
> >  >> > > > >> > > > C*
> >  >> > > > >> > > > > > >>> tables
> >  >> > > > >> > > > > > >>>>> and watch the cluster suffer more! :)
> >  >> > > > >> > > > > > >>>>> Same thing probably holds true for audit
> > logging -
> >  >> > at
> >  >> > > a
> >  >> > > > >> time
> >  >> > > > >> > > > frame
> >  >> > > > >> > > > > > when
> >  >> > > > >> > > > > > >>>>> things are getting hairy w/a cluster, if
> > you're
> >  >> > > writing
> >  >> > > > >> that
> >  >> > > > >> > > > audit
> >  >> > > > >> > > > > > >>>> logging
> >  >> > > > >> > > > > > >>>>> into C* proper (and dealing with ser/deser,
> >  >> > compaction
> >  >> > > > >> > > pressure,
> >  >> > > > >> > > > > > >>> flushing
> >  >> > > > >> > > > > > >>>>> pressure, etc) from that, there's a
> > compo

Re: Stabilising Internode Messaging in 4.0

2019-04-04 Thread Jon Haddad
Given the number of issues that are addressed, I definitely think it's
worth strongly considering merging this in.  I think it might be a
little unrealistic to cut the first alpha after the merge though.
Being realistic, any 20K+ LOC change is going to introduce its own
bugs, and we should be honest with ourselves about that.  It seems
likely the issues the patch addressed would have affected the 4.0
release in some form *anyways* so the question might be do we fix them
now or after someone's cluster burns down because there's no inbound /
outbound message load shedding.

Giving it a quick code review and going through the JIRA comments
(well written, thanks guys) there seem to be some pretty important bug
fixes in here as well as paying off a bit of technical debt.

Jon

On Thu, Apr 4, 2019 at 1:37 PM Pavel Yaskevich  wrote:
>
> Great to see such a significant progress made in the area!
>
> On Thu, Apr 4, 2019 at 1:13 PM Aleksey Yeschenko  wrote:
>
> > I would like to propose CASSANDRA-15066 [1] - an important set of bug fixes
> > and stability improvements to internode messaging code that Benedict, I,
> > and others have been working on for the past couple of months.
> >
> > First, some context.   This work started off as a review of CASSANDRA-14503
> > (Internode connection management is race-prone [2]), CASSANDRA-13630
> > (Support large internode messages with netty) [3], and a pre-4.0
> > confirmatory review of such a major new feature.
> >
> > However, as we dug in, we realized this was insufficient. With more than 50
> > bugs uncovered [4] - dozens of them critical to correctness and/or
> > stability of the system - a substantial rework was necessary to guarantee a
> > solid internode messaging subsystem for the 4.0 release.
> >
> > In addition to addressing all of the uncovered bugs [4] that were unique to
> > trunk + 13630 [3] + 14503 [2], we used this opportunity to correct some
> > long-existing, pre-4.0 bugs and stability issues. For the complete list of
> > notable bug fixes, read the comments to CASSANDRA-15066 [1]. But I’d like
> > to highlight a few.
> >
> > # Lack of message integrity checks
> >
> > It’s known that TCP checksums are too weak [5] and Ethernet CRC cannot be
> > relied upon [6] for integrity. With sufficient scale or time, you will hit
> > bit flips. Sadly, most of the time these go undetected.  Cassandra’s
> > replication model makes this issue much more serious, as the faulty data
> > can infect the cluster.
> >
> > We recognised this problem, and recently introduced a fix for server-client
> > messages, implementing CRCs in CASSANDRA-13304 (Add checksumming to the
> > native protocol) [7].
> >
> > But until CASSANDRA-15066 [1] lands, this is also a critical flaw
> > internode. We have addressed it by ensuring that no matter what, whether
> > you use SSL or not, whether you use internode compression or not, a
> > protocol level CRC is always present, for every message frame. It’s our
> > deep and sincere belief that shipping a new implementation of the messaging
> > protocol without application-level data integrity checks would be
> > unacceptable for a modern database.
> >
>
> I'm all for introducing more correctness checks at all levels especially in
> communication.
> Having dealt with multiple data corruption bugs that could have been easily
> prevented by
> having a checksum, it's great to see that we are moving in this direction.
>
>
> > # Lack of back-pressure and memory usage constraints
> >
> > As it stands today, it’s far too easy for a single slow node to become
> > overwhelmed by messages from its peers.  Conversely, multiple coordinators
> > can be made unstable by the backlog of messages to deliver to just one
> > struggling node.
> >
> > To address this problem, we have introduced strict memory usage constraints
> > that apply TCP-level back-pressure, on both outbound and inbound.  It is
> > now impossible for a node to be swamped on inbound, and on outbound it is
> > made significantly harder to overcommit resources.  It’s a simple, reliable
> > mechanism that drastically improves cluster stability under load, and
> > especially overload.
> >
> > Cassandra is a mature system, and introducing an entirely new messaging
> > implementation without resolving this fundamental stability issue is
> > difficult to justify in our view.
> >
>
> I'd say that this is required to be able to ship 4.0 as a release focused
> on stability.
> I personally have been waiting for this to happen for years. Significant
> step forward in our QoS story.
>
>
> >
> > # State of the patch, feature freeze and 4.0 timeline concerns
> >
> > The patch is essentially complete, with much improved unit tests all
> > passing, dtests green, and extensive fuzz testing underway - with initial
> > results all positive.  We intend to further improve in-code documentation
> > and test coverage in the next week or two, and do some minor additional
> > code review, but we believe it will be basically 

TLP tools for stress testing and building test clusters in AWS

2019-04-12 Thread Jon Haddad
I don't want to derail the discussion about Stabilizing Internode
Messaging, so I'm starting this as a separate thread.  There was a
comment that Josh made [1] about doing performance testing with real
clusters as well as a lot of microbenchmarks, and I'm 100% in support
of this.  We've been working on some tooling at TLP for the last
several months to make this a lot easier.  One of the goals has been
to help improve the 4.0 testing process.

The first tool we have is tlp-stress [2].  It's designed with a "get
started in 5 minutes" mindset.  My goal was to ship a stress tool that
ships with real workloads out of the box that can be easily tweaked,
similar to how fio allows you to design a disk workload and tweak it
with paramaters.  Included are stress workloads that stress LWTs (two
different types), materialized views, counters, time series, and
key-value workloads.  Each workload can be modified easily to change
compaction strategies, concurrent operations, number of partitions.
We can run workloads for a set number of iterations or a custom
duration.  We've used this *extensively* at TLP to help our customers
and most of our blog posts that discuss performance use it as well.
It exports data to both a CSV format and auto sets up prometheus for
metrics collection / aggregation.  As an example, we were able to
determine that the compression length set on the paxos tables imposes
a significant overhead when using the Locking LWT workload, which
simulates locking and unlocking of rows.  See CASSANDRA-15080 for
details.

We have documentation [3] on the TLP website.

The second tool we've been working on is tlp-cluster [4].  This tool
is designed to help provision AWS instances for the purposes of
testing.  To be clear, I don't expect, or want, this tool to be used
for production environments.  It's designed to assist with the
Cassandra build process by generating deb packages or re-using the
ones that have already been uploaded.  Here's a short list of the
things you'll care about:

1. Create instances in AWS for Cassandra using any instance size and
number of nodes.  Also create tlp-stress instances and a box for
monitoring
2. Use any available build of Cassandra, with a quick option to change
YAML config.  For example: tlp-stress use 3.11.4 -c
concurrent_writes:256
3. Do custom builds just by pointing to a local Cassandra git repo.
They can be used the same way as #2.
4. tlp-stress is automatically installed on the stress box.
5. Everything's installed with pure bash.  I considered something more
complex, but since this is for development only, it turns out the
simplest tool possible works well and it means it's easily
configurable.  Just drop in your own bash script starting with a
number in a XX_script_name.sh format and it gets run.
6. The monitoring box is running Prometheus.  It auto scrapes
Cassandra using the Instaclustr metrics library.
7. Grafana is also installed automatically.  There's a couple sample
graphs there now.  We plan on having better default graphs soon.

For the moment it installs java 8 only but that should be easily
fixable to use java 11 to test ZGC (it's on my radar).

Documentation for tlp-cluster is here [5].

There's still some things to work out in the tool, and we've been
working hard to smooth out the rough edges.  I still haven't announced
anything WRT tlp-cluster on the TLP blog, because I don't think it's
quite ready for public consumption, but I think the folks on this list
are smart enough to see the value in it even if it has a few warts
still.

I don't consider myself familiar enough with the networking patch to
give it a full review, but I am qualified to build tools to help test
it and go through the testing process myself.  From what I can tell
the patch is moving the codebase in a positive direction and I'd like
to help build confidence in it so we can get it merged in.

We'll continue to build out and improve the tooling with the goal of
making it easier for people to jump into the QA side of things.

Jon

[1] 
https://lists.apache.org/thread.html/742009c8a77999f4b62062509f087b670275f827d0c1895bf839eece@%3Cdev.cassandra.apache.org%3E
[2] https://github.com/thelastpickle/tlp-stress
[3] http://thelastpickle.com/tlp-stress/
[4] https://github.com/thelastpickle/tlp-cluster
[5] http://thelastpickle.com/tlp-cluster/

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



Re: TLP tools for stress testing and building test clusters in AWS

2019-04-15 Thread Jon Haddad
Hey all,

I've set up a Zoom call for 9AM Pacific time.  Everyone's welcome to join.

https://zoom.us/j/189920888

Looking forward to a good discussion on how we can all pitch in on
getting 4.0 out the door.

Jon

On Sat, Apr 13, 2019 at 9:14 AM Jonathan Koppenhofer
 wrote:
>
> Wednesday would work for me.
>
> We use and (slightly) contribute to tlp tools. We are platform testing and
> beginning 4.0 testing ourselves, so an in person overview would be great!
>
> On Sat, Apr 13, 2019, 8:48 AM Aleksey Yeshchenko 
> wrote:
>
> > Wednesday and Thursday, either, at 9 AM pacific WFM.
> >
> > > On 13 Apr 2019, at 13:31, Stefan Miklosovic <
> > stefan.mikloso...@instaclustr.com> wrote:
> > >
> > > Hi Jon,
> > >
> > > I would like be on that call too but I am off on Thursday.
> > >
> > > I am from Australia so 5pm London time is ours 2am next day so your
> > > Wednesday morning is my Thursday night. Wednesday early morning so
> > > your Tuesday morning and London's afternoon would be the best.
> > >
> > > Recording the thing would be definitely helpful too.
> > >
> > > On Sat, 13 Apr 2019 at 07:45, Jon Haddad  wrote:
> > >>
> > >> I'd be more than happy to hop on a call next week to give you both
> > >> (and anyone else interested) a tour of our dev tools.  Maybe something
> > >> early morning on my end, which should be your evening, could work?
> > >>
> > >> I can set up a Zoom conference to get everyone acquainted.  We can
> > >> record and post it for any who can't make it.
> > >>
> > >> I'm thinking Tuesday, Wednesday, or Thursday morning, 9AM Pacific (5pm
> > >> London)?  If anyone's interested please reply with what dates work.
> > >> I'll be sure to post the details back here with the zoom link in case
> > >> anyone wants to join that didn't get a chance to reply, as well as a
> > >> link to the recorded call.
> > >>
> > >> Jon
> > >>
> > >> On Fri, Apr 12, 2019 at 10:41 AM Benedict Elliott Smith
> > >>  wrote:
> > >>>
> > >>> +1
> > >>>
> > >>> I’m also just as excited to see some standardised workloads and test
> > bed.  At the moment we’re benefiting from some large contributors doing
> > their own proprietary performance testing, which is super valuable and
> > something we’ve lacked before.  But I’m also keen to see some more
> > representative workloads that are reproducible by anybody in the community
> > take shape.
> > >>>
> > >>>
> > >>>> On 12 Apr 2019, at 18:09, Aleksey Yeshchenko
> >  wrote:
> > >>>>
> > >>>> Hey Jon,
> > >>>>
> > >>>> This sounds exciting and pretty useful, thanks.
> > >>>>
> > >>>> Looking forward to using tlp-stress for validating 15066 performance.
> > >>>>
> > >>>> We should touch base some time next week to pick a comprehensive set
> > of workloads and versions, perhaps?
> > >>>>
> > >>>>
> > >>>>> On 12 Apr 2019, at 16:34, Jon Haddad  wrote:
> > >>>>>
> > >>>>> I don't want to derail the discussion about Stabilizing Internode
> > >>>>> Messaging, so I'm starting this as a separate thread.  There was a
> > >>>>> comment that Josh made [1] about doing performance testing with real
> > >>>>> clusters as well as a lot of microbenchmarks, and I'm 100% in support
> > >>>>> of this.  We've been working on some tooling at TLP for the last
> > >>>>> several months to make this a lot easier.  One of the goals has been
> > >>>>> to help improve the 4.0 testing process.
> > >>>>>
> > >>>>> The first tool we have is tlp-stress [2].  It's designed with a "get
> > >>>>> started in 5 minutes" mindset.  My goal was to ship a stress tool
> > that
> > >>>>> ships with real workloads out of the box that can be easily tweaked,
> > >>>>> similar to how fio allows you to design a disk workload and tweak it
> > >>>>> with paramaters.  Included are stress workloads that stress LWTs (two
> > >>>>> different types), materialized views, counters, time series, and
> > >>>>> key-value workloads.  Each workload can be modified easily 

Re: TLP tools for stress testing and building test clusters in AWS

2019-04-16 Thread Jon Haddad
Yes, sorry about that. Wednesday morning 9am PT

On Tue, Apr 16, 2019 at 3:26 AM Benedict Elliott Smith 
wrote:

> Just to confirm, this is on Wednesday?
>
> > On 15 Apr 2019, at 22:38, Jon Haddad  wrote:
> >
> > Hey all,
> >
> > I've set up a Zoom call for 9AM Pacific time.  Everyone's welcome to
> join.
> >
> > https://zoom.us/j/189920888
> >
> > Looking forward to a good discussion on how we can all pitch in on
> > getting 4.0 out the door.
> >
> > Jon
> >
> > On Sat, Apr 13, 2019 at 9:14 AM Jonathan Koppenhofer
> >  wrote:
> >>
> >> Wednesday would work for me.
> >>
> >> We use and (slightly) contribute to tlp tools. We are platform testing
> and
> >> beginning 4.0 testing ourselves, so an in person overview would be
> great!
> >>
> >> On Sat, Apr 13, 2019, 8:48 AM Aleksey Yeshchenko
> 
> >> wrote:
> >>
> >>> Wednesday and Thursday, either, at 9 AM pacific WFM.
> >>>
> >>>> On 13 Apr 2019, at 13:31, Stefan Miklosovic <
> >>> stefan.mikloso...@instaclustr.com> wrote:
> >>>>
> >>>> Hi Jon,
> >>>>
> >>>> I would like be on that call too but I am off on Thursday.
> >>>>
> >>>> I am from Australia so 5pm London time is ours 2am next day so your
> >>>> Wednesday morning is my Thursday night. Wednesday early morning so
> >>>> your Tuesday morning and London's afternoon would be the best.
> >>>>
> >>>> Recording the thing would be definitely helpful too.
> >>>>
> >>>> On Sat, 13 Apr 2019 at 07:45, Jon Haddad  wrote:
> >>>>>
> >>>>> I'd be more than happy to hop on a call next week to give you both
> >>>>> (and anyone else interested) a tour of our dev tools.  Maybe
> something
> >>>>> early morning on my end, which should be your evening, could work?
> >>>>>
> >>>>> I can set up a Zoom conference to get everyone acquainted.  We can
> >>>>> record and post it for any who can't make it.
> >>>>>
> >>>>> I'm thinking Tuesday, Wednesday, or Thursday morning, 9AM Pacific
> (5pm
> >>>>> London)?  If anyone's interested please reply with what dates work.
> >>>>> I'll be sure to post the details back here with the zoom link in case
> >>>>> anyone wants to join that didn't get a chance to reply, as well as a
> >>>>> link to the recorded call.
> >>>>>
> >>>>> Jon
> >>>>>
> >>>>> On Fri, Apr 12, 2019 at 10:41 AM Benedict Elliott Smith
> >>>>>  wrote:
> >>>>>>
> >>>>>> +1
> >>>>>>
> >>>>>> I’m also just as excited to see some standardised workloads and test
> >>> bed.  At the moment we’re benefiting from some large contributors doing
> >>> their own proprietary performance testing, which is super valuable and
> >>> something we’ve lacked before.  But I’m also keen to see some more
> >>> representative workloads that are reproducible by anybody in the
> community
> >>> take shape.
> >>>>>>
> >>>>>>
> >>>>>>> On 12 Apr 2019, at 18:09, Aleksey Yeshchenko
> >>>  wrote:
> >>>>>>>
> >>>>>>> Hey Jon,
> >>>>>>>
> >>>>>>> This sounds exciting and pretty useful, thanks.
> >>>>>>>
> >>>>>>> Looking forward to using tlp-stress for validating 15066
> performance.
> >>>>>>>
> >>>>>>> We should touch base some time next week to pick a comprehensive
> set
> >>> of workloads and versions, perhaps?
> >>>>>>>
> >>>>>>>
> >>>>>>>> On 12 Apr 2019, at 16:34, Jon Haddad  wrote:
> >>>>>>>>
> >>>>>>>> I don't want to derail the discussion about Stabilizing Internode
> >>>>>>>> Messaging, so I'm starting this as a separate thread.  There was a
> >>>>>>>> comment that Josh made [1] about doing performance testing with
> real
> >>>>>>>> clusters as well as a lot of microbenchmarks, and I'm 100% in
> support
> >>>>>>>> of this.  We've be

Re: TLP tools for stress testing and building test clusters in AWS

2019-04-16 Thread Jon Haddad
The one I sent out is open, no separate invite required.

On Tue, Apr 16, 2019 at 3:47 PM Dinesh Joshi  wrote:
>
> I'm slightly confused. The zoom meeting mentioned in this thread is only open 
> to who have registered interest here? If so, can someone please add me?
>
> Dinesh
>
> > On Apr 16, 2019, at 3:29 PM, Anthony Grasso  
> > wrote:
> >
> > Hi Stefan,
> >
> > Thanks for sending the invite out!
> >
> > Just wondering what do you think of the idea of having a Zoom meeting that
> > anyone can join? This way anyone else interested can join us as well. I can
> > set that up if you like?
> >
> > Cheers,
> > Anthony
> >
> > On Tue, 16 Apr 2019 at 21:24, Stefan Miklosovic <
> > stefan.mikloso...@instaclustr.com> wrote:
> >
> >> Hi Anthony,
> >>
> >> sounds good. I ve sent you Hangouts meeting invitation privately.
> >>
> >> Regards
> >>
> >> On Tue, 16 Apr 2019 at 14:53, Anthony Grasso 
> >> wrote:
> >>>
> >>> Hi Stefan,
> >>>
> >>> I have been working with Jon on developing the tool set. I can do a Zoom
> >>> call tomorrow (Wednesday) at 11am AEST if that works for you? We can go
> >>> through all the same information that Jon is going to go through in his
> >>> call. Note that I am in the same timezone as you, so if tomorrow morning
> >> is
> >>> no good we can always do the afternoon.
> >>>
> >>> Cheers,
> >>> Anthony
> >>>
> >>>
> >>> On Sat, 13 Apr 2019 at 22:38, Stefan Miklosovic <
> >>> stefan.mikloso...@instaclustr.com> wrote:
> >>>
> >>>> Hi Jon,
> >>>>
> >>>> I would like be on that call too but I am off on Thursday.
> >>>>
> >>>> I am from Australia so 5pm London time is ours 2am next day so your
> >>>> Wednesday morning is my Thursday night. Wednesday early morning so
> >>>> your Tuesday morning and London's afternoon would be the best.
> >>>>
> >>>> Recording the thing would be definitely helpful too.
> >>>>
> >>>> On Sat, 13 Apr 2019 at 07:45, Jon Haddad  wrote:
> >>>>>
> >>>>> I'd be more than happy to hop on a call next week to give you both
> >>>>> (and anyone else interested) a tour of our dev tools.  Maybe
> >> something
> >>>>> early morning on my end, which should be your evening, could work?
> >>>>>
> >>>>> I can set up a Zoom conference to get everyone acquainted.  We can
> >>>>> record and post it for any who can't make it.
> >>>>>
> >>>>> I'm thinking Tuesday, Wednesday, or Thursday morning, 9AM Pacific
> >> (5pm
> >>>>> London)?  If anyone's interested please reply with what dates work.
> >>>>> I'll be sure to post the details back here with the zoom link in case
> >>>>> anyone wants to join that didn't get a chance to reply, as well as a
> >>>>> link to the recorded call.
> >>>>>
> >>>>> Jon
> >>>>>
> >>>>> On Fri, Apr 12, 2019 at 10:41 AM Benedict Elliott Smith
> >>>>>  wrote:
> >>>>>>
> >>>>>> +1
> >>>>>>
> >>>>>> I’m also just as excited to see some standardised workloads and
> >> test
> >>>> bed.  At the moment we’re benefiting from some large contributors doing
> >>>> their own proprietary performance testing, which is super valuable and
> >>>> something we’ve lacked before.  But I’m also keen to see some more
> >>>> representative workloads that are reproducible by anybody in the
> >> community
> >>>> take shape.
> >>>>>>
> >>>>>>
> >>>>>>> On 12 Apr 2019, at 18:09, Aleksey Yeshchenko
> >>>>  wrote:
> >>>>>>>
> >>>>>>> Hey Jon,
> >>>>>>>
> >>>>>>> This sounds exciting and pretty useful, thanks.
> >>>>>>>
> >>>>>>> Looking forward to using tlp-stress for validating 15066
> >> performance.
> >>>>>>>
> >>>>>>> We should touch base some time next week to pick a comprehensive
> >> set
> >>>> of workloads 

Re: TLP tools for stress testing and building test clusters in AWS

2019-04-12 Thread Jon Haddad
I'd be more than happy to hop on a call next week to give you both
(and anyone else interested) a tour of our dev tools.  Maybe something
early morning on my end, which should be your evening, could work?

I can set up a Zoom conference to get everyone acquainted.  We can
record and post it for any who can't make it.

I'm thinking Tuesday, Wednesday, or Thursday morning, 9AM Pacific (5pm
London)?  If anyone's interested please reply with what dates work.
I'll be sure to post the details back here with the zoom link in case
anyone wants to join that didn't get a chance to reply, as well as a
link to the recorded call.

Jon

On Fri, Apr 12, 2019 at 10:41 AM Benedict Elliott Smith
 wrote:
>
> +1
>
> I’m also just as excited to see some standardised workloads and test bed.  At 
> the moment we’re benefiting from some large contributors doing their own 
> proprietary performance testing, which is super valuable and something we’ve 
> lacked before.  But I’m also keen to see some more representative workloads 
> that are reproducible by anybody in the community take shape.
>
>
> > On 12 Apr 2019, at 18:09, Aleksey Yeshchenko  
> > wrote:
> >
> > Hey Jon,
> >
> > This sounds exciting and pretty useful, thanks.
> >
> > Looking forward to using tlp-stress for validating 15066 performance.
> >
> > We should touch base some time next week to pick a comprehensive set of 
> > workloads and versions, perhaps?
> >
> >
> >> On 12 Apr 2019, at 16:34, Jon Haddad  wrote:
> >>
> >> I don't want to derail the discussion about Stabilizing Internode
> >> Messaging, so I'm starting this as a separate thread.  There was a
> >> comment that Josh made [1] about doing performance testing with real
> >> clusters as well as a lot of microbenchmarks, and I'm 100% in support
> >> of this.  We've been working on some tooling at TLP for the last
> >> several months to make this a lot easier.  One of the goals has been
> >> to help improve the 4.0 testing process.
> >>
> >> The first tool we have is tlp-stress [2].  It's designed with a "get
> >> started in 5 minutes" mindset.  My goal was to ship a stress tool that
> >> ships with real workloads out of the box that can be easily tweaked,
> >> similar to how fio allows you to design a disk workload and tweak it
> >> with paramaters.  Included are stress workloads that stress LWTs (two
> >> different types), materialized views, counters, time series, and
> >> key-value workloads.  Each workload can be modified easily to change
> >> compaction strategies, concurrent operations, number of partitions.
> >> We can run workloads for a set number of iterations or a custom
> >> duration.  We've used this *extensively* at TLP to help our customers
> >> and most of our blog posts that discuss performance use it as well.
> >> It exports data to both a CSV format and auto sets up prometheus for
> >> metrics collection / aggregation.  As an example, we were able to
> >> determine that the compression length set on the paxos tables imposes
> >> a significant overhead when using the Locking LWT workload, which
> >> simulates locking and unlocking of rows.  See CASSANDRA-15080 for
> >> details.
> >>
> >> We have documentation [3] on the TLP website.
> >>
> >> The second tool we've been working on is tlp-cluster [4].  This tool
> >> is designed to help provision AWS instances for the purposes of
> >> testing.  To be clear, I don't expect, or want, this tool to be used
> >> for production environments.  It's designed to assist with the
> >> Cassandra build process by generating deb packages or re-using the
> >> ones that have already been uploaded.  Here's a short list of the
> >> things you'll care about:
> >>
> >> 1. Create instances in AWS for Cassandra using any instance size and
> >> number of nodes.  Also create tlp-stress instances and a box for
> >> monitoring
> >> 2. Use any available build of Cassandra, with a quick option to change
> >> YAML config.  For example: tlp-stress use 3.11.4 -c
> >> concurrent_writes:256
> >> 3. Do custom builds just by pointing to a local Cassandra git repo.
> >> They can be used the same way as #2.
> >> 4. tlp-stress is automatically installed on the stress box.
> >> 5. Everything's installed with pure bash.  I considered something more
> >> complex, but since this is for development only, it turns out the
> >> simplest tool possible works well and it means it's easily
> >> configurable.  Just drop in 

Re: [VOTE] remove the old wiki

2019-06-04 Thread Jon Haddad
I think we could port that page over and clean it up before deleting the
wiki.

On Tue, Jun 4, 2019 at 12:30 PM Joshua McKenzie 
wrote:

> Before I vote, do we have something analogous to this:
> https://wiki.apache.org/cassandra/ArchitectureInternals
> In the new wiki / docs? Looks like it's a stub:
> https://cassandra.apache.org/doc/latest/architecture/overview.html
>
> Having an architectural overview landing page would be critical before
> sunsetting the old one IMO. And yes, that ArchitectureInternals article
> is... very old. But very old > nothing in terms of establishing a framework
> in which to think about something. Maybe.
>
> On Tue, Jun 4, 2019 at 2:47 PM Jon Haddad  wrote:
>
> > I assume everyone here knows the old wiki hasn't been maintained, and is
> > years out of date.  I propose we sunset it completely and delete it
> forever
> > from the world.
> >
> > I'm happy to file the INFRA ticket to delete it, I'd just like to give
> > everyone the opportunity to speak up in case there's something I'm not
> > aware of.
> >
> > In favor of removing the wiki?  That's a +1.
> > -1 if you think we're better off migrating the entire thing to cwiki.
> >
> > If you only need couple pages, feel free to move the content to the
> > documentation.  I'm sure we can also export the wiki in its entirety and
> > put it somewhere offline, if there's a concern about maybe needing some
> of
> > the content at some point in the future.
> >
> > I think 72 hours is enough time to leave a vote open on this topic.
> >
> > Jon
> >
>


[VOTE] remove the old wiki

2019-06-04 Thread Jon Haddad
I assume everyone here knows the old wiki hasn't been maintained, and is
years out of date.  I propose we sunset it completely and delete it forever
from the world.

I'm happy to file the INFRA ticket to delete it, I'd just like to give
everyone the opportunity to speak up in case there's something I'm not
aware of.

In favor of removing the wiki?  That's a +1.
-1 if you think we're better off migrating the entire thing to cwiki.

If you only need couple pages, feel free to move the content to the
documentation.  I'm sure we can also export the wiki in its entirety and
put it somewhere offline, if there's a concern about maybe needing some of
the content at some point in the future.

I think 72 hours is enough time to leave a vote open on this topic.

Jon


Re: [DISCUSS] Moving chats to ASF's Slack instance

2019-05-28 Thread Jon Haddad
+1

On Tue, May 28, 2019, 2:54 PM Joshua McKenzie  wrote:

> +1 to switching over. One less comms client + history + searchability is
> enough to get my vote easy.
>
> On Tue, May 28, 2019 at 5:52 PM Jonathan Ellis  wrote:
>
> > I agree.  This lowers the barrier to entry for new participants.  Slack
> is
> > probably two orders of magnitude more commonly used now than irc for sw
> > devs and three for everyone else.  And then you have the quality-of-life
> > features that you get out of the box with Slack and only with difficulty
> in
> > irc (history, search, file uploads...)
> >
> > On Tue, May 28, 2019 at 4:29 PM Nate McCall  wrote:
> >
> > > Hi Folks,
> > > While working on ApacheCon last week, I had to get setup on ASF's slack
> > > workspace. After poking around a bit, on a whim I created #cassandra
> and
> > > #cassandra-dev. I then invited a couple of people to come signup and
> test
> > > it out - primarily to make sure that the process was seamless for
> non-ASF
> > > account holders as well as committers, etc (it was).
> > >
> > > If you want to jump in, you can signup here:
> > > https://s.apache.org/slack-invite
> > >
> > > That said, I think it's time we transition from IRC to Slack. Now, I
> like
> > > CLI friendly, straight forward tools like IRC as much as anyone, but
> it's
> > > been more than once recently where a user I've talked to has said one
> of
> > > two things regarding our IRC channels: "What's IRC?" or "Yeah, I don't
> > > really do that anymore."
> > >
> > > In short, I think it's time to migrate. I think this will really just
> > > consist of some communications to our lists and updating the site
> > (anything
> > > I'm missing?). The archives of IRC should just kind of persist for
> > > posterity sake without any additional effort or maintenance. The
> > > ASF-requirements are all configured already on the Slack workspace, so
> I
> > > think we are good there.
> > >
> > > Thanks,
> > > -Nate
> > >
> >
> >
> > --
> > Jonathan Ellis
> > co-founder, http://www.datastax.com
> > @spyced
> >
>


Re: "4.0: TBD" -> "4.0: Est. Q4 2019"?

2019-05-28 Thread Jon Haddad
Sept is a pretty long ways off.  I think the ideal case is we can announce
4.0 release at the summit.  I'm not putting this as a "do or die" date, and
I don't think we need to announce it or make promises.  Sticking with "when
it's ready" is the right approach, but we need a target, and this is imo a
good one.

This date also gives us a pretty good runway.  We could cut our first
alphas in mid June / early July, betas in August and release in Sept.
 There's a ton of work going into testing 4.0 already.
Landing CASSANDRA-15066 will put us in a pretty good spot.  We've developed
tooling at TLP that will make it a lot easier to spin up dev clusters in
AWS as well as stress test them.  I've written about this a few times in
the past, and I'll have a few blog posts coming up that will help show this
in more details.

There's some other quality of life things we should try to hammer out
before then.  Updating our default JVM settings would be nice, for
example.  Improving documentation (the data modeling section in
particular), fixing the dynamic snitch issues [1], and some improvements to
virtual tables like exposing the sstable metadata [2], and exposing table
statistics [3] come to mind.  The dynamic snitch improvement will help
performance in a big way, and the virtual tables will go a long way to
helping with quality of life.  I showed a few folks virtual tables at the
Accelerate conference last week and the missing table statistics was a big
shock.  If we can get them in, it'll be a big help to operators.

[1] https://issues.apache.org/jira/browse/CASSANDRA-14459
[2] https://issues.apache.org/jira/browse/CASSANDRA-14630
[3] https://issues.apache.org/jira/browse/CASSANDRA-14572




On Mon, May 27, 2019 at 2:36 PM Nate McCall  wrote:

> Hi Sumanth,
> Thank you so much for taking the time to put this together.
>
> Cheers,
> -Nate
>
> On Tue, May 28, 2019 at 3:27 AM Sumanth Pasupuleti <
> sumanth.pasupuleti...@gmail.com> wrote:
>
> > I have taken an initial stab at documenting release types and exit
> criteria
> > in a google doc, to get us started, and to collaborate on.
> >
> >
> https://docs.google.com/document/d/1bS6sr-HSrHFjZb0welife6Qx7u3ZDgRiAoENMLYlfz8/edit?usp=sharing
> >
> > Thanks,
> > Sumanth
> >
> > On Thu, May 23, 2019 at 12:04 PM Dinesh Joshi  wrote:
> >
> > > Sankalp,
> > >
> > > Great point. This is the page created for testing.
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/4.0+Quality%3A+Components+and+Test+Plans
> > >
> > > I think we need to define the various release types and the exit
> criteria
> > > for each type of release. Anybody want to take a stab at this or start
> a
> > > thread to discuss it?
> > >
> > > Thanks,
> > >
> > > Dinesh
> > >
> > >
> > > > On May 23, 2019, at 11:57 AM, sankalp kohli 
> > > wrote:
> > > >
> > > > Hi,
> > > >Is there a page where it is written what is expected from an
> alpha,
> > > > beta, rc and a 4.0 release?
> > > > Also how are we coming up with Q4 2019 timeline. Is this for alpha,
> > beta,
> > > > rc or 4.0 release?
> > > >
> > > > Thanks,
> > > > Sankalp
> > > >
> > > > On Thu, May 23, 2019 at 11:27 AM Attila Wind  >
> > > wrote:
> > > >
> > > >> +1+1+1 I read a blog post was talking about last sept(?) to freeze
> > > >> features and start extensive testing. Maybe its really time to hit
> it!
> > > :-)
> > > >>
> > > >> Attila Wind
> > > >>
> > > >> http://www.linkedin.com/in/attilaw
> > > >> Mobile: +36 31 7811355
> > > >>
> > > >>
> > > >> On 2019. 05. 23. 19:30, ajs6f wrote:
> > > >>> +1 in the fullest degree. A date that needs to be changed is still
> > > >> enormously more attractive than no date at all.
> > > >>>
> > > >>> Adam Soroka
> > > >>>
> > >  On May 23, 2019, at 12:01 PM, Sumanth Pasupuleti <
> > > >> spasupul...@netflix.com.INVALID> wrote:
> > > 
> > >  Having at least a ballpark target on the website will definitely
> > help.
> > > >> +1
> > >  on setting it to Q4 2019 for now.
> > > 
> > >  On Thu, May 23, 2019 at 8:52 AM Dinesh Joshi 
> > > wrote:
> > > 
> > > > +1 on setting a date.
> > > >
> > > > Dinesh
> > > >
> > > >> On May 23, 2019, at 11:07 AM, Michael Shuler <
> > > mich...@pbandjelly.org>
> > > > wrote:
> > > >> We've had 4.0 listed as TBD release date for a very long time.
> > > >>
> > > >> Yesterday, Alexander Dejanovski got a "when's 4.0 going to
> > release?"
> > > > question after his repair talk and he suggested possibly Q4 2019.
> > > This
> > > > morning Nate McCall hinted at possibly being close by ApacheCon
> Las
> > > >> Vegas
> > > > in September. These got me thinking..
> > > >> Think we can we shoot for having a 4.0 alpha/beta/rc ready to
> > > > announce/release at ApacheCon? At that time, we'll have been
> frozen
> > > >> for 1
> > > > year, and I think we can. We'll GA release when it's ready, but I
> > > >> think Q4
> > > > could be an realistic target.
> > > >> With 

Re: "4.0: TBD" -> "4.0: Est. Q4 2019"?

2019-05-28 Thread Jon Haddad
My thinking is I'd like to be able to recommend 4.0.0 as a production ready
database for business critical cases of TLP customers.  If it's not ready
for prod, there's no way I'd vote to release it.  The TLP tooling I've
mentioned was developed over the last 6 months with the specific goal of
being able to test custom builds for the 4.0 release, and I've run several
clusters using it already.  The stress tool we built just got a --ttl
option so I should be able to start some longer running clusters that TTL
data out, so we can see the impact of running a cluster under heavy load
for several weeks.



On Tue, May 28, 2019 at 9:57 AM sankalp kohli 
wrote:

> Hi Jon,
>When you say 4.0 release, how do u match it with 3.0 minor
> releases. The unofficial rule is to not upgrade to prod till .10 is cut.
> Also due to heavy investment in testing, I dont think it will take as long
> as 3.0 but want to know what is your thinking with this.
>
> Thanks,
> Sankalp
>
> On Tue, May 28, 2019 at 9:40 AM Jon Haddad  wrote:
>
> > Sept is a pretty long ways off.  I think the ideal case is we can
> announce
> > 4.0 release at the summit.  I'm not putting this as a "do or die" date,
> and
> > I don't think we need to announce it or make promises.  Sticking with
> "when
> > it's ready" is the right approach, but we need a target, and this is imo
> a
> > good one.
> >
> > This date also gives us a pretty good runway.  We could cut our first
> > alphas in mid June / early July, betas in August and release in Sept.
> >  There's a ton of work going into testing 4.0 already.
> > Landing CASSANDRA-15066 will put us in a pretty good spot.  We've
> developed
> > tooling at TLP that will make it a lot easier to spin up dev clusters in
> > AWS as well as stress test them.  I've written about this a few times in
> > the past, and I'll have a few blog posts coming up that will help show
> this
> > in more details.
> >
> > There's some other quality of life things we should try to hammer out
> > before then.  Updating our default JVM settings would be nice, for
> > example.  Improving documentation (the data modeling section in
> > particular), fixing the dynamic snitch issues [1], and some improvements
> to
> > virtual tables like exposing the sstable metadata [2], and exposing table
> > statistics [3] come to mind.  The dynamic snitch improvement will help
> > performance in a big way, and the virtual tables will go a long way to
> > helping with quality of life.  I showed a few folks virtual tables at the
> > Accelerate conference last week and the missing table statistics was a
> big
> > shock.  If we can get them in, it'll be a big help to operators.
> >
> > [1] https://issues.apache.org/jira/browse/CASSANDRA-14459
> > [2] https://issues.apache.org/jira/browse/CASSANDRA-14630
> > [3] https://issues.apache.org/jira/browse/CASSANDRA-14572
> >
> >
> >
> >
> > On Mon, May 27, 2019 at 2:36 PM Nate McCall  wrote:
> >
> > > Hi Sumanth,
> > > Thank you so much for taking the time to put this together.
> > >
> > > Cheers,
> > > -Nate
> > >
> > > On Tue, May 28, 2019 at 3:27 AM Sumanth Pasupuleti <
> > > sumanth.pasupuleti...@gmail.com> wrote:
> > >
> > > > I have taken an initial stab at documenting release types and exit
> > > criteria
> > > > in a google doc, to get us started, and to collaborate on.
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1bS6sr-HSrHFjZb0welife6Qx7u3ZDgRiAoENMLYlfz8/edit?usp=sharing
> > > >
> > > > Thanks,
> > > > Sumanth
> > > >
> > > > On Thu, May 23, 2019 at 12:04 PM Dinesh Joshi 
> > wrote:
> > > >
> > > > > Sankalp,
> > > > >
> > > > > Great point. This is the page created for testing.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/4.0+Quality%3A+Components+and+Test+Plans
> > > > >
> > > > > I think we need to define the various release types and the exit
> > > criteria
> > > > > for each type of release. Anybody want to take a stab at this or
> > start
> > > a
> > > > > thread to discuss it?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Dinesh
> > > > >
> > > > >
> > > > > > On May 23, 2019, at

Re: Jira Suggestion

2019-05-14 Thread Jon Haddad
Great idea. +1

On Tue, May 14, 2019, 12:10 PM Benedict Elliott Smith 
wrote:

> It will be possible to insert n/a.  It will simply be a text field - Jira
> doesn’t know anything about the concept of a SHA, and I don’t intend to
> introduce validation logic.  It’s just a logical and consistent place for
> it to live, and a strong reminder to include it.  My intention is for it to
> be a text field supporting Jira markup, like Test and Doc Plan, so that we
> can insert cleanly formatted links to GitHub just like we do now in
> comments.
>
>
>
> > On 14 May 2019, at 20:04, Dinesh Joshi  wrote:
> >
> > I am +0.5 on this. I think it is a good idea. I want to ensure that we
> capture use-cases such as Tasks that may not have a git commit associated
> with them. There might be tickets that may have multiple git commits across
> repos. SVN commits may also need to be handled.
> >
> > Dinesh
> >
> >> On May 14, 2019, at 11:34 AM, Jeff Jirsa  wrote:
> >>
> >> Please
> >>
> >> --
> >> Jeff Jirsa
> >>
> >>
> >>> On May 14, 2019, at 7:53 AM, Benedict Elliott Smith <
> bened...@apache.org> wrote:
> >>>
> >>> How would people feel about introducing a field for the (git) commit
> SHA, to be required on (Jira) commit?
> >>>
> >>> The norm is that we comment the SHA, but given this is the norm
> perhaps we should codify it instead, while we have the chance?  It would
> also make it easier to find.
> >>> -
> >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>>
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>


Re: TLP tools for stress testing and building test clusters in AWS

2019-04-17 Thread Jon Haddad
Hey folks.  I've opened the 9am zoom session.

You can join here: https://zoom.us/j/189920888


On Tue, Apr 16, 2019 at 10:49 PM Stefan Miklosovic
 wrote:
>
> Thanks Anthony for going that proverbial extra mile to cover people in
> different time zones too.
>
> I believe other people will find your talk as helpful as we did.
>
> Regards
>
> On Wed, 17 Apr 2019 at 10:08, Anthony Grasso  wrote:
> >
> > Hi Stefan and devs,
> >
> > I have set up a zoom link for the TLP tool set intro that will be on in an
> > hours time (17 April 2019 @ 11:00AM AEST): https://zoom.us/j/272648772
> >
> > This link is open so if anyone else wishes to join they are welcome to do
> > so. I will be covering the same topics Jon is covering in his meeting
> > tomorrow.
> >
> > Regards,
> > Anthony
> >
> >
> > On Wed, 17 Apr 2019 at 08:29, Anthony Grasso 
> > wrote:
> >
> > > Hi Stefan,
> > >
> > > Thanks for sending the invite out!
> > >
> > > Just wondering what do you think of the idea of having a Zoom meeting that
> > > anyone can join? This way anyone else interested can join us as well. I 
> > > can
> > > set that up if you like?
> > >
> > > Cheers,
> > > Anthony
> > >
> > > On Tue, 16 Apr 2019 at 21:24, Stefan Miklosovic <
> > > stefan.mikloso...@instaclustr.com> wrote:
> > >
> > >> Hi Anthony,
> > >>
> > >> sounds good. I ve sent you Hangouts meeting invitation privately.
> > >>
> > >> Regards
> > >>
> > >> On Tue, 16 Apr 2019 at 14:53, Anthony Grasso 
> > >> wrote:
> > >> >
> > >> > Hi Stefan,
> > >> >
> > >> > I have been working with Jon on developing the tool set. I can do a 
> > >> > Zoom
> > >> > call tomorrow (Wednesday) at 11am AEST if that works for you? We can go
> > >> > through all the same information that Jon is going to go through in his
> > >> > call. Note that I am in the same timezone as you, so if tomorrow
> > >> morning is
> > >> > no good we can always do the afternoon.
> > >> >
> > >> > Cheers,
> > >> > Anthony
> > >> >
> > >> >
> > >> > On Sat, 13 Apr 2019 at 22:38, Stefan Miklosovic <
> > >> > stefan.mikloso...@instaclustr.com> wrote:
> > >> >
> > >> > > Hi Jon,
> > >> > >
> > >> > > I would like be on that call too but I am off on Thursday.
> > >> > >
> > >> > > I am from Australia so 5pm London time is ours 2am next day so your
> > >> > > Wednesday morning is my Thursday night. Wednesday early morning so
> > >> > > your Tuesday morning and London's afternoon would be the best.
> > >> > >
> > >> > > Recording the thing would be definitely helpful too.
> > >> > >
> > >> > > On Sat, 13 Apr 2019 at 07:45, Jon Haddad  wrote:
> > >> > > >
> > >> > > > I'd be more than happy to hop on a call next week to give you both
> > >> > > > (and anyone else interested) a tour of our dev tools.  Maybe
> > >> something
> > >> > > > early morning on my end, which should be your evening, could work?
> > >> > > >
> > >> > > > I can set up a Zoom conference to get everyone acquainted.  We can
> > >> > > > record and post it for any who can't make it.
> > >> > > >
> > >> > > > I'm thinking Tuesday, Wednesday, or Thursday morning, 9AM Pacific
> > >> (5pm
> > >> > > > London)?  If anyone's interested please reply with what dates work.
> > >> > > > I'll be sure to post the details back here with the zoom link in
> > >> case
> > >> > > > anyone wants to join that didn't get a chance to reply, as well as 
> > >> > > > a
> > >> > > > link to the recorded call.
> > >> > > >
> > >> > > > Jon
> > >> > > >
> > >> > > > On Fri, Apr 12, 2019 at 10:41 AM Benedict Elliott Smith
> > >> > > >  wrote:
> > >> > > > >
> > >> > > > > +1
> > >> > > > >
> > >> > > > > I’m also just as excited to see