Re: [VOTE] CEP-8 Datastax Drivers Donation

[DuyHai Doan]

+1 nb

On Tue, Jun 13, 2023 at 8:00 PM C. Scott Andreas 
wrote:

> +1nb
>
> On Jun 13, 2023, at 10:25 AM, German Eichberger via dev <
> dev@cassandra.apache.org> wrote:
>
> 
> + 1
>
> Great to see this moving forward!
> --
> *From:* Abe Ratnofsky 
> *Sent:* Tuesday, June 13, 2023 10:09 AM
> *To:* dev@cassandra.apache.org 
> *Subject:* [EXTERNAL] Re: [VOTE] CEP-8 Datastax Drivers Donation
>
> +1 (nb)
>
> On Jun 13, 2023, at 09:23, Andrés de la Peña  wrote:
>
> 
> +1
>
> On Tue, 13 Jun 2023 at 16:40, Yifan Cai  wrote:
>
> +1
> --
> *From:* David Capwell 
> *Sent:* Tuesday, June 13, 2023 8:37:10 AM
> *To:* dev 
> *Subject:* Re: [VOTE] CEP-8 Datastax Drivers Donation
>
> +1
>
> On Jun 13, 2023, at 7:59 AM, Josh McKenzie  wrote:
>
> +1
>
> On Tue, Jun 13, 2023, at 10:55 AM, Jeremiah Jordan wrote:
>
> +1 nb
>
> On Jun 13, 2023 at 9:14:35 AM, Jeremy Hanna 
> wrote:
>
>
> Calling for a vote on CEP-8 [1].
>
> To clarify the intent, as Benjamin said in the discussion thread [2], the
> goal of this vote is simply to ensure that the community is in favor of
> the donation. Nothing more.
> The plan is to introduce the drivers, one by one. Each driver donation
> will need to be accepted first by the PMC members, as it is the case for
> any donation. Therefore the PMC should have full control on the pace at
> which new drivers are accepted.
>
> If this vote passes, we can start this process for the Java driver under
> the direction of the PMC.
>
> Jeremy
>
> 1.
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-8%3A+Datastax+Drivers+Donation
> 2. https://lists.apache.org/thread/opt630do09phh7hlt28odztxdv6g58dp
>
>
>

Re: Vector search demo, and query syntax

[DuyHai Doan]

Hello all

Sorry to disturb the discussion but there is an official announcement from
Microsoft about CosmosDB supporting Vector Search

https://devblogs.microsoft.com/cosmosdb/introducing-vector-search-in-azure-cosmos-db-for-mongodb-vcore/

Looks like Jonathan is spot on about this feature, it's quite a hot topic
for the moment !

Regards

Duy Hai DOAN

On Wed, May 24, 2023 at 2:44 PM Josh McKenzie 
wrote:

> +1 to the flow of:
>
> 1: ORDER BY?
>
> 2:  Oh. Yeah. That *does *makes sense.
>
> ;)
>
> (sending from fastmail in the hopes the image doesn't get stripped. Thanks
> ASF smtp server...)
>
> ~Josh
>
> On Wed, May 24, 2023, at 1:00 AM, Jeremiah D Jordan wrote:
>
> At first I wasn’t sure about using ORDER BY, but the more I think about
> what is actually going on, I think it does make sense.
>
> This also matches up with some ideas that have been floating around about
> being able to ORDER BY a sorted SAI index.
>
> -Jeremiah
>
> On May 22, 2023, at 2:28 PM, Jonathan Ellis  wrote:
>
> Hi all,
>
> I have a branch of vector search based on cep-7-sai at
> *https://github.com/datastax/cassandra/tree/cep-vsearch*
> . Compared to the
> original POC branch, this one is based on the SAI code that will be
> mainline soon, and handles distributed scatter/gather.  Updates and deletes
> to vector values are still not supported.
>
> I also put together a demo that uses this branch to provide context to
> OpenAI’s GPT, available here: *https://github.com/jbellis/cassgpt*
> .
>
> Here is the query that gets executed:
>
> SELECT id, start, end, text
> FROM {self.keyspace}.{self.table}
> WHERE embedding ANN OF %s
> LIMIT %s
>
> The more I used the proposed `ANN OF` syntax, the less I liked it.  This
> is because we don’t want an actual boolean predicate; we just want to order
> results.  Put another way, `ANN OF` will include all rows of the table
> given a high enough `LIMIT`, and that makes it a bad fit for expression
> processing that expects to be able to filter out rows before it starts
> LIMIT-ing.  And in fact the code to support executing the query looks
> suspiciously like what you’d want for `ORDER BY`.
>
> I propose that we adopt `ORDER BY` syntax, supporting it for vector
> indexes first and eventually for all SAI indexes.  So this query would
> become
>
> SELECT id, start, end, text
> FROM {self.keyspace}.{self.table}
> ORDER BY embedding ANN OF %s
> LIMIT %s
>
> And it would compose with other SAI indexes with syntax like
>
> SELECT id, start, end, text
> FROM {self.keyspace}.{self.table}
> WHERE publish_date > %s
> ORDER BY embedding ANN OF %s
> LIMIT %s
>
> Related work:
>
> This is similar to the approach used by pgvector, except they invented the
> symbolic operator `<->` that has the same semantics as `ANN OF`.  I am okay
> with adopting their operator, but I think ANN OF is more readable.
>
> --
> Jonathan Ellis
> co-founder, http://www.datastax.com
> @spyced
>
>
>

Re: Welcome Patrick McFadin as Cassandra Committer

[DuyHai Doan]

Congratulations to Patrick ! After those years serving the community it is
very well deserved !

Le mer. 8 févr. 2023, 18:43, Mick Semb Wever  a écrit :

>
> Long overdue with so much you have done for so many years. Congrats!
>
> On Thu, 2 Feb 2023 at 23:26, Molly Monroy  wrote:
>
>> Congrats, Patrick... much deserved!
>>
>> On Thu, Feb 2, 2023 at 1:59 PM Derek Chen-Becker 
>> wrote:
>>
>>> Congrats!
>>>
>>> On Thu, Feb 2, 2023 at 10:58 AM Benjamin Lerer 
>>> wrote:
>>>
 The PMC members are pleased to announce that Patrick McFadin has
 accepted
 the invitation to become committer today.

 Thanks a lot, Patrick, for everything you have done for this project
 and its community through the years.

 Congratulations and welcome!

 The Apache Cassandra PMC members

>>>
>>>
>>> --
>>> +---+
>>> | Derek Chen-Becker |
>>> | GPG Key available at https://keybase.io/dchenbecker
>>> 
>>> and   |
>>> | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org
>>> 
>>> |
>>> | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
>>> +---+
>>>
>>>

Re: [VOTE] CEP-7: Storage Attached Index

[DuyHai Doan]

+1 nb

On Fri, Feb 18, 2022 at 7:41 AM Berenguer Blasi 
wrote:

> +1
> On 18/2/22 2:15, Jasonstack Zhao Yang wrote:
>
> +1
>
> On Fri, 18 Feb 2022 at 08:15, Jeremy Hanna 
> wrote:
>
>> +1 nb. Thanks Caleb, Mike, Jason, and everyone involved with the effort.
>>
>> On Feb 17, 2022, at 4:23 PM, Caleb Rackliffe 
>> wrote:
>>
>> 
>> Hi everyone,
>>
>> I'd like to call a vote to approve CEP-7.
>>
>> Proposal:
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-7%3A+Storage+Attached+Index
>>
>> Discussion:
>> https://lists.apache.org/thread/hh67k3t86m7299qkt61gmzb4h96bl90w
>>
>> The vote will be open for 72 hours.
>> Votes by committers are considered binding.
>> A vote passes if there are at least three binding +1s and no binding
>> vetoes.
>>
>> Thanks!
>> Caleb
>>
>>

Re: Implementing a secondary index

[DuyHai Doan]

Hello Claude

I have written a blog post about 2nd index architecture a long time ago but
most of the content should still be relevant, worth checking

https://www.doanduyhai.com/blog/?p=13191

Regards

Duy Hai DOAN

Le mer. 17 nov. 2021 à 10:17, Claude Warren 
a écrit :

> Greetings,
>
> I am looking to implement a Multidimensional Bloom filter index [1] [2] on
> a Cassandra table.  OK, I know that is a lot to take in.  What I need is
> any documentation that explains the architecture of the index options, or
> someone I can ask questions of -- a mentor if you will.
>
> I have a proof of concept for the index that works from the client side
> [3].  What I want to do is move some of that processing to the server
> side.
>
> I basically I think I need to do the following:
>
>1. On each partition create an SST to store the index data.  This table
>comprises, 2 integer data points and the primary key for the data table.
>2. When the index cell gets updated in the original table (there will
>only be on column), update one or more rows in the SST table.
>3. When querying perform multiple queries against the index data, and
>return the primary key values (or the data associated with the primary
> keys
>-- I am unclear on this bit).
>
> Any help or guidance would be appreciated,
> Claude
>
> [1] https://archive.org/details/arxiv-1501.01941/mode/2up
> [2] https://archive.fosdem.org/2020/schedule/event/bloom_filters/
> [3] https://github.com/Claude-at-Instaclustr/blooming_cassandra
>
>
>
>
> --
>
> [image: Instaclustr logo]
>
>
> *Claude Warren*
>
> Principal Software Engineer
>
> Instaclustr
>

Re: [DISCUSS] CEP-7 Storage Attached Index

[DuyHai Doan]

t;>>>>> based
> >>>>>>>>
> >>>>>>>> on a
> >>>>>>>>
> >>>>>>>> discussion
> >>>>>>>>
> >>>>>>>> thread or threads for higher bandwidth discussion.
> >>>>>>>>
> >>>>>>>> I would be happy to schedule on for next week to
> >>>>>>>>
> >>>>>>>> specifically
> >>>>>>>>
> >>>>>>>> discuss
> >>>>>>>>
> >>>>>>>> CEP-7. I can attach the recorded call to the CEP after.
> >>>>>>>>
> >>>>>>>> +1 or -1?
> >>>>>>>>
> >>>>>>>> Patrick
> >>>>>>>>
> >>>>>>>> On Tue, Aug 25, 2020 at 7:03 AM Joshua McKenzie <
> >>>>>>>>
> >>>>>>>> jmcken...@apache.org>
> >>>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Does community plan to open another discussion or CEP
> >>>>>>>>
> >>>>>>>> on
> >>>>>>>>
> >>>>>>>> modularization?
> >>>>>>>>
> >>>>>>>> We probably should have a discussion on the ML or
> >>>>>>>>
> >>>>>>>> monthly
> >>>>>>>>
> >>>>>>>> contrib
> >>>>>>>>
> >>>>>>>> call
> >>>>>>>>
> >>>>>>>> about it first to see how aligned the interested
> >>>>>>>>
> >>>>>>>> contributors
> >>>>>>>>
> >>>>>>>> are.
> >>>>>>>>
> >>>>>>>> Could
> >>>>>>>>
> >>>>>>>> do
> >>>>>>>>
> >>>>>>>> that through CEP as well but CEP's (at least thus far
> >>>>>>>>
> >>>>>>>> sans k8s
> >>>>>>>>
> >>>>>>>> operator)
> >>>>>>>>
> >>>>>>>> tend to start with a strong, deeply thought out point of
> >>>>>>>>
> >>>>>>>> view
> >>>>>>>>
> >>>>>>>> being
> >>>>>>>>
> >>>>>>>> expressed.
> >>>>>>>>
> >>>>>>>> On Tue, Aug 25, 2020 at 3:26 AM Jasonstack Zhao Yang <
> >>>>>>>>
> >>>>>>>> jasonstack.z...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>> SASI's performance, specifically the search in the
> >>>>>>>>
> >>>>>>>> B+
> >>>>>>>>
> >>>>>>>> tree
> >>>>>>>>
> >>>>>>>> component,
> >>>>>>>>
> >>>>>>>> depends a lot on the component file's header being
> >>>>>>>>
> >>>>>>>> available
> >>>>>>>>
> >>>>>>>> in
> >>>>>>>>
> >>>>>>>> the
> >>>>>>>>
> >>>>>>>> pagecache. SASI benefits from (needs) nodes with
> >>>>>>>>
> >>>>>>>> lots of
> >>>>>>>>
> >>>>>>>> RAM.
> >>>>>>>>
> >>>>>>>> Is
> >>>>>>>>
> >>>>>>>> SAI
> >>>>>>>>
> >>>>>>>> bound
> >>>>>>>>
> >>>>>>>> to this same or similar limitation?
> >>>>>>>>
> >>>>>>>> SAI also benefits from larger memory because SAI puts
> >>>>>>>>
> >>>>>>>> block
> >>>>>>>>
> >>>>>>>> info
> >>>>>>>

Re: [RELEASE] Apache Cassandra 4.0.0 released

[DuyHai Doan]

Congrats !!!

3 years worth of waiting and finally released !!!



On Tue, Jul 27, 2021 at 12:02 AM Ben Slater 
wrote:

> Congratulations and thank you to everyone involved in getting 4.0 released!
> It has been a very impressive community effort.
>
> ---
>
>
> *Ben Slater**Chief Product Officer*
>
>
>    
> 
>
> Read our latest technical blog posts here
> .
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>
>
> On Tue, 27 Jul 2021 at 07:34, Stefan Miklosovic <
> stefan.mikloso...@instaclustr.com> wrote:
>
> > Super exciting, congratulations everybody. Great times ahead!
> >
> > On Mon, 26 Jul 2021 at 22:19, Patrick McFadin 
> wrote:
> > >
> > > Wow. Just wow. Congratulations to everyone involved in this huge
> > milestone.
> > >
> > > On Mon, Jul 26, 2021 at 1:04 PM Brandon Williams 
> > wrote:
> > >
> > > > The Cassandra team is pleased to announce the release of Apache
> > > > Cassandra version 4.0.0.
> > > >
> > > > Apache Cassandra is a fully distributed database. It is the right
> > > > choice when you need scalability and high availability without
> > > > compromising performance.
> > > >
> > > > http://cassandra.apache.org/
> > > >
> > > > Downloads of source and binary distributions are available in our
> > > > download section:
> > > >
> > > > http://cassandra.apache.org/download/
> > > >
> > > > This version is the initial release in the 4.0 series. As always,
> > > > please pay attention to the release notes[2] and Let us know[3] if
> you
> > > > were to encounter any problem.
> > > >
> > > > Enjoy!
> > > >
> > > > [1]: CHANGES.txt
> > > >
> > > >
> >
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0.0
> > > > [2]: NEWS.txt
> > > >
> >
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0.0
> > > > [3]: https://issues.apache.org/jira/browse/CASSANDRA
> > > >
> > > > -
> > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > > >
> > > >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>

Re: [DISCUSS] CEP-7 Storage Attached Index

[DuyHai Doan]

Last but not least

4) Are collections, static columns, composite partition key composent and
UDT indexings (at any depth) on the roadmap of SAI ? I strongly believe
that those features are the bare minimum to make SAI an interesting
replacement for the native 2nd index as well as SASI. SASI limited support
for those advanced data structures has hindered its wide adoption (among
other issues and bugs)

Regards

Duy Hai DOAN

Le mar. 18 août 2020 à 13:02, Jasonstack Zhao Yang <
jasonstack.z...@gmail.com> a écrit :

> Mick thanks for your questions.
>
> > During the 4.0 beta phase this was intended to be addressed, i.e.>
> defining more specific QA guidelines for 4.0-rc. This would be an important
> > step towards QA guidelines for all changes and CEPs post-4.0.
>
> Agreed, I think CASSANDRA-15536
>  (4.0 Quality:
> Components and Test Plans) has set a good example of QA/Testing.
>
> >  - How will this be tested, how will its QA status and lifecycle be>
> defined? (per above)
>
> SAI will follow the same QA/Testing guideline as in CASSANDRA-15536.
>
> >  - With existing C* code needing to be changed, what is the proposed
> plan> for making those changes ensuring maintained QA, e.g. is there
> separate QA
> > cycles planned for altering the SPI before adding a new SPI
> implementation?
>
> The plan is to have interface changes and their new implementations to be
> reviewed/tested/merged at once to reduce overhead.
>
> But if having interface changes reviewed/tested/merged separately helps
> quality, I don't think anyone will object.
>
> > - Despite being out of scope, it would be nice to have some idea from
> the>  CEP author of when users might still choose afresh 2i or SASI over
> SAI
>
> I'd like SAI to be the only index for users, but this is a decision to be
> made by the community.
>
> > - Who fills the roles involved?
>
> Contributors that are still active on C* or related projects:
>
> Andres de la Peña
> Caleb Rackliffe
> Dan LaRocque
> Jason Rutherglen
> Mike Adamson
> Rocco Varela
> Zhao Yang
>
> I will shepherd.
>
> Anyone that is interested in C* index, feel free to join us at slack
> #cassandra-sai.
>
> > - Is there a preference to use gdoc instead of the project's wiki, and>
> why? (the CEP process suggest a wiki page, and feedback on why another
> > approach is considered better helps evolve the CEP process itself)
>
> Didn't notice wiki is required. Will port CEP to wiki.
>
>
> On Tue, 18 Aug 2020 at 17:39, Mick Semb Wever  wrote:
>
> > >
> > > We are looking forward to the community's feedback and suggestions.
> > >
> >
> >
> > What comes immediately to mind is testing requirements. It has been
> > mentioned already that the project's testability and QA guidelines are
> > inadequate to successfully introduce new features and refactorings to the
> > codebase. During the 4.0 beta phase this was intended to be addressed,
> i.e.
> > defining more specific QA guidelines for 4.0-rc. This would be an
> important
> > step towards QA guidelines for all changes and CEPs post-4.0.
> >
> > Questions from me
> >  - How will this be tested, how will its QA status and lifecycle be
> > defined? (per above)
> >  - With existing C* code needing to be changed, what is the proposed plan
> > for making those changes ensuring maintained QA, e.g. is there separate
> QA
> > cycles planned for altering the SPI before adding a new SPI
> implementation?
> >  - Despite being out of scope, it would be nice to have some idea from
> the
> > CEP author of when users might still choose afresh 2i or SASI over SAI,
> >  - Who fills the roles involved? Who are the contributors in this
> DataStax
> > team? Who is the shepherd? Are there other stakeholders willing to be
> > involved?
> >  - Is there a preference to use gdoc instead of the project's wiki, and
> > why? (the CEP process suggest a wiki page, and feedback on why another
> > approach is considered better helps evolve the CEP process itself)
> >
> > cheers,
> > Mick
> >
>

Re: [DISCUSS] CEP-7 Storage Attached Index

[DuyHai Doan]

Thank you Zhao Yang for starting this topic

After reading the short design doc, I have a few questions

1) SASI was pretty inefficient indexing wide partitions because the index
structure only retains the partition token, not the clustering colums. As
per design doc SAI has row id mapping to partition offset, can we hope that
indexing wide partition will be more efficient with SAI ? One detail that
worries me is that in the beggining of the design doc, it is said that the
matching rows are post filtered while scanning the partition. Can you
confirm or infirm that SAI is efficient with wide partitions and provides
the partition offsets to the matching rows ?

2) About space efficiency, one of the biggest drawback of SASI was the huge
space required for index structure when using CONTAINS logic because of the
decomposition of text columns into n-grams. Will SAI suffer from the same
issue in future iterations ? I'm anticipating a bit

3) If I'm querying using SAI and providing complete partition key, will it
be more efficient than querying without partition key. In other words, does
SAI provide any optimisation when partition key is specified ?

Regards

Duy Hai DOAN

Le mar. 18 août 2020 à 11:39, Mick Semb Wever  a écrit :

> >
> > We are looking forward to the community's feedback and suggestions.
> >
>
>
> What comes immediately to mind is testing requirements. It has been
> mentioned already that the project's testability and QA guidelines are
> inadequate to successfully introduce new features and refactorings to the
> codebase. During the 4.0 beta phase this was intended to be addressed, i.e.
> defining more specific QA guidelines for 4.0-rc. This would be an important
> step towards QA guidelines for all changes and CEPs post-4.0.
>
> Questions from me
>  - How will this be tested, how will its QA status and lifecycle be
> defined? (per above)
>  - With existing C* code needing to be changed, what is the proposed plan
> for making those changes ensuring maintained QA, e.g. is there separate QA
> cycles planned for altering the SPI before adding a new SPI implementation?
>  - Despite being out of scope, it would be nice to have some idea from the
> CEP author of when users might still choose afresh 2i or SASI over SAI,
>  - Who fills the roles involved? Who are the contributors in this DataStax
> team? Who is the shepherd? Are there other stakeholders willing to be
> involved?
>  - Is there a preference to use gdoc instead of the project's wiki, and
> why? (the CEP process suggest a wiki page, and feedback on why another
> approach is considered better helps evolve the CEP process itself)
>
> cheers,
> Mick
>

Re: [proposal] Introduce AssertJ in test framework

[DuyHai Doan]

Definitely +1

Coding in Java every day, I can't write test without assertJ. Once you get
to know assertJ, it's impossible to go back to basic assertions of JUnit
that feels really boilerplate



On Tue, Mar 10, 2020 at 8:53 PM Jordan West  wrote:

> If it encourages more  and higher quality test writing +1 (nb). Also, low
> risk given it’s a test dep.
>
> Using QuickTheories as an example, merging it with a new or updated test
> could be a good way to get it merged
>
> Jordan
>
> On Tue, Mar 10, 2020 at 10:33 AM Benjamin Lerer <
> benjamin.le...@datastax.com>
> wrote:
>
> > +1
> >
> > On Tue, Mar 10, 2020 at 6:18 PM Jon Haddad  wrote:
> >
> > > I've used assertj in a lot of projects, I prefer it by a wide margin
> over
> > > using only junit.
> > >
> > > On Tue, Mar 10, 2020 at 9:45 AM David Capwell 
> > wrote:
> > >
> > > > +1 from me
> > > >
> > > > In CASSANDRA-15564 I build my own assert chain to make the tests
> > cleaner;
> > > > did it since assertj wasn't there.
> > > >
> > > > On Tue, Mar 10, 2020, 9:28 AM Kevin Gallardo <
> > > kevin.galla...@datastax.com>
> > > > wrote:
> > > >
> > > > > I would like to propose adding AssertJ <
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__assertj.github.io_doc_=DwIFaQ=adz96Xi0w1RHqtPMowiL2g=Jad7nE1Oab1mebx31r7AOfSsa0by8th6tCxpykmmOBA=WrkWi_LeOnAl7rqft1DM27OEkXD7sc2fnZMy_-c7IS8=D4FAaGRQi2WlwKAOQbQMfyt_cRqsOuZdePUDgchdLhA=
> > > >
> > > > as
> > > > > a test dependency and therefore have it available for writing
> > > > > unit/distributed/any test assertions.
> > > > >
> > > > > In addition to the examples mentioned on the AssertJ docs page
> > (allows
> > > to
> > > > > do elaborate and comprehensible assertions on Collections, String,
> > and
> > > > > *custom
> > > > > assertions*), here's an example of a dtest I was looking at, that
> > could
> > > > be
> > > > > translated to AssertJ syntax, just to give an idea of how the
> syntax
> > > > would
> > > > > apply:
> > > > >
> > > > > *JUnit asserts*:
> > > > > try {
> > > > >[...]
> > > > > } catch (Exception e) {
> > > > > Assert.assertTrue(e instanceof RuntimeException);
> > > > > RuntimeException re = ((RuntimeException) e);
> > > > > Assert.assertTrue(re.getCause() instanceof
> ReadTimeoutException);
> > > > > ReadTimeoutException rte = ((ReadTimeoutException)
> e.getCause());
> > > > > Assert.assertTrue(rte.getMessage().contains("blabla")
> > > > >   && rte.getMessage().contains("andblablo"));
> > > > > }
> > > > >
> > > > > *AssertJ style:*
> > > > > try {
> > > > > [...]
> > > > > } catch (Exception e) {
> > > > > Assertions.assertThat(e)
> > > > > .isInstanceOf(RuntimeException.class)
> > > > > .hasCauseExactlyInstanceOf(ReadTimeoutException.class)
> > > > > .hasMessageContaining("blabla")
> > > > > .hasMessageContaining("andblablo");
> > > > > }
> > > > >
> > > > > The syntax is more explicit and more comprehensible, but more
> > > > importantly,
> > > > > when one of the JUnit assertTrue() fails, you don't know *why*, you
> > > only
> > > > > know that the resulting boolean expression is false.
> > > > > If a failure happened with the assertJ tests, the failure would say
> > > > > "Exception
> > > > > did not contain expected message, expected "blabla", actual
> > > "notblabla""
> > > > > (same for a lot of other situations), this makes debugging a
> failure,
> > > > after
> > > > > a test ran and failed much easier. With JUnit asserts you would
> have
> > to
> > > > > additionally add a message explaining what the expected value is
> > *and*
> > > > > what the
> > > > > actual value is, for each assert that is more complex than a
> > > assertEquals
> > > > > on a number, I suppose. I have seen a lot of tests so far that only
> > > test
> > > > > the expected behavior via assertTrue and does not show the
> incorrect
> > > > values
> > > > > when the test fails, which would come for free with AssertJ.
> > > > >
> > > > > Other examples randomly picked from the test suite:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> *org.apache.cassandra.repair.RepairJobTest#testNoTreeRetainedAfterDistance:*
> > > > > Replace assertion:
> > > > > assertTrue(messages.stream().allMatch(m -> m.verb() ==
> > Verb.SYNC_REQ));
> > > > > With:
> > > > > assertThat(messages)
> > > > > .extracting(Message::verb)
> > > > > .containsOnly(Verb.SYNC_REQ);
> > > > >
> > > > > As a result, if any of the messages is not a Verb.SYNC_REQ, the
> test
> > > > > failure will show the actual "Verb"s of messages.
> > > > >
> > > > > Replace:
> > > > > assertTrue(millisUntilFreed < TEST_TIMEOUT_S * 1000);
> > > > > With:
> > > > > assertThat(millisUntilFreed)
> > > > > .isLessThan(TEST_TIMEOUT_S * 1000);
> > > > >
> > > > > Same effect if the condition is not satisfied, more explicit error
> > > > message
> > > > > explaining why the test failed.
> > > > >
> > > > > AssertJ also allows Custom assertions 

Re: time for a release?

[DuyHai Doan]

+1 too (non binding)

On Fri, Oct 4, 2019 at 10:33 PM Scott Andreas  wrote:
>
> +1nb for me for the 3.x releases.
>
> The user-facing issues resolved in 2.2 are slimmer and relatively minor (just 
> 15225, 15050, 15045, 15041), but if it makes sense to release all three 
> together, sounds good to me.
>
> – Scott
>
> On 10/4/19, 11:00 AM, "Jon Haddad"  wrote:
>
> It's been a while since we did a release and I think there's enough in 
> here
> to put one out.
>
> 2.2.15 changes:
> 
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20CASSANDRA%20AND%20fixVersion%20%3D%202.2.15%20and%20status%20%3D%20Resolved%20
>
> 3.0.19 changes:
> 
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20CASSANDRA%20AND%20fixVersion%20%3D%203.0.19%20and%20status%20%3D%20Resolved%20%20
>
> 3.11.5 changes:
> 
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20CASSANDRA%20AND%20fixVersion%20%3D%203.11.5%20and%20status%20%3D%20Resolved%20
>
> Any reason not to put this up for a vote?  If I don't hear anything by
> Monday I'll start a vote thread.
>
> Jon
>
>

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: ACNA 2019 Schedule is out

[DuyHai Doan]

So sad, at least let's hope that the slides will be available online so
people who can't attend will catch up

Thanks

On Sun, Jun 9, 2019 at 10:27 PM Nate McCall  wrote:

> The topics look really interesting. Do you know if it will be recorded for
> > people to catch up later?
> >
> > Regards
> >
> >
>  That's being discussed on the conference planning list. Unfortunately at
> this point It doesnt look like they have the budget/volunteers for it this
> year.
>

Re: ACNA 2019 Schedule is out

[DuyHai Doan]

The topics look really interesting. Do you know if it will be recorded for
people to catch up later?

Regards

On Sat, Jun 8, 2019 at 1:15 AM Nate McCall  wrote:

> Also, quick note that if you got an acceptance email from the ASF, but dont
> see your track on the schedule yet, it's probably slotted into one of the
> NGCC spots where we had to squeeze two talks in to fit in with the ASF's
> scheduling requirements. These requirements were significantly less
> flexible than we originally thought, so apologies for the complexity. We
> felt it a fair tradeoff to get more topics covered than what the original
> six slots for NGCC allowed.
>
> Specifically slots for NGCC on the current schedule with generic groupings
> include these talks (note that these will have to be 20min presentations
> with 5 to 10mins for questions):
> "Cassandra Community"
> - Committing to our Users: Product + Release Management in Apache Cassandra
> - Apache Cassandra community health
>
> "Partition Management"
> - Row-Level Repair
> - Partition based compaction strategy
>
> "Transactions and consistent ownership"
> - Consistent ownership management for Cassandra
> - Transaction Management on Cassandra
>
> Please let us know if you have questions. The main ACNA19 schedule will
> update in a few days to better reflect this more specifically once the dust
> settles a bit.
>
> Thanks,
> -Nate
>
>
> On Fri, Jun 7, 2019 at 11:18 AM Nate McCall  wrote:
>
> > Hi Folks,
> > Thanks to everyone who submitted talks and helped out reviewing. We ended
> > up with a lot more material than we could make room for, but I think we
> put
> > together what will be a good track:
> > https://www.apachecon.com/acna19/schedule.html
> >
> > If you submitted a talk and got accepted the ASF folks should be in touch
> > shortly with details. There were some good proposals that we did not end
> up
> > accepting in the interests of creating a well rounded schedule. Either
> way,
> > thank you again for taking the time to put something together and submit
> > it.
> >
> > For all of you, but particularly if you got a talk accepted, please help
> > promote as much as possible. This will be a good conf for us regardless
> > since we are starting off with an NGCC, but given the strength of our
> > community, I think we can really help the ASF get the word out and get
> > people in chairs.
> >
> > Thanks again,
> > -Nate
> >
>

Level N atomic updates for UDT necessary

[DuyHai Doan]

Hello devs

After CASSANDRA-7423 (Cassandra 3.6), it is possible to declare un-frozen
UDT at 1st level and more important/interesting, it is possible to update
atomically individual fields on an UDT (without the need to rewrite the UDT
entirely)

This JIRA opens tremendous new opportunity in term of data modeling. It is
sensible to store hierarchical data (think JSON/document) using UDT and
collections.

However, update of individual fields at level 2 and deeper is still not
supported (UDT must be frozen at level 2 and onward)

This limitation makes the update of individual fields at deeper level a
nightmare.

Let's take a contrive example:

CREATE TYPE phone_number (
international_prefix text,
local_prefix text,
suffix text,
type text // HOME, WORK, MOBILE, LANDLINE ...
);

CREATE TYPE contact (
firstname text,
lastname text,
phone phone_number,
email text,
...
);

CREATE TABLE user_contacts(
   user_id uuid,
   contact_id uuid,
   contact_details contact,
   PRIMARY KEY ((user_id), contact_id)
);

Now, to update a contact phone_number, one has to:

- perform a SELECT contact_details.phone_number FROM user_contacts WHERE
user_id=.. AND contact_id = ...
- perform an UPDATE user_contacts SET contact_details.phone_number = ...
WHERE user_id=.. AND contact_id = ...

Not only this update procedure is bad (but mandatory) because it implies a
read-before-write anti-pattern but it also exposes the end-user to
horrendous concurrency issues.

If there are 2 concurrent updates on the same phone_number nested UDT, one
changing the type field and the other changing the suffix value, we will
face data loss:
  - either the 1st update wins (in term of LWW) and the type is updated but
not the suffix
  - or the 2nd update wins and the suffix is updated but not the type

Ideally we would like to have both fields type & suffix updated. The only
fix currently available is to rely on LWT and using a version column as
optimistic concurrency locking:

UPDATE user_contacts SET contact_details.phone_number = ..., version=2
WHERE user_id=.. AND contact_id = ... IF version=1

This guarantees that only 1 concurrent update succeeds at a time and force
the failing updates re-fetching the fresh data and retry

Of course, LWT has a huge cost and is overkill to solve such a problem.

Thus my question are:

- is there any plan to extend the UDT field updates to deeper level
- is it complicated to do so ? I'm tempted to cast a glance and attempt a
patch but I would like to know if it is going a gigantic task or not

Thanks

Regards

Duy Hai DOAN

Re: Using Cassandra as local db without cluster

[DuyHai Doan]

I have an application with a purpose to store a dynamic number of colones
on each rows (thing that i cannot do with classical relational database),

---> Postgresql  allows you tu use array type or map type with dynamic
number of records, provided of course that the cardinality of the
collection is not "too" big

On Thu, Oct 18, 2018 at 12:31 PM Abdelkrim Fitouri 
wrote:

> Hello,
>
> I am wondering if using cassandra as one local database without the cluster
> capabilities has a sens, (i cannot do multi node cluster due to a technical
> constraint)
>
> I have an application with a purpose to store a dynamic number of colones
> on each rows (thing that i cannot do with classical relational database),
> and i don't want to use documents based nosql database to avoid using Json
> marshal and unmarshal treatments...
>
> Does cassandra with only one node and with a well designer model based on
> queries and partition keys can lead to best performance than postgresql ?
>
> Does cassandra have some limitation about the size of data ? about the
> number of partition on a node ?
>
> Thanks for any details or help.
>
> --
>
> Best Regards.
>

Re: Roadmap for 4.0

[DuyHai Doan]

> I'd like to see pluggable storage and transient replica tickets land, for
> starters.

So after all the fuss and scandal about incremental repair and MV not
stable and being downgraded to experimental, I would like to suggest that
those new features are also flagged as experimental for some time for the
community to use them extensively before being promoted as first class
features

Thoughts ?

On Mon, Apr 9, 2018 at 11:36 PM, Jeff Beck  wrote:

> If you are going to make 4 bigger as long as we call out that 3.11.x (or
> whatever) will keep getting patches for stability only that's all that's
> needed. We haven't gone to 3.x releases many places yet as we wait for a
> release that will be stable longer. Knowing 4 is going to be bigger I
> wouldn't want to see more feature releases in 3.x
>
> I wouldn't want to greatly slow features down if they require a major
> release and 5 is too far off.
>
> Jeff
>
> On Mon, Apr 9, 2018, 4:05 PM Josh McKenzie  wrote:
>
> > >
> > > If they're not close to finished now why even consider them for the 4.0
> > > release?
> >
> > Merging in major features at the end of a release cycle is not the path
> to
> > stability, imo.
> >
> > On Mon, Apr 9, 2018 at 4:56 PM, Jonathan Haddad 
> wrote:
> >
> > > There's always more stuff to try to shoehorn in.  We've done big
> releases
> > > with all the things, it never was stable.  We tried the opposite end of
> > the
> > > spectrum, release every month, that really wasn't great either.
> > Personally
> > > I'd be OK with stopping new features by the end of this month and
> aiming
> > to
> > > release a stable 4.0 when we agree we would be comfortable dogfooding
> it
> > in
> > > production at our own companies (in a few months), and aim for 4.1 (or
> > 5.0
> > > I don't want to bikeshed the version) for pluggable storage and
> transient
> > > replicas.  If they're not close to finished now why even consider them
> > for
> > > the 4.0 release?  They're so core they should be merged into trunk at
> the
> > > beginning of the cycle for the follow up release in order to get as
> much
> > > exposure as possible.
> > >
> > > Jon
> > >
> > > On Mon, Apr 9, 2018 at 1:46 PM Nate McCall  wrote:
> > >
> > > > > I'd like to see pluggable storage and transient replica tickets
> land,
> > > for
> > > > > starters.
> > > >
> > > > I think both those features are, frankly, necessary for our future.
> On
> > > > the other hand, they both have the following risks:
> > > > 1. core behavioral changes
> > > > 2. require changing a (relatively) large surface area of code
> > > >
> > > > We can aim to de-risk 4.0 by focusing on what we have now which is
> > > > solid repair and NIO internode (maybe we move the 4.0 branch timeline
> > > > up?), aiming for a 4.1 following soon-ish.
> > > >
> > > > Or we can go in eyes open and agree on a larger footprint 4.0.
> > > >
> > > > I'm on the fence, tbh (can't emphasize enough how big both those
> > > > features will be). I just want everyone to know what we are getting
> > > > into and that we are potentially impacting our goals of "stable" ==
> > > > "exciting."
> > > >
> > > > 
> -
> > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > > >
> > > >
> > >
> >
>

Re: Roadmap for 4.0

[DuyHai Doan]

My wish list:

* Add support for arithmetic operators (CASSANDRA-11935)
* Allow IN restrictions on column families with collections
(CASSANDRA-12654)
* Add support for + and - operations on dates (CASSANDRA-11936)
* Add the currentTimestamp, currentDate, currentTime and currentTimeUUID
functions (CASSANDRA-13132)
* Allow selecting Map values and Set elements (CASSANDRA-7396)

Those are mostly useful for timeseries data models and I guess has no
significant impact on the internals and operations so the risk of
regression is low

On Mon, Apr 2, 2018 at 4:33 PM, Jeff Jirsa  wrote:

> 9608 (java9)
>
> --
> Jeff Jirsa
>
>
> > On Apr 2, 2018, at 3:45 AM, Jason Brown  wrote:
> >
> > The only additional tickets I'd like to mention are:
> >
> > https://issues.apache.org/jira/browse/CASSANDRA-13971 - Automatic
> > certificate management using Vault
> > - Stefan's Vault integration work. A sub-ticket, CASSANDRA-14102,
> addresses
> > encryption at-rest, subsumes CASSANDRA-9633 (SSTable encryption) - which
> I
> > doubt I would be able to get to any time this year. It would definitely
> be
> > nice to have a clarified encryption/security story for 4.0.
> >
> > https://issues.apache.org/jira/browse/CASSANDRA-11990 - Address rows
> rather
> > than partitions in SASI
> > - a nice update for SASI, but not critical.
> >
> > -Jason
> >
> >> On Sat, Mar 31, 2018 at 6:53 PM, Ben Bromhead 
> wrote:
> >>
> >> Apologies all, I didn't realize I was responding to this discussion
> only on
> >> the @user list. One of the perils of responding to a thread that is on
> both
> >> user and dev...
> >>
> >> For context, I have included my response to Kurt's previous discussion
> on
> >> this topic as it only ended up on the user list.
> >>
> >> *After some further discussions with folks offline, I'd like to revive
> this
> >> discussion. *
> >>
> >> *As Kurt mentioned, to keep it simple I if we can simply build consensus
> >> around what is in for 4.0 and what is out. We can then start the
> process of
> >> working off a 4.0 branch towards betas and release candidates. Again as
> >> Kurt mentioned, assigning a timeline to it right now is difficult, but
> >> having a firm line in the sand around what features/patches are in, then
> >> limiting future 4.0 work to bug fixes will give folks a less nebulous
> >> target to work on. *
> >>
> >> *The other thing to mention is that once we have a 4.0 branch to work
> off,
> >> we at Instaclustr have a commitment to dogfooding the release
> candidates on
> >> our internal staging and internal production workloads before 4.0
> becomes
> >> generally available. I know other folks have similar commitments and
> simply
> >> having a 4.0 branch with a clear list of things that are in or out will
> >> allow everyone to start testing and driving towards a quality release. *
> >>
> >> *The other thing is that there are already a large number of changes
> ready
> >> for 4.0, I would suggest not recommending tickets for 4.0 that have not
> yet
> >> been finished/have outstanding work unless you are the person working
> on it
> >> (or are offering to work on it instead) and can get it ready for review
> in
> >> a timely fashion. That way we can build a more realistic working target.
> >> For other major breaking changes, there is always 5.0 or 4.1 or
> whatever we
> >> end up doing :)*
> >>
> >> Thinking further about it, I would suggest a similar process that was
> >> applied to releasing 3.0, in order to get to 4.0:
> >>
> >>   - Clean up ticket labeling. Move tickets unlikely to make it / be
> worked
> >>   on for 4.0 to something else (e.g. 4.x or whatever).
> >>   - Tickets labeled 4.0 will be the line in the sand, with some trigger
> >>   ("done") event where all features not done by a certain event will
> >> simply
> >>   move into the next release. For the 3.0 branch, this occurred after a
> >>   large review of 8099. For 4.0 it could simply be resolving all current
> >>   blockers/major tickets tagged 4.0... doesn't have to be / nor is it
> >>   something I would strongly advocate.
> >>   - Once we hit this "done" event. Cut a Cassandra-4.0 branch and start
> >>   the alpha/beta/rc cycle from that branch, with only bugfixes going
> into
> >>   it
> >>   - This, in my mind, is similar to the 3.0 approach
> >>   https://mail-archives.apache.org/mod_mbox/cassandra-dev/
> >> 201503.mbox/%3CCALdd-zjAyiTbZksMeq2LxGwLF5LPhoi_
> >> 4vsjy8JBHBRnsxH%3D8A%40mail.gmail.com%3E,
> >>   but without the subsequent tick-tock :)
> >>
> >> There are currently 3 open blockers tagged 4.0, some are old and
> probably
> >> not really blockers anymore, there are other tickets that may/should be
> >> blockers on 4.0:
> >>
> >>   - https://issues.apache.org/jira/browse/CASSANDRA-13951
> >>   - https://issues.apache.org/jira/browse/CASSANDRA-13994
> >>   - https://issues.apache.org/jira/browse/CASSANDRA-12042
> >>
> >> In terms of major tickets that I would like 

Re: Paying off tech debt and correctly naming things

[DuyHai Doan]

+10

There are 2 hard problems in CS: naming things and cache invalidation

Le 20 mars 2018 23:04, "Jon Haddad"  a écrit :

> Whenever I hop around in the codebase, one thing that always manages to
> slow me down is needing to understand the context of the variable names
> that I’m looking at.  We’ve now removed thrift the transport, but the
> variables, classes and comments still remain.  Personally, I’d like to go
> in and pay off as much technical debt as possible by refactoring the code
> to be as close to CQL as possible.  Rows should be rows, not partitions,
> I’d love to see the term column family removed forever in favor of always
> using tables.  That said, it’s a big task.  I did a quick refactor in a
> branch, simply changing the ColumnFamilyStore class to TableStore, and
> pushed it up to GitHub. [1]
>
> Didn’t click on the link?  That’s ok.  The TL;DR is that it’s almost 2K
> LOC changed across 275 files.  I’ll note that my branch doesn’t change any
> of the almost 1000 search results of “columnfamilystore” found in the
> codebase and hundreds of tests failed on my branch in CircleCI, so that 2K
> LOC change would probably be quite a bit bigger.  There is, of course, a
> lot more than just renaming this one class, there’s thousands of variable
> names using any manner of “cf”, “cfs”, “columnfamily”, names plus comments
> and who knows what else.  There’s lots of references in probably every file
> that would have to get updated.
>
> What are people’s thoughts on this?  We should be honest with ourselves
> and know this isn’t going to get any easier over time.  It’s only going to
> get more confusing for new people to the project, and having to figure out
> “what kind of row am i even looking at” is a waste of time.  There’s
> obviously a much bigger impact than just renaming a bunch of files, there’s
> any number of patches and branches that would become outdated, plus anyone
> pulling in Cassandra as a dependency would be affected.  I don’t really
> have a solution for the disruption other than “leave it in place”, but in
> my mind that’s not a great (or even good) solution.
>
> Anyways, enough out of me.  My concern for ergonomics and naming might be
> significantly higher than the rest of the folks working in the code, and I
> wanted to put a feeler out there before I decided to dig into this in a
> more serious manner.
>
> Jon
>
> [1] https://github.com/apache/cassandra/compare/trunk...
> rustyrazorblade:refactor_column_family_store?expand=1 <
> https://github.com/apache/cassandra/compare/trunk...
> rustyrazorblade:refactor_column_family_store?expand=1>

Re: Cassandra Needs to Grow Up by Version Five!

[DuyHai Doan]

So before buying any marketing claims from Microsoft or whoever, maybe
should you try to use it extensively ?

And talking about backup, have a look at DynamoDB:
http://i68.tinypic.com/n1b6yr.jpg

>From my POV, if a multi-billions company like Amazon doesn't get it right
or can't make it easy for end-user (without involving  an unwieldy Hadoop
machinery:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBPipeline.html),
what Cassandra offers in term of back-up restore is more than satisfactory




On Wed, Feb 21, 2018 at 8:56 PM, Kenneth Brotman <
kenbrot...@yahoo.com.invalid> wrote:

>  Josh,
>
> To say nothing is indifference.  If you care about your community,
> sometimes don't you have to bring up a subject even though you know it's
> also temporarily adding some discomfort?
>
> As to opening a JIRA, I've got a very specific topic to try in mind now.
> An easy one I'll work on and then announce.  Someone else will have to do
> the coding.  A year from now I would probably just knock it out to make
> sure it's as easy as I expect it to be but to be honest, as I've been
> saying, I'm not set up to do that right now.  I've barely looked at any
> Cassandra code; for one; everyone on this list probably codes more than I
> do, secondly; and lastly, it's a good one for someone that wants an easy
> one to start with: vNodes.  I've already seen too many people seeking
> assistance with the vNode setting.
>
> And you can expect as others have been mentioning that there should be
> similar ones on compaction, repair and backup.
>
> Microsoft knows poor usability gives them an easy market to take over. And
> they make it easy to switch.
>
> Beginning at 4:17 in the video, it says the following:
>
> "You don't need to worry about replica sets, quorum or read
> repair.  You can focus on writing correct application logic."
>
> At 4:42, it says:
> "Hopefully this gives you a quick idea of how seamlessly you can
> bring your existing Cassandra applications to Azure Cosmos DB.  No code
> changes are required.  It works with your favorite Cassandra tools and
> drivers including for example native Cassandra driver for Spark. And it
> takes seconds to get going, and it's elastically and globally scalable."
>
> More to come,
>
> Kenneth Brotman
>
> -Original Message-
> From: Josh McKenzie [mailto:jmcken...@apache.org]
> Sent: Wednesday, February 21, 2018 8:28 AM
> To: dev@cassandra.apache.org
> Cc: User
> Subject: Re: Cassandra Needs to Grow Up by Version Five!
>
> There's a disheartening amount of "here's where Cassandra is bad, and
> here's what it needs to do for me for free" happening in this thread.
>
> This is open-source software. Everyone is *strongly encouraged* to submit
> a patch to move the needle on *any* of these things being complained about
> in this thread.
>
> For the Apache Way  to
> work, people need to step up and meaningfully contribute to a project to
> scratch their own itch instead of just waiting for a random
> corporation-subsidized engineer to happen to have interests that align with
> them and contribute that to the project.
>
> Beating a dead horse for things everyone on the project knows are serious
> pain points is not productive.
>
> On Wed, Feb 21, 2018 at 5:45 AM, Oleksandr Shulgin <
> oleksandr.shul...@zalando.de> wrote:
>
> > On Mon, Feb 19, 2018 at 10:01 AM, Kenneth Brotman <
> > kenbrot...@yahoo.com.invalid> wrote:
> >
> > >
> > > >> Cluster wide management should be a big theme in any next major
> > release.
> > > >>
> > > >Na. Stability and testing should be a big theme in the next major
> > release.
> > > >
> > >
> > > Double Na on that one Jeff.  I think you have a concern there about
> > > the need to test sufficiently to ensure the stability of the next
> > > major release.  That makes perfect sense.- for every release,
> > > especially the major ones.  Continuous improvement is not a phase of
> > > development for example.  CI should be in everything, in every
> > > phase.  Stability and testing a part of every release not just one.
> > > A major release should be
> > a
> > > nice step from the previous major release though.
> > >
> >
> > I guess what Jeff refers to is the tick-tock release cycle experiment,
> > which has proven to be a complete disaster by popular opinion.
> >
> > There's also the "materialized views" feature which failed to
> > materialize in the end (pun intended) and had to be declared
> > experimental retroactively.
> >
> > Another prominent example is incremental repair which was introduced
> > as the default option in 2.2 and now is not recommended to use because
> > of so many corner cases where it can fail.  So again experimental as an
> afterthought.
> >
> > Not to mention that even if you are aware of the default incremental
> > and go with full repair instead, you're still up for a sad surprise:
> > anti-compaction will be triggered despite the "full" 

Re: Cassandra 2017 Wrapup

[DuyHai Doan]

Thanks Jeff for the very comprehensive list of actions taken this year.
Can't wait to put my hands on 4.0 once it's released



On Fri, Dec 22, 2017 at 10:20 PM, Jeff Jirsa  wrote:

> Happy holidays all,
>
> I imagine most people are about to disappear to celebrate holidays, so I
> wanted to try to summarize the state of Cassandra dev for 2017, as I see
> it. Standard disclaimers apply (this is my personal opinion, not that of my
> employer, not officially endorsed by the Apache Cassandra PMC, or the ASF).
>
> Some quick stats about Cassandra development efforts in 2017 (using
> imperfect git log | awk/sed counting, only looking at trunk, buyer beware,
> it's probably off by a few):
>
> The first commit of 2017 was: Ben Manes, transforming the on-heap cache to
> Caffeine (
> https://github.com/apache/cassandra/commit/c607d76413be81a0e125c5780e068d
> 7ab7594612
> )
> Alex Petrov removed the most code (~7500 lines, according to github)
> Benjamin Lerer added the most code (~8000 lines, according to github)
> We put to bed the tick/tock release cycle, but still cut 14 different
> releases across 5 different branches.
> We had a total of 136 different contributors, with 48 of those contributors
> contributing more than one patch during the year.
> We had a total of 47 different reviewers
> There were 661 non-merge commits to trunk
> There were 56 non-merge commits to docs/
> We end the year with roughly 173 pending changes for 4.0
> We resolved (either fixed or disqualified) 781 issues in JIRA
> I count something like 273 email threads to dev@, and 903 email threads to
> user@
> The project added Stefan Podkowinski, Joel Knighton, Ariel Weisberg, Alex
> Petrov, Blake Eggleston, and Philip Thompson as committers.
> The project added Josh McKenzie, Marcus Eriksson and Jon Haddad to the
> Apache Cassandra PMC
>
> At NGCC (which Eric and Gary managed to organize with the help of
> Instaclustr sponsoring, an achievement in itself), we had people talk
> about:
> - Two different talks (from Apple and FB/Instagram). I'm struggling to
> describe these in simple terms, they both sorta involving using hints and
> changing some of the consistency concepts to help deal with latency /
> durability / availability, especially in cross-DC workloads. Grouping these
> together isn't really fair, but no one-email summary is going to be fair to
> either of these talks. If you missed NGCC, I guess you get to wait for the
> JIRAs / patches.
> - A new storage engine (FB/Instagram) using RocksDB
> - Some notes on using CDC at scale (and some proposed changes to make it
> easier) from Uber (
> https://github.com/ngcc/ngcc2017/blob/master/CassandraDataIngestion.pdf )
> - Michael Shuler (Datastax /  Cassandra PMC / release master / etc) spent
> some time talking about testing and CI.
>
> Some other big'ish development efforts worth mentioning (from personal
> memory, perhaps the worst possible way to create such a list):
> - We spent a fair amount of time talking about testing. Francois @
> Instagram lead the way in codifying a new set of principles around testing
> and quality (
> https://lists.apache.org/thread.html/0854341ae3ab41ceed2ae8a03f2486
> cf2325e4fca6fd800bf4297dd4@%3Cdev.cassandra.apache.org%3E
> / https://issues.apache.org/jira/browse/CASSANDRA-13497 ).
> - We've also spent some time making tests work in CircleCI, which should
> make life much easier for occasional contributors - no need to figure out
> how to run tests in ASF Jenkins.
> - The internode messaging rewrite to use async/netty is probably the single
> largest that comes to mind. It went in earlier this year, and should make
> it easier to have HUGE clusters. All of you running thousand instance
> clusters will probably benefit from this patch (I know you're out there,
> I've talked to you in IRC) - will be in 4.0 (
> https://issues.apache.org/jira/browse/CASSANDRA-8457 )
> - We have a company working on making Cassandra happy with proprietary
> flash storage and PPC64LE (IBM's recent patches,
> https://developer.ibm.com/linuxonpower/2017/03/31/using-
> capi-improve-performance-apache-cassandra-work-progress-update/
> )
> - We have a new commitlog mode added for the first time in quite some time
> - the GroupCommitLog will be in 4.0 (
> https://issues.apache.org/jira/browse/CASSANDRA-13530 )
> - Michael Kjellman spent some time porting dtests from nose to pytest, and
> from python 2.7 to python 3, removing dependencies on dead projects like
> pycassa and the old thrift-cql library. Still needs to be reviewed (
> https://issues.apache.org/jira/browse/CASSANDRA-14134 )
> - Robert Stupp spent some time porting to java9 - again, still need to be
> reviewed ( https://issues.apache.org/jira/browse/CASSANDRA-9608 )
>
> Overall, the state of the project appears to be strong. We're seeing active
> contributions driven primarily by users (like you), the 8099/3.0 engine is
> looking pretty good here in December, and the code base is stabilizing
> towards 

Re: CASSANDRA-8527

[DuyHai Doan]

+1 to report range tombstones. This one is quite tricky indeed to track

+1 to Mockito too, with the reserve that it should be used wisely

On Thu, Dec 21, 2017 at 9:11 PM, Jon Haddad  wrote:

> I had suggested to Alex we kick this discussion over to the ML because the
> change will have a significant impact on the behavior of Cassandra when
> doing reads with range tombstones that cover a lot of rows.  The behavior
> now is a little weird, a single tombstone could shadow hundreds of
> thousands or even millions of rows, and the query would probably just time
> out.  Personally, I’m in favor of the change in behavior of this patch but
> I wanted to get some more feedback before committing to it.  Are there any
> objections to what Alex described?
>
> Regarding Mockito, I’ve been meaning to bring this up for a while, and I’m
> a solid +1 on introducing it to help with testing.  In an ideal world we’d
> have no singletons and could test everything in isolation, but
> realistically that’s a multi year process and we just aren’t there.
>
>
> > On Dec 19, 2017, at 11:07 PM, Alexander Dejanovski <
> a...@thelastpickle.com> wrote:
> >
> > Hi folks,
> >
> > I'm working on CASSANDRA-8527
> >  and would need to
> > discuss a few things.
> >
> > The ticket makes it visible in tracing and metrics that rows shadowed by
> > range tombstones were scanned during reads.
> > Currently, scanned range tombstones aren't reported anywhere which hides
> > the cause of performance issues during reads when the users perform
> primary
> > key deletes.
> > As such, they do not count in the warn and failure thresholds.
> >
> > While the number of live rows and tombstone cells is counted in the
> > ReadCommand class, it is currently not possible to count the number of
> > range tombstones there as they are merged with the rows they shadow
> before
> > reaching the class.
> > Instead, we can count the number of deleted rows that were read , which
> > already improves diagnosis and show that range tombstones were scanned :
> >
> > if (row.hasLiveData(ReadCommand.this.nowInSec(), enforceStrictLiveness))
> >++liveRows;
> > else if (!row.primaryKeyLivenessInfo().isLive(ReadCommand.this.
> nowInSec()))
> > {
> >// We want to detect primary key deletions only.
> >// If all cells have expired they will count as tombstones.
> >   ++deletedRows;
> > }
> >
> > Deleted rows would be part of the warning threshold so that we can spot
> the
> > range tombstone scans in the logs and tracing would look like this :
> >
> > WARN  [ReadStage-2] 2017-12-18 18:22:31,352 ReadCommand.java:491 -
> > Read 2086 live rows, 2440 deleted rows and 0 tombstone cells for
> > query..
> >
> >
> > Are there any caveats to that approach ?
> > Should we include the number of deleted rows in the failure threshold or
> > make it optional, knowing that it could make some queries fail while they
> > were passing before ?
> >
> > On a side note, is it ok to bring in Mockito in order to make it easier
> > writing tests ? I would like to use a Spy in order to write some tests
> > without starting the whole stack.
> >
> > Thanks,
> >
> >
> > --
> > -
> > Alexander Dejanovski
> > France
> > @alexanderdeja
> >
> > Consultant
> > Apache Cassandra Consulting
> > http://www.thelastpickle.com
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Cassandra pluggable storage engine (update)

[DuyHai Doan]

Excellent docs, thanks for the update Dikang.

A question about a design choice, is there any technical reason to specify
the storage engine at keyspace level rather than table level ?

It's not overly complicated to move all tables sharing the same storage
engine into the same keyspace but then it makes tables organization
strongly tied to technical storage engine choice rather than functional
splitting

Regards

On Wed, Oct 4, 2017 at 10:47 PM, Dikang Gu  wrote:

> Hi Blake,
>
> Great questions!
>
> 1. Yeah, we implement the encoding algorithms, which could encode C* data
> types into byte array, and keep the same sorting order. Our implementation
> is based on the orderly lib used in HBase,
> https://github.com/ndimiduk/orderly .
> 2. Repair is not supported yet, we are still working on figure out the work
> need to be done to support repair or incremental repair.
>
> Thanks
> Dikang.
>
> On Wed, Oct 4, 2017 at 1:39 PM, Blake Eggleston 
> wrote:
>
> > Hi Dikang,
> >
> > Cool stuff. 2 questions. Based on your presentation at ngcc, it seems
> like
> > rocks db stores things in byte order. Does this mean that you have code
> > that makes each of the existing types byte comparable, or is clustering
> > order implementation dependent? Also, I don't see anything in the draft
> api
> > that seems to support splitting the data set into arbitrary categories
> (ie
> > repaired and unrepaired data living in the same token range). Is support
> > for incremental repair planned for v1?
> >
> > Thanks,
> >
> > Blake
> >
> >
> > On October 4, 2017 at 1:28:01 PM, Dikang Gu (dikan...@gmail.com) wrote:
> >
> > Hello C* developers:
> >
> > In my previous email (https://www.mail-archive.com/
> > dev@cassandra.apache.org/msg11024.html), I presented that Instagram was
> > kicking off a project to make C*'s storage engine to be pluggable, as
> other
> > modern databases, like mysql, mongoDB etc, so that users will be able to
> > choose most suitable storage engine for different work load, or to use
> > different features. In addition to that, a pluggable storage engine
> > architecture will improve the modularity of the system, help to increase
> > the testability and reliability of Cassandra.
> >
> > After months of development and testing, we'd like to share the work we
> > have done, including the first(draft) version of the C* storage engine
> API,
> > and the first version of the RocksDB based storage engine.
> >
> > ​
> >
> >
> > For the C* storage engine API, here is the draft version we proposed,
> > https://docs.google.com/document/d/1PxYm9oXW2jJtSDiZ-
> > SR9O20jud_0jnA-mW7ttp2dVmk/edit. It contains the APIs for read/write
> > requests, streaming, and table management. The storage engine related
> > functionalities, like data encoding/decoding format, on-disk data
> > read/write, compaction, etc, will be taken care by the storage engine
> > implementation.
> >
> > Each storage engine is a class with each instance of the class is stored
> > in the Keyspace instance. So all the column families within a keyspace
> will
> > share one storage engine instance.
> >
> > Once a storage engine instance is created, Cassandra sever issues
> commands
> > to the engine instance to performance data storage and retrieval tasks
> such
> > as opening a column family, managing column families and streaming.
> >
> > How to config storage engine for different keyspaces? It's still open for
> > discussion. One proposal is that we can add the storage engine option in
> > the create keyspace cql command, and potentially we can overwrite the
> > option per C* node in its config file.
> >
> > Under that API, we implemented a new storage engine, based on RocksDB,
> > called RocksEngine. In long term, we want to support most of C* existing
> > features in RocksEngine, and we want to build it in a progressive manner.
> > For the first version of the RocksDBEngine, we support following
> features:
> > Most of non-nested data types
> > Table schema
> > Point query
> > Range query
> > Mutations
> > Timestamp
> > TTL
> > Deletions/Cell tombstones
> > Streaming
> > We do not supported following features in first version yet:
> > Multi-partition query
> > Nested data types
> > Counters
> > Range tombstone
> > Materialized views
> > Secondary indexes
> > SASI
> > Repair
> > At this moment, we've implemented the V1 features, and deployed it to our
> > shadow cluster. Using shadowing traffic of our production use cases, we
> saw
> > ~3X P99 read latency drop, compared to our C* 2.2 prod clusters. Here are
> > some detailed metrics: https://docs.google.com/
> document/d/1DojHPteDPSphO0_
> > N2meZ3zkmqlidRwwe_cJpsXLcp10.
> >
> > So if you need the features in existing storage engine, please keep using
> > the existing storage engine. If you want to have a more predictable and
> > lower read latency, also the features supported by RocksEngine are enough
> > for your use cases, then RocksEngine could be a fit for 

Re: Proposal to retroactively mark materialized views experimental

[DuyHai Doan]

Ok so IF there is a flag to enable MV (à-la UDA/UDF in cassandra.yaml) then
I'm fine with it. I initially understood that we wanted to disable it
definitively. Maybe we should then add an explicit error message when MV is
disabled and someone tries to use it, something like:

"MV has been disabled, to enable it, turn on the flag  in
cassandra.yaml" so users don't spend 3h searching around


On Mon, Oct 2, 2017 at 9:07 PM, Jon Haddad <j...@jonhaddad.com> wrote:

> There’s a big difference between removal of a protocol that every single
> C* user had to use and disabling a feature which is objectively broken and
> almost nobody is using.  Nobody is talking about removing MVs.  If you want
> to use them you can enable them very trivially, but it should be an
> explicit option because they really aren’t ready for general use.
>
> Claiming disabling by default == removal is not helpful to the
> conversation and is very misleading.
>
> Let’s be practical here.  The people that are most likely to put MVs in
> production right now are people new to Cassandra that don’t know any
> better.  The people that *should* be using MVs are the contributors to the
> project.  People that actually wrote Cassandra code that can do a patch and
> push it into prod, and get it submitted upstream when they fix something.
> Yes, a lot of this stuff requires production usage to shake out the bugs,
> that’s fine, but we shouldn’t lie to people and say “feature X is ready”
> when it’s not.  That’s a great way to get a reputation as “unstable” or
> “not fit for production."
>
> Jon
>
>
> > On Oct 2, 2017, at 11:54 AM, DuyHai Doan <doanduy...@gmail.com> wrote:
> >
> > "I would (in a patch release) disable MV CREATE statements, and emit
> > warnings for ALTER statements and on schema load if they’re not
> explicitly
> > enabled"
> >
> > --> I find this pretty extreme. Now we have an existing feature sitting
> > there in the base code but forbidden from version xxx onward.
> >
> > Since when do we start removing feature in a patch release ? (forbidding
> to
> > create new MV == removing the feature, defacto)
> >
> > Even the Thrift protocol has gone through a long process of deprecation
> and
> > will be removed on 4.0
> >
> > And if we start opening the Pandora box like this, what's next ?
> Forbidding
> > to create SASI index too ? Removing Vnodes ?
> >
> >
> >
> >
> > On Mon, Oct 2, 2017 at 8:16 PM, Jeremiah D Jordan <
> jeremiah.jor...@gmail.com
> >> wrote:
> >
> >>> Only emitting a warning really reduces visibility where we need it: in
> >> the development process.
> >>
> >> How does emitting a native protocol warning reduce visibility during the
> >> development process?  If you run CREATE MV and cqlsh then prints out a
> >> giant warning statement about how it is an experimental feature I think
> >> that is pretty visible during development?
> >>
> >> I guess I can see just blocking new ones without a flag set, but we need
> >> to be careful here.  We need to make sure we don’t cause a problem for
> >> someone that is using them currently, even with all the edge cases
> issues
> >> they have now.
> >>
> >> -Jeremiah
> >>
> >>
> >>> On Oct 2, 2017, at 2:01 PM, Blake Eggleston <beggles...@apple.com>
> >> wrote:
> >>>
> >>> Yeah, I'm not proposing that we disable MVs in existing clusters.
> >>>
> >>>
> >>> On October 2, 2017 at 10:58:11 AM, Aleksey Yeshchenko (
> alek...@apple.com)
> >> wrote:
> >>>
> >>> The idea is to check the flag in CreateViewStatement, so creation of
> new
> >> MVs doesn’t succeed without that flag flipped.
> >>>
> >>> Obviously, just disabling existing MVs working in a minor would be
> silly.
> >>>
> >>> As for the warning - yes, that should also be emitted. Unconditionally.
> >>>
> >>> —
> >>> AY
> >>>
> >>> On 2 October 2017 at 18:18:52, Jeremiah D Jordan (
> >> jeremiah.jor...@gmail.com) wrote:
> >>>
> >>> These things are live on clusters right now, and I would not want
> >> someone to upgrade their cluster to a new *patch* release and suddenly
> >> something that may have been working for them now does not function.
> >> Anyway, we need to be careful about how this gets put into practice if
> we
> >> are going to do it retroactively.
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>
> >>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Proposal to retroactively mark materialized views experimental

[DuyHai Doan]

"I would (in a patch release) disable MV CREATE statements, and emit
warnings for ALTER statements and on schema load if they’re not explicitly
enabled"

--> I find this pretty extreme. Now we have an existing feature sitting
there in the base code but forbidden from version xxx onward.

Since when do we start removing feature in a patch release ? (forbidding to
create new MV == removing the feature, defacto)

Even the Thrift protocol has gone through a long process of deprecation and
will be removed on 4.0

And if we start opening the Pandora box like this, what's next ? Forbidding
to create SASI index too ? Removing Vnodes ?




On Mon, Oct 2, 2017 at 8:16 PM, Jeremiah D Jordan  wrote:

> > Only emitting a warning really reduces visibility where we need it: in
> the development process.
>
> How does emitting a native protocol warning reduce visibility during the
> development process?  If you run CREATE MV and cqlsh then prints out a
> giant warning statement about how it is an experimental feature I think
> that is pretty visible during development?
>
> I guess I can see just blocking new ones without a flag set, but we need
> to be careful here.  We need to make sure we don’t cause a problem for
> someone that is using them currently, even with all the edge cases issues
> they have now.
>
> -Jeremiah
>
>
> > On Oct 2, 2017, at 2:01 PM, Blake Eggleston 
> wrote:
> >
> > Yeah, I'm not proposing that we disable MVs in existing clusters.
> >
> >
> > On October 2, 2017 at 10:58:11 AM, Aleksey Yeshchenko (alek...@apple.com)
> wrote:
> >
> > The idea is to check the flag in CreateViewStatement, so creation of new
> MVs doesn’t succeed without that flag flipped.
> >
> > Obviously, just disabling existing MVs working in a minor would be silly.
> >
> > As for the warning - yes, that should also be emitted. Unconditionally.
> >
> > —
> > AY
> >
> > On 2 October 2017 at 18:18:52, Jeremiah D Jordan (
> jeremiah.jor...@gmail.com) wrote:
> >
> > These things are live on clusters right now, and I would not want
> someone to upgrade their cluster to a new *patch* release and suddenly
> something that may have been working for them now does not function.
> Anyway, we need to be careful about how this gets put into practice if we
> are going to do it retroactively.
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Proposal to retroactively mark materialized views experimental

[DuyHai Doan]

So basically we're saying that even with a lot of tests, you're never sure
to cover all the possible edge cases and the real stamp for "production
readiness" is only when the "experimental features" have been deployed in
various clusters with various scenarios/use-cases, just re-phrasing Blake
here. Totally +1 on the idea.

Now I can foresee a problem with the "experimental" flag, that is nobody
(in the community) will use it or even dare to play with it and thus the
"experimental" features never get a chance to be tested and then we break
the bug-reports/bug-fixes iterations ...

How many times have I seen users on the ML asking which version of C* is
the most fit for production and the answer was always at least 1 major
version behind the current released major (2.1 was recommended when 3.x was
released and so one ...) ?

The fundamental issue here is that a lot of folks in the community do not
want to take any risk and take a conservative approach for the production,
which is fine and perfectly understandable. But it means that the implicit
contract for OSS software, e.g. "you have a software for free in exchange
you will give feedbacks and bug reports to improve it", is completely
broken.

Let's take the example of MV. MV was shipped with 3.0 --> considered not
stable --> nobody/few people uses MV --> few bug reports --> bugs didn't
have chance to get fixed --> the problem lasts until now

About SASI, how many people really played with thoroughly apart from some
toy examples ? Same causes, same consequences. And we can't even blame its
design because fundamentally the architecture is pretty solid, just a
question of usage and feedbacks.

I suspect that this broken community QA/feedback loop did also explain
partially the failure of tic/toc releases but it's only my own
interpretation here.

So if we don't figure out how to restore the "new feature/community bug
report" strong feedback loop, we're going to face again the same issues and
same debate in the future


On Sun, Oct 1, 2017 at 5:30 PM, Blake Eggleston <beggles...@apple.com>
wrote:

> I'm not sure the main issue in the case of MVs is testing. In this case it
> seems to be that there are some design issues and/or the design was only
> works in some overly restrictive use cases. That MVs were committed knowing
> these were issues seems to be the real problem. So in the case of MVs, sure
> I don't think they should have ever made it to an experimental stage.
>
> Thinking of how an experimental flag fits in the with the project going
> forward though, I disagree that we should avoid adding experimental
> features. On the contrary, I think leaning towards classifying new features
> as  experimental would be better for users. Especially larger features and
> changes.
>
> Even with well spec'd, well tested, and well designed features, there will
> always be edge cases that you didn't think of, or you'll have made
> assumptions about the other parts of C* it relies on that aren't 100%
> correct. Small problems here can often affect correctness, or result in
> data loss. So, I think it makes sense to avoid marking them as ready for
> regular use until they've had time to bake in clusters where there are some
> expert operators that are sophisticated enough to understand the
> implications of running them, detect issues, and report bugs.
>
> Regarding historical examples, in hindsight I think committing 8099, or at
> the very least, parts of it, behind an experimental flag would have been
> the right thing to do. It was a huge change that we're still finding issues
> with 2 years later.
>
> On October 1, 2017 at 6:08:50 AM, DuyHai Doan (doanduy...@gmail.com)
> wrote:
>
> How should we transition one feature from the "experimental" state to
> "production ready" state ? On which criteria ?
>
>
>
> On Sun, Oct 1, 2017 at 12:12 PM, Marcus Eriksson <krum...@gmail.com>
> wrote:
>
> > I was just thinking that we should try really hard to avoid adding
> > experimental features - they are experimental due to lack of testing
> right?
> > There should be a clear path to making the feature non-experimental (or
> get
> > it removed) and having that path discussed on dev@ might give more
> > visibility to it.
> >
> > I'm also struggling a bit to find good historic examples of "this would
> > have been better off as an experimental feature" - I used to think that
> it
> > would have been good to commit DTCS with some sort of experimental flag,
> > but that would not have made DTCS any better - it would have been better
> to
> > do more testing, realise that it does not work and then not commit it at
> > all of course.
> >
> > Does anyone have go

Re: Proposal to retroactively mark materialized views experimental

[DuyHai Doan]

How should we transition one feature from the "experimental" state to
"production ready" state ? On which criteria ?



On Sun, Oct 1, 2017 at 12:12 PM, Marcus Eriksson  wrote:

> I was just thinking that we should try really hard to avoid adding
> experimental features - they are experimental due to lack of testing right?
> There should be a clear path to making the feature non-experimental (or get
> it removed) and having that path discussed on dev@ might give more
> visibility to it.
>
> I'm also struggling a bit to find good historic examples of "this would
> have been better off as an experimental feature" - I used to think that it
> would have been good to commit DTCS with some sort of experimental flag,
> but that would not have made DTCS any better - it would have been better to
> do more testing, realise that it does not work and then not commit it at
> all of course.
>
> Does anyone have good examples of features where it would have made sense
> to commit them behind an experimental flag? SASI might be a good example,
> but for MVs - if we knew how painful they would be, they really would not
> have gotten committed at all, right?
>
> /Marcus
>
> On Sat, Sep 30, 2017 at 7:42 AM, Jeff Jirsa  wrote:
>
> > Reviewers should be able to suggest when experimental is warranted, and
> > conversation on dev+jira to justify when it’s transitioned from
> > experimental to stable?
> >
> > We should remove the flag as soon as we’re (collectively) confident in a
> > feature’s behavior - at least correctness, if not performance.
> >
> >
> > > On Sep 29, 2017, at 10:31 PM, Marcus Eriksson 
> wrote:
> > >
> > > +1 on marking MVs experimental, but should there be some point in the
> > > future where we consider removing them from the code base unless they
> > have
> > > gotten significant improvement as well?
> > >
> > > We probably need to enforce some kind of process for adding new
> > > experimental features in the future - perhaps a mail like this one to
> > dev@
> > > motivating why it should be experimental?
> > >
> > > /Marcus
> > >
> > > On Sat, Sep 30, 2017 at 1:15 AM, Vinay Chella
> > 
> > > wrote:
> > >
> > >> We tried perf testing MVs internally here but did not see good results
> > with
> > >> it, hence paused its usage. +1 on tagging certain features which are
> not
> > >> PROD ready or not stable enough.
> > >>
> > >> Regards,
> > >> Vinay Chella
> > >>
> > >>> On Fri, Sep 29, 2017 at 7:22 PM, Ben Bromhead 
> > wrote:
> > >>>
> > >>> I'm a fan of introducing experimental flags in general as well, +1
> > >>>
> > >>>
> > >>>
> >  On Fri, 29 Sep 2017 at 13:22 Jon Haddad  wrote:
> > 
> >  I’m very much +1 on this, and to new features in general.
> > 
> >  I think having a clear line in which we classify something as
> > >> production
> >  ready would be nice.  It would be great if committers were using the
> >  feature in prod and could vouch for it’s stability.
> > 
> > > On Sep 29, 2017, at 1:09 PM, Blake Eggleston  >
> >  wrote:
> > >
> > > Hi dev@,
> > >
> > > I’d like to propose that we retroactively classify materialized
> views
> > >>> as
> >  an experimental feature, disable them by default, and require users
> to
> >  enable them through a config setting before using.
> > >
> > > Materialized views have several issues that make them (effectively)
> >  unusable in production. Some of the issues aren’t just
> implementation
> >  problems, but problems with the design that aren’t easily fixed.
> It’s
> >  unfair of us to make features available to users in this state
> without
> >  providing a clear warning that bad or unexpected things are likely
> to
> >  happen if they use it.
> > >
> > > Obviously, this isn’t great news for users that have already
> adopted
> >  MVs, and I don’t have a great answer for that. I think that’s sort
> of
> > a
> >  sunk cost at this point. If they have any MV related problems,
> they’ll
> > >>> have
> >  them whether they’re marked experimental or not. I would expect this
> > to
> >  reduce the number of users adopting MVs in the future though, and if
> > >> they
> >  do, it would be opt-in.
> > >
> > > Once MVs reach a point where they’re usable in production, we can
> > >>> remove
> >  the flag. Specifics of how the experimental flag would work can be
> > >>> hammered
> >  out in a forthcoming JIRA, but I’d imagine it would just prevent
> users
> > >>> from
> >  creating new MVs, and maybe log warnings on startup for existing MVs
> > if
> > >>> the
> >  flag isn’t enabled.
> > >
> > > Let me know what you think.
> > >
> > > Thanks,
> > >
> > > Blake
> > 
> > 
> >  
> -
> > 

Re: Does partition size limitation still exists in Cassandra 3.10 given there is a B-tree implementation?

[DuyHai Doan]

Yes the recommendation still applies

Wide partitions have huge impact on repair (over streaming), compaction and
bootstrap

Le 10 mai 2017 23:54, "Kant Kodali"  a écrit :

Hi All,

Cassandra community had always been recommending 100MB per partition as a
sweet spot however does this limitation still exist given there is a B-tree
implementation to identify rows inside a partition?

https://github.com/apache/cassandra/blob/trunk/src/java/
org/apache/cassandra/db/rows/BTreeRow.java

Thanks!

Re: Cassandra on RocksDB experiment result

[DuyHai Doan]

"I have no clue what it would take to accomplish a pluggable storage
engine, but I love this idea."

This was a long and old debate we had several times in the past. One of the
difficulty of pluggable storage engine is that we need to manage the
differences between the LSMT of native C* and RockDB engine for compaction,
repair, streaming etc...

Right now all the compaction strategies share the assumption that the data
structure and layout on disk is fixed. With pluggable storage engine, we
need to special case each compaction strategy (or at least the Abstract
class of compaction strategy) for each engine.

The current approach is one storage engine, many compaction strategies for
different use-cases (TWCS for time series, LCS for heavy update...).

With pluggable storage engine, we'll have a matrix of storage engine x
compaction strategies.

And not even mentioning the other operations to handle like streaming and
repair.

Another question that arose is: will the storage engine be run in the same
JVM as the C* server or is it a separate process ? For the later, we're
opening the door to yet-another-distributed-system complexity. For
instance, how the C* JVM will communicate with the storage engine process ?
How to handle failure, crash, resume etc ...

That being said, if we manage to get the code base to this stage eventually
it'd be super cool !

On Wed, Apr 19, 2017 at 12:03 PM, Salih Gedik  wrote:

> Hi Dikang,
>
> I guess there is something wrong with the link that you shared.
>
>
> 19.04.2017 19:21 tarihinde Dikang Gu yazdı:
>
> Hi Cassandra developers,
>>
>> This is Dikang from Instagram, I'd like to share you some experiment
>> results we did recently, to use RocksDB as Cassandra's storage engine. In
>> the experiment, I built a prototype to integrate Cassandra 3.0.12 and
>> RocksDB on single column (key-value) use case, shadowed one of our
>> production use case, and saw about 4-6X P99 read latency drop during peak
>> time, compared to 3.0.12. Also, the P99 latency became more predictable as
>> well.
>>
>> Here is detailed note with more metrics:
>>
>> https://docs.google.com/document/d/1Ztqcu8Jzh4USKoWBgDJQw82DBurQm
>> sV-PmfiJYvu_Dc/edit?usp=sharing
>>
>> Please take a look and let me know your thoughts. I think the biggest
>> latency win comes from we get rid of most Java garbages created by current
>> read/write path and compactions, which reduces the JVM overhead and makes
>> the latency to be more predictable.
>>
>> We are very excited about the potential performance gain. As the next
>> step,
>> I propose to make the Cassandra storage engine to be pluggable (like Mysql
>> and MongoDB), and we are very interested in providing RocksDB as one
>> storage option with more predictable performance, together with community.
>>
>> Thanks.
>>
>>
>

Re: Code quality, principles and rules

[DuyHai Doan]

"Otherwise it'll be difficult to write unit test cases."

Having to pull in dependency injection framework to make unit testing
easier is generally a sign of code design issue.

With constructor injection & other techniques, there is more than enough to
unit test your code without having to pull external libs

On Thu, Mar 16, 2017 at 10:18 PM, Jason Brown  wrote:

> >> do we have plan to integrate with a dependency injection framework?
>
> No, we (the maintainers) have been pretty much against more frameworks due
> to performance reasons, overhead, and dependency management problems.
>
> On Thu, Mar 16, 2017 at 2:04 PM, Qingcun Zhou 
> wrote:
>
> > Since we're here, do we have plan to integrate with a dependency
> injection
> > framework like Dagger2? Otherwise it'll be difficult to write unit test
> > cases.
> >
> > On Thu, Mar 16, 2017 at 1:16 PM, Edward Capriolo 
> > wrote:
> >
> > > On Thu, Mar 16, 2017 at 3:10 PM, Jeff Jirsa  wrote:
> > >
> > > >
> > > >
> > > > On 2017-03-16 10:32 (-0700), François Deliège <
> franc...@instagram.com>
> > > > wrote:
> > > > >
> > > > > To get this started, here is an initial proposal:
> > > > >
> > > > > Principles:
> > > > >
> > > > > 1. Tests always pass.  This is the starting point. If we don't care
> > > > about test failures, then we should stop writing tests. A recurring
> > > failing
> > > > test carries no signal and is better deleted.
> > > > > 2. The code is tested.
> > > > >
> > > > > Assuming we can align on these principles, here is a proposal for
> > their
> > > > implementation.
> > > > >
> > > > > Rules:
> > > > >
> > > > > 1. Each new release passes all tests (no flakinesss).
> > > > > 2. If a patch has a failing test (test touching the same code
> path),
> > > the
> > > > code or test should be fixed prior to being accepted.
> > > > > 3. Bugs fixes should have one test that fails prior to the fix and
> > > > passes after fix.
> > > > > 4. New code should have at least 90% test coverage.
> > > > >
> > > > First I was
> > > > I agree with all of these and hope they become codified and
> followed. I
> > > > don't know anyone who believes we should be committing code that
> breaks
> > > > tests - but we should be more strict with requiring green test runs,
> > and
> > > > perhaps more strict with reverting patches that break tests (or cause
> > > them
> > > > to be flakey).
> > > >
> > > > Ed also noted on the user list [0] that certain sections of the code
> > > > itself are difficult to test because of singletons - I agree with the
> > > > suggestion that it's time to revisit CASSANDRA-7837 and
> CASSANDRA-10283
> > > >
> > > > Finally, we should also recall Jason's previous notes [1] that the
> > actual
> > > > test infrastructure available is limited - the system provided by
> > > Datastax
> > > > is not generally open to everyone (and not guaranteed to be
> permanent),
> > > and
> > > > the infrastructure currently available to the ASF is somewhat limited
> > > (much
> > > > slower, at the very least). If we require tests passing (and I agree
> > that
> > > > we should), we need to define how we're going to be testing (or how
> > we're
> > > > going to be sharing test results), because the ASF hardware isn't
> going
> > > to
> > > > be able to do dozens of dev branch dtest runs per day in its current
> > > form.
> > > >
> > > > 0: https://lists.apache.org/thread.html/
> f6f3fc6d0ad1bd54a6185ce7bd7a2f
> > > > 6f09759a02352ffc05df92eef6@%3Cuser.cassandra.apache.org%3E
> > > > 1: https://lists.apache.org/thread.html/
> 5fb8f0446ab97644100e4ef987f36e
> > > > 07f44e8dd6d38f5dc81ecb3cdd@%3Cdev.cassandra.apache.org%3E
> > > >
> > > >
> > > >
> > > Ed also noted on the user list [0] that certain sections of the code
> > itself
> > > are difficult to test because of singletons - I agree with the
> suggestion
> > > that it's time to revisit CASSANDRA-7837 and CASSANDRA-10283
> > >
> > > Thanks for the shout out!
> > >
> > > I was just looking at a patch about compaction. The patch was to
> > calculate
> > > free space correctly in case X. Compaction is not something that
> requires
> > > multiple nodes to test. The logic on the surface seems simple: find
> > tables
> > > of similar size and select them and merge them. The reality is it turns
> > out
> > > now to be that way. The coverage itself both branch and line may be
> very
> > > high, but what the code does not do is directly account for a wide
> > variety
> > > of scenarios. Without direct tests you end up with a mental
> approximation
> > > of what it does and that varies from person to person and accounts for
> > the
> > > cases that fit in your mind. For example, you personally are only
> running
> > > LevelDB inspired compaction.
> > >
> > > Being that this this is not a multi-node problem you should be able to
> re
> > > factor this heavily. Pulling everything out to a static method where
> all
> > > the 

Re: Hopefully Weekly Apache Cassandra JIRA Topics of Interest

[DuyHai Doan]

Static columns can already be indexed by custom 2nd index impl, for example
SASI : https://issues.apache.org/jira/browse/CASSANDRA-11183


On Sun, Mar 12, 2017 at 10:40 PM, Jeff Jirsa  wrote:

> Hi folks,
>
> Cassandra JIRA is huge, active, and ever-changing. It's easy to miss
> tickets you may care about, or if you want to start contributing, sometimes
> it's hard to know where to start. I'm going to make an attempt to pick a
> few dozen JIRAs each week (or month?) that would benefit from some more
> eyeballs - if you have any tickets that you feel haven't gotten the
> attention they deserve, feel free to add them to the list.
>
>
> Topics for discussion (not necessarily a "go write a patch" type ticket,
> but things that could use interested parties discussing options):
>
> https://issues.apache.org/jira/browse/CASSANDRA-12944 - Internal
> diagnostic
> events. Pretty involved proposal attached.
> https://issues.apache.org/jira/browse/CASSANDRA-12912 - Warnings when the
> superuser logs in
> https://issues.apache.org/jira/browse/CASSANDRA-9608 - Get Cassandra ready
> for java9 - if anyone has time to help find issues with jdk9, the oracle
> team asked for feedback some time ago, I'm not sure if they're still
> interested, but it'd be good for us to identify issues anyway.
> https://issues.apache.org/jira/browse/CASSANDRA-13315 - talks about using
> more meaningful names for consistency levels (as aliases)
> https://issues.apache.org/jira/browse/CASSANDRA-13241 - talks about
> dropping default chunk size for better IO performance (may have a blocking
> subtask of making compression info more efficient before we do it, but you
> may still be interested in the discussion)
>
> Patches that could use a reviewer:
>
> LHF (MAY be suitable for someone who is a solid programmer, even if they're
> not super familiar with Cassandra internals - even if you're not a regular
> contributor, you can review and +1 if you're confident that the fix is
> correct and has been reasonably tested - see the 'How to review' checklist
> http://cassandra.apache.org/doc/latest/development/how_to_review.html ):
> https://issues.apache.org/jira/browse/CASSANDRA-13307 - Making CQLSH
> downgrade protocol version automatically
> https://issues.apache.org/jira/browse/CASSANDRA-13067 - AWS EFS fixes
> (presents as huge filesystem, overflows internal variables)
> https://issues.apache.org/jira/browse/CASSANDRA-13263 - Incorrect
> ComplexColumnData hash code implementation
> https://issues.apache.org/jira/browse/CASSANDRA-12719 - Typo in docs
> examples
>
> Proposed New features without a reviewer:
> https://issues.apache.org/jira/browse/CASSANDRA-13269 - Snapshot support
> for secondary index
> https://issues.apache.org/jira/browse/CASSANDRA-13270 - Add function hooks
> to deliver Elasticsearch as a Cassandra plugin
> https://issues.apache.org/jira/browse/CASSANDRA-13304 - Add checksumming
> to
> the native protocol
> https://issues.apache.org/jira/browse/CASSANDRA-13268 - Add 2i to static
> columns
>
> Enhancements without a reviewer:
> https://issues.apache.org/jira/browse/CASSANDRA-9989 - Optimize BTree
> builder
> https://issues.apache.org/jira/browse/CASSANDRA-9988 - BTree Leaf-only
> iterator
> https://issues.apache.org/jira/browse/CASSANDRA-12837 - Multithreaded
> support for rebuild_index
>
> Bugs without a reviewer:
> https://issues.apache.org/jira/browse/CASSANDRA-13323 -
> IncomingTcpConnection closed on single bad message
> https://issues.apache.org/jira/browse/CASSANDRA-13020 - Hint delivery
> fails
> when prefer_local enabled
>
> The new "official" docs sections that could use some contributors! This may
> be the lowest barrier to entry if you want to get involved - the docs are
> kept in source control ( https://github.com/apache/
> cassandra/tree/trunk/doc
> ) , so we can change them with patches against those files submitted
> through JIRA. No github pull requests, please, just submit a patch on JIRA
> (see https://issues.apache.org/jira/browse/CASSANDRA-13160 or
> https://issues.apache.org/jira/browse/CASSANDRA-12449 as examples):
>
> Some examples that are just stubbed out:
>
> http://cassandra.apache.org/doc/latest/architecture/index.html
> http://cassandra.apache.org/doc/latest/architecture/dynamo.html
> http://cassandra.apache.org/doc/latest/architecture/storage_engine.html
> http://cassandra.apache.org/doc/latest/data_modeling/index.html
> http://cassandra.apache.org/doc/latest/operating/read_repair.html
> http://cassandra.apache.org/doc/latest/operating/repair.html
> http://cassandra.apache.org/doc/latest/operating/hints.html
> http://cassandra.apache.org/doc/latest/operating/backups.html
> http://cassandra.apache.org/doc/latest/operating/bulk_loading.html
>
>
> I'll try to do this regularly. Again, if you have any tickets you feel need
> attention, please reply!
>
> - Jeff
>

Re: committing performance patches without performance numbers

[DuyHai Doan]

After looking at the patch, my thoughts (beware, it's getting very
technical):


Original code:

-t = new ListType(elements, isMultiCell);
-ListType t2 = internMap.putIfAbsent(elements, t);
-t = (t2 == null) ? t : t2;


Optimized code:

+t = internMap.computeIfAbsent(elements, K -> new
ListType<>(K, isMultiCell) );


This is a very obvious and *classical* optimization. In the original code,
we create an instance of ListType() but it is ONLY usefull if the interMap
doesn't have any value (putIfAbsent). Meaning that the creation of the
ListType object is useless if the map already contains a value.

The optimized code use the computeIfAbsent with a lambda (K -> new
ListType...) which will be evaluated iff the map doesn't contains the key

 For a high performance project like Cassandra, avoiding to create
un-necessary objects to help keeping the heap size small is always a good
idea.

 And this kind of optimization using lambda for lazy evaluation is quite
common (see a PR here on my Achilles project:
https://github.com/doanduyhai/Achilles/commit/9b9c09aa26d0edb8c58222b1ad069327b6f40da3
)

My guess is that the perf improvement and the gain is so obvious for Java
people that the JIRA creator thought a perf benchmark is not necessary

On Thu, Mar 9, 2017 at 9:00 PM, Jonathan Haddad  wrote:

> I'd like to discuss what I consider to be a pretty important matter -
> patches which are written for the sole purpose of improving performance
> without including a single performance benchmark in the JIRA.
>
> My original email was in "Testing and Jira Tickets", i'll copy it here
> for posterity:
>
> If you don't mind, I'd like to broaden the discussion a little bit to also
> discuss performance related patches.  For instance, CASSANDRA-13271 was a
> performance / optimization related patch that included *zero* information
> on if there was any perf improvement or a regression as a result of the
> change, even though I've asked twice for that information.
>
> In addition to "does this thing break anything" we should be asking "how
> does this patch affect performance?" (and were the appropriate docs
> included, but that's another topic altogether)
>
> There's a minor note about perf related stuff here:
> http://cassandra.apache.org/doc/latest/development/
> testing.html#performance-testing
>
>
> "Performance tests for Cassandra are a special breed of tests that are not
> part of the usual patch contribution process. In fact you can contribute
> tons of patches to Cassandra without ever running performance tests. They
> are important however when working on performance improvements, as such
> improvements must be measurable."
>
> I think performance numbers aren't just important, but should be a
> *requirement* to merge a performance patch.
>
> Thoughts?
> Jon
>

Re: State of triggers

[DuyHai Doan]

No problem, distributed systems are hard to reason about, I got caught many
times in the past

On Sun, Mar 5, 2017 at 9:23 AM, benjamin roth <brs...@gmail.com> wrote:

> Sorry. Answer was to fast. Maybe you are right.
>
> Am 05.03.2017 09:21 schrieb "benjamin roth" <brs...@gmail.com>:
>
> > No. You just change the partitioner. That's all
> >
> > Am 05.03.2017 09:15 schrieb "DuyHai Doan" <doanduy...@gmail.com>:
> >
> >> "How can that be achieved? I haven't done "scientific researches" yet
> but
> >> I
> >> guess a "MV partitioner" could do the trick. Instead of applying the
> >> regular partitioner, an MV partitioner would calculate the PK of the
> base
> >> table (which is always possible) and then apply the regular
> partitioner."
> >>
> >> The main purpose of MV is to avoid the drawbacks of 2nd index
> >> architecture,
> >> e.g. to scan a lot of nodes to fetch the results.
> >>
> >> With MV, since you give the partition key, the guarantee is that you'll
> >> hit
> >> a single node.
> >>
> >> Now if you put MV data on the same node as base table data, you're doing
> >> more-or-less the same thing as 2nd index.
> >>
> >> Let's take a dead simple example
> >>
> >> CREATE TABLE user (user_id uuid PRIMARY KEY, email text);
> >> CREATE MV user_by_email AS SELECT * FROM user WHERE user_id IS NOT NULL
> >> AND
> >> email IS NOT NULL PRIMARY KEY((email),user_id);
> >>
> >> SELECT * FROM user_by_email WHERE email = xxx;
> >>
> >> With this query, how can you find the user_id that corresponds to email
> >> 'xxx' so that your MV partitioner idea can work ?
> >>
> >>
> >>
> >> On Sun, Mar 5, 2017 at 9:05 AM, benjamin roth <brs...@gmail.com> wrote:
> >>
> >> > While I was reading the MV paragraph in your post, an idea popped up:
> >> >
> >> > The problem with MV inconsistencies and inconsistent range movement is
> >> that
> >> > the "MV contract" is broken. This only happens because base data and
> >> > replica data reside on different hosts. If base data + replicas would
> >> stay
> >> > on the same host then a rebuild/remove would always stream both
> matching
> >> > parts of a base table + mv.
> >> >
> >> > So my idea:
> >> > Why not make a replica ALWAYS stay local regardless where the token of
> >> a MV
> >> > would point at. That would solve these problems:
> >> > 1. Rebuild / remove node would not break MV contract
> >> > 2. A write always stays local:
> >> >
> >> > a) That means replication happens sync. That means a quorum write to
> the
> >> > base table guarantees instant data availability with quorum read on a
> >> view
> >> >
> >> > b) It saves network roundtrips + request/response handling and helps
> to
> >> > keep a cluster healthier in case of bulk operations (like repair
> >> streams or
> >> > rebuild stream). Write load stays local and is not spread across the
> >> whole
> >> > cluster. I think it makes the load in these situations more
> predictable.
> >> >
> >> > How can that be achieved? I haven't done "scientific researches" yet
> >> but I
> >> > guess a "MV partitioner" could do the trick. Instead of applying the
> >> > regular partitioner, an MV partitioner would calculate the PK of the
> >> base
> >> > table (which is always possible) and then apply the regular
> partitioner.
> >> >
> >> > I'll create a proper Jira for it on monday. Currently it's sunday here
> >> and
> >> > my family wants me back so just a few thoughts on this right now.
> >> >
> >> > Any feedback is appreciated!
> >> >
> >> > 2017-03-05 6:34 GMT+01:00 Edward Capriolo <edlinuxg...@gmail.com>:
> >> >
> >> > > On Sat, Mar 4, 2017 at 10:26 AM, Jeff Jirsa <jji...@gmail.com>
> wrote:
> >> > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > > On Mar 4, 2017, at 7:06 AM, Edward Capriolo <
> >> edlinuxg...@gmail.com>
> >> > > > wrote:
> >> > > > >
> >> > > > >> On Fri, Mar 3, 2017 at 12:04 PM, Jeff

Re: State of triggers

[DuyHai Doan]

"How can that be achieved? I haven't done "scientific researches" yet but I
guess a "MV partitioner" could do the trick. Instead of applying the
regular partitioner, an MV partitioner would calculate the PK of the base
table (which is always possible) and then apply the regular partitioner."

The main purpose of MV is to avoid the drawbacks of 2nd index architecture,
e.g. to scan a lot of nodes to fetch the results.

With MV, since you give the partition key, the guarantee is that you'll hit
a single node.

Now if you put MV data on the same node as base table data, you're doing
more-or-less the same thing as 2nd index.

Let's take a dead simple example

CREATE TABLE user (user_id uuid PRIMARY KEY, email text);
CREATE MV user_by_email AS SELECT * FROM user WHERE user_id IS NOT NULL AND
email IS NOT NULL PRIMARY KEY((email),user_id);

SELECT * FROM user_by_email WHERE email = xxx;

With this query, how can you find the user_id that corresponds to email
'xxx' so that your MV partitioner idea can work ?



On Sun, Mar 5, 2017 at 9:05 AM, benjamin roth  wrote:

> While I was reading the MV paragraph in your post, an idea popped up:
>
> The problem with MV inconsistencies and inconsistent range movement is that
> the "MV contract" is broken. This only happens because base data and
> replica data reside on different hosts. If base data + replicas would stay
> on the same host then a rebuild/remove would always stream both matching
> parts of a base table + mv.
>
> So my idea:
> Why not make a replica ALWAYS stay local regardless where the token of a MV
> would point at. That would solve these problems:
> 1. Rebuild / remove node would not break MV contract
> 2. A write always stays local:
>
> a) That means replication happens sync. That means a quorum write to the
> base table guarantees instant data availability with quorum read on a view
>
> b) It saves network roundtrips + request/response handling and helps to
> keep a cluster healthier in case of bulk operations (like repair streams or
> rebuild stream). Write load stays local and is not spread across the whole
> cluster. I think it makes the load in these situations more predictable.
>
> How can that be achieved? I haven't done "scientific researches" yet but I
> guess a "MV partitioner" could do the trick. Instead of applying the
> regular partitioner, an MV partitioner would calculate the PK of the base
> table (which is always possible) and then apply the regular partitioner.
>
> I'll create a proper Jira for it on monday. Currently it's sunday here and
> my family wants me back so just a few thoughts on this right now.
>
> Any feedback is appreciated!
>
> 2017-03-05 6:34 GMT+01:00 Edward Capriolo :
>
> > On Sat, Mar 4, 2017 at 10:26 AM, Jeff Jirsa  wrote:
> >
> > >
> > >
> > >
> > > > On Mar 4, 2017, at 7:06 AM, Edward Capriolo 
> > > wrote:
> > > >
> > > >> On Fri, Mar 3, 2017 at 12:04 PM, Jeff Jirsa 
> wrote:
> > > >>
> > > >> On Fri, Mar 3, 2017 at 5:40 AM, Edward Capriolo <
> > edlinuxg...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>>
> > > >>> I used them. I built do it yourself secondary indexes with them.
> They
> > > >> have
> > > >>> there gotchas, but so do all the secondary index implementations.
> > Just
> > > >>> because datastax does not write about something. Lets see like 5
> > years
> > > >> ago
> > > >>> there was this: https://github.com/hmsonline/cassandra-triggers
> > > >>>
> > > >>>
> > > >> Still in use? How'd it work? Production ready? Would you still do it
> > > that
> > > >> way in 2017?
> > > >>
> > > >>
> > > >>> There is a fairly large divergence to what actual users do and what
> > > other
> > > >>> groups 'say' actual users do in some cases.
> > > >>>
> > > >>
> > > >> A lot of people don't share what they're doing (for business
> reasons,
> > or
> > > >> because they don't think it's important, or because they don't know
> > > >> how/where), and that's fine but it makes it hard for anyone to know
> > what
> > > >> features are used, or how well they're really working in production.
> > > >>
> > > >> I've seen a handful of "how do we use triggers" questions in IRC,
> and
> > > they
> > > >> weren't unreasonable questions, but seemed like a lot of pain, and
> > more
> > > >> than one of those people ultimately came back and said they used
> some
> > > other
> > > >> mechanism (and of course, some of them silently disappear, so we
> have
> > no
> > > >> idea if it worked or not).
> > > >>
> > > >> If anyone's actively using triggers, please don't keep it a secret.
> > > Knowing
> > > >> that they're being used would be a great way to justify continuing
> to
> > > >> maintain them.
> > > >>
> > > >> - Jeff
> > > >>
> > > >
> > > > "Still in use? How'd it work? Production ready? Would you still do it
> > > that way in 2017?"
> > > >
> > > > I mean that is a loaded question. How long has cassandra had
> Secondary
> > > > 

Re: If reading from materialized view with a consistency level of quorum am I guaranteed to have the most recent view?

[DuyHai Doan]

See my blog post to understand how MV is implemented:
http://www.doanduyhai.com/blog/?p=1930

On Fri, Feb 10, 2017 at 7:48 PM, Benjamin Roth 
wrote:

> Same partition key:
>
> PRIMARY KEY ((a, b), c, d) and
> PRIMARY KEY ((a, b), d, c)
>
> PRIMARY KEY ((a), b, c) and
> PRIMARY KEY ((a), c, b)
>
> Different partition key:
>
> PRIMARY KEY ((a, b), c, d) and
> PRIMARY KEY ((a), b, d, c)
>
> PRIMARY KEY ((a), b) and
> PRIMARY KEY ((b), a)
>
>
> 2017-02-10 19:46 GMT+01:00 Kant Kodali :
>
> > Okies now I understand what you mean by "same" partition key.  I think
> you
> > are saying
> >
> > PRIMARY KEY(col1, col2, col3) == PRIMARY KEY(col2, col1, col3) // so far
> I
> > assumed they are different partition keys.
> >
> > On Fri, Feb 10, 2017 at 10:36 AM, Benjamin Roth  >
> > wrote:
> >
> > > There are use cases where the partition key is the same. For example if
> > you
> > > need a sorting within a partition or a filtering different from the
> > > original clustering keys.
> > > We actually use this for some MVs.
> > >
> > > If you want "dumb" denormalization with simple append only cases (or
> more
> > > general cases that don't require a read before write on update) you are
> > > maybe better off with batched denormalized atomics writes.
> > >
> > > The main benefit of MVs is if you need denormalization to sort or
> filter
> > by
> > > a non-primary key field.
> > >
> > > 2017-02-10 19:31 GMT+01:00 Kant Kodali :
> > >
> > > > yes thanks for the clarification.  But why would I ever have MV with
> > the
> > > > same partition key? if it is the same partition key I could just read
> > > from
> > > > the base table right? our MV Partition key contains the columns from
> > the
> > > > base table partition key but in a different order plus an additional
> > > column
> > > > (which is allowed as of today)
> > > >
> > > > On Fri, Feb 10, 2017 at 10:23 AM, Benjamin Roth <
> > benjamin.r...@jaumo.com
> > > >
> > > > wrote:
> > > >
> > > > > It depends on your model.
> > > > > If the base table + MV have the same partition key, then the MV
> > > mutations
> > > > > are applied synchronously, so they are written as soon the write
> > > request
> > > > > returns.
> > > > > => In this case you can rely on the R+F > RF
> > > > >
> > > > > If the partition key of the MV is different, the partition of the
> MV
> > is
> > > > > probably placed on a different host (or said differently it cannot
> be
> > > > > guaranteed that it is on the same host). In this case, the MV
> updates
> > > are
> > > > > executed async in a logged batch. So it can be guaranteed they will
> > be
> > > > > applied eventually but not at the time the write request returns.
> > > > > => You cannot rely and there is no possibility to absolutely
> > guarantee
> > > > > anything, not matter what CL you choose. A MV update may always
> > "arrive
> > > > > late". I guess it has been implemented like this to not block in
> case
> > > of
> > > > > remote request to prefer the cluster sanity over consistency.
> > > > >
> > > > > Is it now 100% clear?
> > > > >
> > > > > 2017-02-10 19:17 GMT+01:00 Kant Kodali :
> > > > >
> > > > > > So R+W > RF doesnt apply for reads on MV right because say I set
> > > QUORUM
> > > > > > level consistency for both reads and writes then there can be a
> > > > scenario
> > > > > > where a write is successful to the base table and then say
> > > immediately
> > > > I
> > > > > do
> > > > > > a read through MV but prior to MV getting the update from the
> base
> > > > table.
> > > > > > so there isn't any way to make sure to read after MV had been
> > > > > successfully
> > > > > > updated. is that correct?
> > > > > >
> > > > > > On Fri, Feb 10, 2017 at 6:30 AM, Benjamin Roth <
> > > > benjamin.r...@jaumo.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Kant
> > > > > > >
> > > > > > > Is it clear now?
> > > > > > > Sorry for the confusion!
> > > > > > >
> > > > > > > Have a nice one
> > > > > > >
> > > > > > > Am 10.02.2017 09:17 schrieb "Kant Kodali" :
> > > > > > >
> > > > > > > thanks!
> > > > > > >
> > > > > > > On Thu, Feb 9, 2017 at 8:51 PM, Benjamin Roth <
> > > > benjamin.r...@jaumo.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Yes it is
> > > > > > > >
> > > > > > > > Am 10.02.2017 00:46 schrieb "Kant Kodali"  >:
> > > > > > > >
> > > > > > > > > If reading from materialized view with a consistency level
> of
> > > > > quorum
> > > > > > am
> > > > > > > I
> > > > > > > > > guaranteed to have the most recent view? other words is w
> + r
> > > > n
> > > > > > > > contract
> > > > > > > > > maintained for MV's as well for both reads and writes?
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Benjamin Roth
> > > > > Prokurist
> > > > >
> > > > > 

Re: Why does CockroachDB github website say Cassandra has no Availability on datacenter failure?

[DuyHai Doan]

The link you posted doesn't say anything about Cassandra
Le 7 févr. 2017 11:41, "Kant Kodali"  a écrit :

> Why does CockroachDB github website say Cassandra has no Availability on
> datacenter failure?
>
> https://github.com/cockroachdb/cockroach
>

Re: Rough roadmap for 4.0

[DuyHai Doan]

Be very careful, there is a serious bug about AND/OR semantics, not solved
yet and not going to be solved any soon:
https://issues.apache.org/jira/browse/CASSANDRA-12674

On Thu, Nov 17, 2016 at 7:32 PM, Jeff Jirsa 
wrote:

>
> We’ll be voting in the very near future on timing of major releases and
> release strategy. 4.0 won’t happen until that vote takes place.
>
> But since you asked, I have ONE tick/tock (3.9) cluster being qualified
> for production because it needs SASI.
>
> - Jeff
>
> On 11/17/16, 9:59 AM, "Jonathan Haddad"  wrote:
>
> >I think it might be worth considering adopting the release strategy before
> >4.0 release.  Are any PMC members putting tick tock in prod? Does anyone
> >even trust it?  What's the downside of changing the release cycle
> >independently from 4.0?
> >
> >On Thu, Nov 17, 2016 at 9:03 AM Jason Brown  wrote:
> >
> >Jason,
> >
> >That's a separate topic, but we will have a different vote on how the
> >branching/release strategy should be for the future.
> >
> >On Thursday, November 17, 2016, jason zhao yang <
> zhaoyangsingap...@gmail.com
> >>
> >wrote:
> >
> >> Hi,
> >>
> >> Will we still use tick-tock release for 4.x and 4.0.x ?
> >>
> >> Stefan Podkowinski >于2016年11月16日周三
> >> 下午4:52写道：
> >>
> >> > From my understanding, this will also effect EOL dates of other
> >branches.
> >> >
> >> > "We will maintain the 2.2 stability series until 4.0 is released, and
> >3.0
> >> > for six months after that.".
> >> >
> >> >
> >> > On Wed, Nov 16, 2016 at 5:34 AM, Nate McCall  >> > wrote:
> >> >
> >> > > Agreed. As long as we have a goal I don't see why we have to adhere
> to
> >> > > arbitrary date for 4.0.
> >> > >
> >> > > On Nov 16, 2016 1:45 PM, "Aleksey Yeschenko"  >> >
> >> > wrote:
> >> > >
> >> > > > I’ll comment on the broader issue, but right now I want to
> elaborate
> >> on
> >> > > > 3.11/January/arbitrary cutoff date.
> >> > > >
> >> > > > Doesn’t matter what the original plan was. We should continue with
> >> 3.X
> >> > > > until all the 4.0 blockers have been
> >> > > > committed - and there are quite a few of them remaining yet.
> >> > > >
> >> > > > So given all the holidays, and the tickets remaining, I’ll
> >personally
> >> > be
> >> > > > surprised if 4.0 comes out before
> >> > > > February/March and 3.13/3.14. Nor do I think it’s an issue.
> >> > > >
> >> > > > —
> >> > > > AY
> >> > > >
> >> > > > On 16 November 2016 at 00:39:03, Mick Semb Wever (
> >> > m...@thelastpickle.com 
> >> > > )
> >> > > > wrote:
> >> > > >
> >> > > > On 4 November 2016 at 13:47, Nate McCall  >> > wrote:
> >> > > >
> >> > > > > Specifically, this should be "new stuff that could/will break
> >> things"
> >> > > > > given we are upping
> >> > > > > the major version.
> >> > > > >
> >> > > >
> >> > > >
> >> > > > How does this co-ordinate with the tick-tock versioning¹ leading
> up
> >> to
> >> > > the
> >> > > > 4.0 release?
> >> > > >
> >> > > > To just stop tick-tock and then say yeehaa let's jam in all the
> >> > breaking
> >> > > > changes we really want seems to be throwing away some of the
> learnt
> >> > > wisdom,
> >> > > > and not doing a very sane transition from tick-tock to
> >> > > > features/testing/stable². I really hope all this is done in a way
> >> that
> >> > > > continues us down the path towards a stable-master.
> >> > > >
> >> > > > For example, are we fixing the release of 4.0 to November? or
> >> > continuing
> >> > > > tick-tocks until we complete the 4.0 roadmap? or starting the
> >> > > > features/testing/stable branching approach with 3.11?
> >> > > >
> >> > > >
> >> > > > Background:
> >> > > > ¹) Sylvain wrote in an earlier thread titled "A Home for 4.0"
> >> > > >
> >> > > > > And as 4.0 was initially supposed to come after 3.11, which is
> >> > coming,
> >> > > > it's probably time to have a home for those tickets.
> >> > > >
> >> > > > ²) The new versioning scheme slated for 4.0, per the "Proposal -
> >> 3.5.1"
> >> > > > thread
> >> > > >
> >> > > > > three branch plan with “features”, “testing”, and “stable”
> >starting
> >> > > with
> >> > > > 4.0?
> >> > > >
> >> > > >
> >> > > > Mick
> >> > > >
> >> > >
> >> >
> >>
>

Re: Is there a way to do Read and Set at Cassandra level?

[DuyHai Doan]

"But then don't I need to evict for every batch of writes?"

Yes, that's why I think an in-memory distributed data structure is the good
fit for your scenario. Using a log structured merged tree like C* for this
use-case is not the most efficient choice

On Sat, Nov 5, 2016 at 11:52 AM, Kant Kodali <k...@peernova.com> wrote:

> But then don't I need to evict for every batch of writes? I thought cache
> would make sense when reads/writes > 1 per say. What do you think?
>
> On Sat, Nov 5, 2016 at 3:33 AM, DuyHai Doan <doanduy...@gmail.com> wrote:
>
>> "I have a requirement where I need to know last value that is written
>> successfully so I could read that value and do some computation and include
>> it in the subsequent write"
>>
>> Maybe keeping the last written value in a distributed cache is cheaper
>> than doing a read before write in Cassandra ?
>>
>> On Sat, Nov 5, 2016 at 11:24 AM, Kant Kodali <k...@peernova.com> wrote:
>>
>>> I have a requirement where I need to know last value that is written
>>> successfully so I could read that value and do some computation and include
>>> it in the subsequent write. For now we are doing read before write which
>>> significantly degrades the performance. Light weight transactions are more
>>> of a compare and set than a Read and Set. The very first thing I tried is
>>> to see if I can eliminate this need by the application but looks like it is
>>> a strong requirement for us so I am wondering if there is any way I can
>>> optimize that? I know batching could help in the sense I can do one read
>>> for every batch so that the writes in the batch doesn't take a read
>>> performance hit but I wonder if there is any clever ideas or tricks I can
>>> do?
>>>
>>
>>
>

Re: Is there a way to do Read and Set at Cassandra level?

[DuyHai Doan]

"I have a requirement where I need to know last value that is written
successfully so I could read that value and do some computation and include
it in the subsequent write"

Maybe keeping the last written value in a distributed cache is cheaper than
doing a read before write in Cassandra ?

On Sat, Nov 5, 2016 at 11:24 AM, Kant Kodali  wrote:

> I have a requirement where I need to know last value that is written
> successfully so I could read that value and do some computation and include
> it in the subsequent write. For now we are doing read before write which
> significantly degrades the performance. Light weight transactions are more
> of a compare and set than a Read and Set. The very first thing I tried is
> to see if I can eliminate this need by the application but looks like it is
> a strong requirement for us so I am wondering if there is any way I can
> optimize that? I know batching could help in the sense I can do one read
> for every batch so that the writes in the batch doesn't take a read
> performance hit but I wonder if there is any clever ideas or tricks I can
> do?
>

Re: Is SASI index in Cassandra efficient for high cardinality columns?

[DuyHai Doan]

If you read my blog post about 2nd index deep dive, you'll get all the
answers
Le 21 oct. 2016 10:20, "Kant Kodali" <k...@peernova.com> a écrit :

> Why Secondary index cannot be broken down into token ranges like primary
> index at least for exact matches? That way dont need to scan the whole
> cluster atleast for exact matches. I understand if it is a substring search
> then there will 2^n substrings which equates to 2^n hashes/tokens which can
> be a lot!
>
> On Sat, Oct 15, 2016 at 4:35 AM, DuyHai Doan <doanduy...@gmail.com> wrote:
>
> > If each indexed value has very few matching rows, then querying using
> SASI
> > (or any impl of secondary index) may scan the whole cluster.
> >
> > This is because the index are "distributed" e.g. the indexed values stay
> > on the same nodes as the base data. And even SASI with its own
> > data-structure will not help much here.
> >
> > One should understand that the 2nd index query has to deal with 2 layers:
> >
> > 1) The cluster layer, which is common for any impl of 2nd index. Read my
> > blog post here: http://www.planetcassandra.org/blog/
> > cassandra-native-secondary-index-deep-dive/
> >
> > 2) The local read path, which depends on the impl of 2nd index. Some are
> > using Lucene library like Stratio impl, some rolls in its own data
> > structures like SASI
> >
> > If you have a 1-to-1 relationship between the index value and the
> matching
> > row (or 1-to-a few), I would recommend using materialized views instead:
> >
> > http://www.slideshare.net/doanduyhai/sasi-cassandra-on-
> > the-full-text-search-ride-voxxed-daybelgrade-2016/25
> >
> > Materialized views guarantee that for each search indexed value, you only
> > hit a single node (or N replicas depending on the used consistency level)
> >
> > However, materialized views have their own drawbacks (weeker consistency
> > guarantee) and you can't use range queries (<,  >, ≤, ≥) or full text
> > search on the indexed value
> >
> >
> >
> >
> >
> > On Sat, Oct 15, 2016 at 11:55 AM, Kant Kodali <k...@peernova.com> wrote:
> >
> >> Well I went with the definition from wikipedia and that definition rules
> >> out #1 so it is #2 and it is just one matching row in my case.
> >>
> >>
> >>
> >> On Sat, Oct 15, 2016 at 2:40 AM, DuyHai Doan <doanduy...@gmail.com>
> >> wrote:
> >>
> >> > Define precisely what you mean by "high cardinality columns". Do you
> >> mean:
> >> >
> >> > 1) a single indexed value is present in a lot of rows
> >> > 2) a single indexed value has only a few (if not just one) matching
> row
> >> >
> >> >
> >> > On Sat, Oct 15, 2016 at 8:37 AM, Kant Kodali <k...@peernova.com>
> wrote:
> >> >
> >> >> I understand Secondary Indexes in general are inefficient on high
> >> >> cardinality columns but since SASI is built from scratch I wonder if
> >> the
> >> >> same argument applies there? If not, Why? Because I believe primary
> >> keys in
> >> >> Cassandra are indeed indexed and since Primary key is supposed to be
> >> the
> >> >> column with highest cardinality why not do the same for secondary
> >> indexes?
> >> >>
> >> >
> >> >
> >>
> >
> >
>

Re: Is SASI index in Cassandra efficient for high cardinality columns?

[DuyHai Doan]

If each indexed value has very few matching rows, then querying using SASI
(or any impl of secondary index) may scan the whole cluster.

This is because the index are "distributed" e.g. the indexed values stay on
the same nodes as the base data. And even SASI with its own data-structure
will not help much here.

One should understand that the 2nd index query has to deal with 2 layers:

1) The cluster layer, which is common for any impl of 2nd index. Read my
blog post here:
http://www.planetcassandra.org/blog/cassandra-native-secondary-index-deep-dive/

2) The local read path, which depends on the impl of 2nd index. Some are
using Lucene library like Stratio impl, some rolls in its own data
structures like SASI

If you have a 1-to-1 relationship between the index value and the matching
row (or 1-to-a few), I would recommend using materialized views instead:

http://www.slideshare.net/doanduyhai/sasi-cassandra-on-the-full-text-search-ride-voxxed-daybelgrade-2016/25

Materialized views guarantee that for each search indexed value, you only
hit a single node (or N replicas depending on the used consistency level)

However, materialized views have their own drawbacks (weeker consistency
guarantee) and you can't use range queries (<,  >, ≤, ≥) or full text
search on the indexed value





On Sat, Oct 15, 2016 at 11:55 AM, Kant Kodali <k...@peernova.com> wrote:

> Well I went with the definition from wikipedia and that definition rules
> out #1 so it is #2 and it is just one matching row in my case.
>
>
>
> On Sat, Oct 15, 2016 at 2:40 AM, DuyHai Doan <doanduy...@gmail.com> wrote:
>
> > Define precisely what you mean by "high cardinality columns". Do you
> mean:
> >
> > 1) a single indexed value is present in a lot of rows
> > 2) a single indexed value has only a few (if not just one) matching row
> >
> >
> > On Sat, Oct 15, 2016 at 8:37 AM, Kant Kodali <k...@peernova.com> wrote:
> >
> >> I understand Secondary Indexes in general are inefficient on high
> >> cardinality columns but since SASI is built from scratch I wonder if the
> >> same argument applies there? If not, Why? Because I believe primary
> keys in
> >> Cassandra are indeed indexed and since Primary key is supposed to be the
> >> column with highest cardinality why not do the same for secondary
> indexes?
> >>
> >
> >
>

Re: Is SASI index in Cassandra efficient for high cardinality columns?

[DuyHai Doan]

Define precisely what you mean by "high cardinality columns". Do you mean:

1) a single indexed value is present in a lot of rows
2) a single indexed value has only a few (if not just one) matching row


On Sat, Oct 15, 2016 at 8:37 AM, Kant Kodali  wrote:

> I understand Secondary Indexes in general are inefficient on high
> cardinality columns but since SASI is built from scratch I wonder if the
> same argument applies there? If not, Why? Because I believe primary keys in
> Cassandra are indeed indexed and since Primary key is supposed to be the
> column with highest cardinality why not do the same for secondary indexes?
>

Re: Question regd CDC in cassandra 3.7+

[DuyHai Doan]

You need to use the CommitLogReader, there is no way to access CDC data
using CQL. I'm not even sure it will be possible one day

On Fri, Oct 7, 2016 at 11:19 PM, sridhar nemani 
wrote:

> Hi,
>
>
>
> I am fairly new to Cassandra. I have a requirement to be able to read any
> changes to tables, as in inserts deletes or updates from a given timestamp
> onwards. I believe the new implementation of CDC should help me with this.
> However, with CDC enabled, I want to know if there is yet a way to read the
> data inserts,updates or deletes to a table through CQL. I do see
> implementations of CommitLogReader. But, I want to know if it is possible
> to read the changes using CQL. If yes, how?
>
> Please advise.
>
>
>
> Thank you,
> Sridhar
>

Re: [VOTE] Release Apache Cassandra 3.9

[DuyHai Doan]

 Java driver  3.1.0 for support of the latest features of Cassandra 3.x
(PER PARTITION LIMIT, JSON support ...)

Java driver 3.0.4 is also compatible with Cassandra 3.x but has less
support for bleeding edge features

On Sat, Oct 1, 2016 at 4:29 PM, Jonathan Ekwempu 
wrote:

> Is there a Java driver for version 3.9? Which Java driver are people using
> with 3.9?
>
> On Mon, Sep 26, 2016 at 11:12 AM, Michael Shuler 
> wrote:
>
> > I propose the following artifacts for release as 3.9.
> >
> > sha1: c1fa21458777b51a9b21795330ed6f298103b436
> > Git:
> > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> > shortlog;h=refs/tags/3.9-tentative
> > Artifacts:
> > https://repository.apache.org/content/repositories/
> > orgapachecassandra-1127/org/apache/cassandra/apache-cassandra/3.9/
> > Staging repository:
> > https://repository.apache.org/content/repositories/
> > orgapachecassandra-1127/
> >
> > The Debian packages are available here: http://people.apache.org/~
> mshuler
> >
> > The vote will be open for 72 hours (longer if needed).
> >
> > [1]: (CHANGES.txt) https://goo.gl/TEMHqi
> > [2]: (NEWS.txt) https://goo.gl/1w6Ec1
> >
> > --
> > Kind regards,
> > Michael Shuler
> >
>

Re: Question on assert

[DuyHai Doan]

I found that SO question very interesting to fuel the discussion about
assert vs exception :
http://stackoverflow.com/questions/1957645/when-to-use-an-assertion-and-when-to-use-an-exception

On Wed, Sep 21, 2016 at 8:20 PM, Michael Kjellman <
mkjell...@internalcircle.com> wrote:

> Yeah, I understand what you're saying, don't get me wrong.
>
> However, I just spent close to a year total working and writing
> CASSANDRA-9754 and when you're dealing with IO, sometimes asserts are the
> right way to go. I found putting them there are sanity checks mostly to
> ensure that code changes to other parts of the code don't have unexpected
> interactions with the input bounds expected by a method. I think asserts
> are fine (and correct) in these cases.
>
>
> > On Sep 21, 2016, at 11:16 AM, Edward Capriolo 
> wrote:
> >
> > You are essentially arguing, "if you turn off -ea your screwed" which is
> a
> > symptom of a larger problem that I am pointing out.
> >
> > Forget the "5%" thing. I am having a discussion about use of assert.
> >
> > You have:
> > 1) checked exceptions
> > 2) unchecked exceptions
> > 3) Error (like ioError which we sometime have to track)
> >
> > The common case for assert is to only be used in testing. This is why -ea
> > is off by default.
> >
> > My point is that using assert as a Apache Cassandra specific "psuedo
> > exception" seems problematic. I can point at tickets in the Cassandra
> Jira
> > where the this is not trapped properly. It appears to me that having deal
> > with a 4th "pseudo exception" is code smell.
> >
> > Sometimes you see assert in place of a bounds check or a null check that
> > you would never want to turn off. Other times it is uses as a quasi
> > IllegalStateException. Other times an class named "estimator" asserts
> when
> > the "estimate" "overflows". This seem far away from the defined purpose
> of
> > assert.
> >
> > The glaring issue is that it bubbles through try catch so it hardly makes
> > me feel "safe" either on or off.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Wed, Sep 21, 2016 at 1:34 PM, Michael Kjellman <
> > mkjell...@internalcircle.com> wrote:
> >
> >> Asserts have their place as sanity checks. Just like exceptions have
> their
> >> place.
> >>
> >> They can both live in harmony and they both serve a purpose.
> >>
> >> What doesn't serve a purpose is that comment encouraging n00b users to
> get
> >> a mythical 5% performance increase and then get silent corruption when
> >> their disk/io goes sideways and the asserts might have caught things
> before
> >> it went really wrong.
> >>
> >> Sent from my iPhone
> >>
> >> On Sep 21, 2016, at 10:31 AM, Edward Capriolo  >> > wrote:
> >>
> >> " potential 5% performance win when you've corrupted all their data."
> >> This is somewhat of my point. Why do assertions that sometimes are
> trapped
> >> "protect my data" better then a checked exception?
> >>
> >> On Wed, Sep 21, 2016 at 1:24 PM, Michael Kjellman <
> >> mkjell...@internalcircle.com>
> wrote:
> >>
> >> I hate that comment with a passion. Please please please please do
> >> yourself a favor and *always* run with asserts on. `-ea` for life. In
> >> practice I'd be surprised if you actually got a reliable 5% performance
> win
> >> and I doubt your customers will care about a potential 5% performance
> win
> >> when you've corrupted all their data.
> >>
> >> best,
> >> kjellman
> >>
> >> On Sep 21, 2016, at 10:21 AM, Edward Capriolo  >> >
> >> wrote:
> >>
> >> There are a variety of assert usages in the Cassandra. You can find
> >> several
> >> tickets like mine.
> >>
> >> https://issues.apache.org/jira/browse/CASSANDRA-12643
> >>
> >> https://issues.apache.org/jira/browse/CASSANDRA-11537
> >>
> >> Just to prove that I am not the only one who runs into these:
> >>
> >> https://issues.apache.org/jira/browse/CASSANDRA-12484
> >>
> >> To paraphrase another ticket that I read today and can not find,
> >> "The problem is X throws Assertion which is not caught by the Exception
> >> handler and it bubbles over and creates a thread death."
> >>
> >> The jvm.properties file claims this:
> >>
> >> # enable assertions.  disabling this in production will give a modest
> >> # performance benefit (around 5%).
> >> -ea
> >>
> >> If assertions incur a "5% penalty" but are not always trapped what value
> >> do
> >> they add?
> >>
> >> These are common sentiments about how assert should be used: (not trying
> >> to
> >> make this a this is what the internet says type debate)
> >>
> >> http://stackoverflow.com/questions/2758224/what-does-
> >> the-java-assert-keyword-do-and-when-should-it-be-used
> >>
> >> "Assertions
> >>  >
> >> (by
> >> way of the *assert* keyword) were added in Java 1.4. They are used to
> >> verify the 

Re: Github pull requests

[DuyHai Doan]

Just a question before moving forward, do we want to trigger CI build for
each proposed pull request on Github ? And if yes, on which infra ?

I'm asking because opening PR to the crowd is a good thing but we may have
tons of PRs coming and the infra may be heavily loaded.

Apache Zeppelin project is using Travis as CI infra and there are
constantly errors related to resource shortage (not enough CPU/memory
available)



On Mon, Aug 29, 2016 at 6:48 AM, Jonathan Ellis  wrote:

> Don't we need something on the infra side to turn a merged pull request
> into a commit to the ASF repo?
>
> On Sun, Aug 28, 2016 at 11:07 PM, Nate McCall 
> wrote:
>
> > >
> > >
> > > Infra is exploring options for giving PMCs greater control over GitHub
> > > config (including allowing GitHub to be the master with a golden copy
> > > held at the ASF) but that is a work in progress.
> > >
> > >
> > ^  Per Mark's comment, there is not anything we can really do past what
> > Jake F. described with Thrift. We dealt with this with Usergrid back in
> > incubation two years ago (Jake F. actually helped us get it all sorted at
> > the time) when we were using https://github.com/usergrid/usergrid as the
> > source:
> > http://mail-archives.apache.org/mod_mbox/usergrid-dev/201405.mbox/%
> > 3CCANyrgvdTVzZQD7w3C96LUHa=h7-h4qmu4h7ajsxoat0gd0f...@mail.gmail.com%3E
> >
> > Here is the Thrift guide again for reference:
> > https://github.com/apache/thrift/blob/master/
> CONTRIBUTING.md#contributing-
> > via-github-pull-requests
> >
> > JClouds also has a nice write up/how-to (we based Usergrid on this,
> > initially):
> > https://cwiki.apache.org/confluence/display/JCLOUDS/Git+workflow
> >
> > Maybe we just amend our 'how-to-commit' with similar details as the two
> > references above?
> > http://cassandra.apache.org/doc/latest/development/how_to_commit.html
> >
> > -Nate
> >
> > On Mon, Aug 29, 2016 at 10:44 AM, Nate McCall 
> > wrote:
> >
> > >
> > >> Nate, since you have experience with this from Usergrid, can you
> figure
> > >> out
> > >> what we need to do to make this happen and follow up with infra?
> > >>
> > >
> > > Yep - i'll look into this.
> > >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder, http://www.datastax.com
> @spyced
>

Re: Distributed masterless architecture

[DuyHai Doan]

You can read this blog post, there are a handful of interesting links:
http://the-paper-trail.org/blog/distributed-systems-theory-for-the-distributed-systems-engineer/

On Wed, Aug 24, 2016 at 1:45 PM, Salih Gedik  wrote:

> Hi everyone,
> I am an undergrad student and working on a simple distributed database for
> learning purposes. I was wondering if you guys can give me tips about
> designing and coding distributed no master nodes. For instance what classes
> should I be looking for in source code? I am so sorry if this is not the
> right place.
> Thank you so much!
>
>  Best regards
> --
> Salih Gedik
>

Re: Using writetime in CAS Lightweight transactions

[DuyHai Doan]

It is not (yet) possible to use functions in LWT predicates. LWT only
supports = and != plus IF (NOT) EXISTS right now
Le 11 mai 2016 16:50, "Bhuvan Rawal"  a écrit :

> Hi,
>
> I was working on maintaining counters in cassandra, I read that it is not
> 100% accurate, I have tested them to be extremely accurate with 3.0.5 but
> going by docs there may be slight in accuracy.
>
> Thinking about this I thought there may be 2 approaches to resolve this:
> 1. Read the existing count and write the new count if table count is the
> count earlier fetched using LWT transaction.
> 2. Read the existing writetime(count) and write the new count if
> writetime(count) is the writetime count earlier fetched using lightweight
> transaction.-
>
> For me approach1 is working, but approach 2 doesnt. Please Find below the
> same:
>
> superuser@cqlsh:test_keyspace> CREATE TABLE test_keyspace.test (
> ... part_k int,
> ... clust_k text,
> ... count int,
> ... PRIMARY KEY (part_k, clust_k)
> ... );
> superuser@cqlsh:test_keyspace> insert into test(part_k , clust_k , count )
> values (2390, 'Test Ck Value', 007);
> superuser@cqlsh:test_keyspace> select * from test;
>
>  part_k | clust_k   | count
> +---+---
>2390 | Test Ck Value | 7
>
> superuser@cqlsh:test_keyspace> update test set count=8 where part_k=2390
> and clust_k='Test Ck Value' if count=7;
> # Works perfectly
>
> superuser@cqlsh:test_keyspace> select
> writetime(count),part_k,clust_k,count
> from test;
>
>  writetime(count) | part_k | clust_k   | count
> --++---+---
>  1462974475292000 |   2390 | Test Ck Value | 8
>
> # Now try to use the write time in LWT
> superuser@cqlsh:test_keyspace> update test set count=8 where part_k=2390
> and clust_k='Test Ck Value' if writetime(count)=1462974475292000;
> SyntaxException:  message="line 1:82 no viable alternative at input '(' (... and
> clust_k='Test Ck Value' if writetime[(]...)">
>
> Best Regards,
> Bhuvan Rawal
>

Re: Datastax including Cassandra 3.0

[DuyHai Doan]

The next major release of Datastax Enterprise 5.0 will include C* 3.x. It
is expected to be released somewhere around June/July, no exact date has
been announced yet.
Le 10 mai 2016 09:08, "Joly, Josselin"  a écrit :

> Hello,
>
> I am wondering if the version of Cassandra 3.0 is already included in the
> lastest version of Datastax entreprise, do you know somthing about it ?
>
>
> Best regards,
>
> Josselin Joly.
>
> [cpu_time.png]
>