Re: thread architecture of Cassandra

[Suli Yang]

Hi,

Please find the attachment in the following URL: 
http://pages.cs.wisc.edu/~suli/cassandra.pdf

Thanks a lot for your help!

Suli

On 2017-03-22 15:40 (-0500), Jeff Jirsa  wrote: 
> Hi Suli,
> 
> Attachments to mailing lists often get filtered - can you paste it online
> and send a link?
> 
> - Jeff
> 
> 
> On Wed, Mar 22, 2017 at 1:10 PM, 杨苏立 Yang Su Li  
> wrote:
> 
> > Hi,
> >
> > I am a graduate student working on scheduling on storage systems, and we
> > are interested in how different threads in Cassandra interact with each
> > other and how it might affect scheduling.
> >
> > I have written down my understanding on how Cassandra works based on its
> > current thread architecture (attached). I am wondering if the developers of
> > Cassandra could take a look at it and let me know if anything is incorrect
> > or inaccurate, or if I have missed anything.
> >
> > Thanks a lot for your help!
> >
> > Suli
> >
> >
> > --
> > Suli Yang
> >
> > Department of Physics
> > University of Wisconsin Madison
> >
> > 4257 Chamberlin Hall
> > Madison WI 53703
> >
> 

Re: thread architecture of Cassandra

[Jeff Jirsa]

Hi Suli,

Attachments to mailing lists often get filtered - can you paste it online
and send a link?

- Jeff


On Wed, Mar 22, 2017 at 1:10 PM, 杨苏立 Yang Su Li  wrote:

> Hi,
>
> I am a graduate student working on scheduling on storage systems, and we
> are interested in how different threads in Cassandra interact with each
> other and how it might affect scheduling.
>
> I have written down my understanding on how Cassandra works based on its
> current thread architecture (attached). I am wondering if the developers of
> Cassandra could take a look at it and let me know if anything is incorrect
> or inaccurate, or if I have missed anything.
>
> Thanks a lot for your help!
>
> Suli
>
>
> --
> Suli Yang
>
> Department of Physics
> University of Wisconsin Madison
>
> 4257 Chamberlin Hall
> Madison WI 53703
>

Re: thread architecture of Cassandra

[Tyler Hobbs]

It looks like your attachment didn't make it -- I'm not sure that the
mailing list supports them.  Maybe try posting it somewhere and linking it?

On Wed, Mar 22, 2017 at 3:10 PM, 杨苏立 Yang Su Li  wrote:

> Hi,
>
> I am a graduate student working on scheduling on storage systems, and we
> are interested in how different threads in Cassandra interact with each
> other and how it might affect scheduling.
>
> I have written down my understanding on how Cassandra works based on its
> current thread architecture (attached). I am wondering if the developers of
> Cassandra could take a look at it and let me know if anything is incorrect
> or inaccurate, or if I have missed anything.
>
> Thanks a lot for your help!
>
> Suli
>
>
> --
> Suli Yang
>
> Department of Physics
> University of Wisconsin Madison
>
> 4257 Chamberlin Hall
> Madison WI 53703
>



-- 
Tyler Hobbs
DataStax 

thread architecture of Cassandra

[杨苏立 Yang Su Li]

Hi,

I am a graduate student working on scheduling on storage systems, and we
are interested in how different threads in Cassandra interact with each
other and how it might affect scheduling.

I have written down my understanding on how Cassandra works based on its
current thread architecture (attached). I am wondering if the developers of
Cassandra could take a look at it and let me know if anything is incorrect
or inaccurate, or if I have missed anything.

Thanks a lot for your help!

Suli


-- 
Suli Yang

Department of Physics
University of Wisconsin Madison

4257 Chamberlin Hall
Madison WI 53703

Re: Code quality, principles and rules

[Michael Shuler]

On 03/22/2017 12:41 PM, François Deliège wrote:
> A first actionable step is to increase the visibility of the test
> coverage.  Ideally this would be integrated in the Jenkins run on
> Apache.  Michael Shuler, is this something you can take a look at?
> Let me know if we can help.

We've been running the jacoco coverage ant target on cassci for a very
long time, but cassci will be going away in the future. I will not have
time to add coverage to ASF Jenkins in the foreseeable future, so help
would certainly be appreciated in all things testing for the Apache
Cassandra project.

Someone interested in including coverage runs could add to the Job DSL
groovy under the jenkins-dsl/ dir - this is where all the jobs are
configured on the ASF Jenkins instance. Dikang /msg'ed me on IRC earlier
about coverage, too.

Here's the cassandra-builds repo that drives everything on the project's
ASF Jenkins jobs:
https://git1-us-west.apache.org/repos/asf?p=cassandra-builds.git

Here's where all the current project jobs are on ASF Jenkins:
https://builds.apache.org/view/A-D/view/Cassandra/

Here's info about how the Job DSL works - all job configs are managed
with DSL, with the exception of a couple scratch utility jobs:
https://github.com/jenkinsci/job-dsl-plugin/wiki

Here's info about ASF Jenkins:
https://cwiki.apache.org/confluence/display/INFRA/Jenkins

Basically, my task over the last few months has been to migrate Apache
Cassandra testing jobs to run on the Apache Foundation's Jenkins and
sunset CassCI. 95% of that task is done.

-- 
Warm regards,
Michael

Re: Code quality, principles and rules

[François Deliège]

Thanks everybody for chiming in.  I have not heard any concerns about the 
rules, so I’d like to move forward with some concrete steps in that direction.

A first actionable step is to increase the visibility of the test coverage.  
Ideally this would be integrated in the Jenkins run on Apache.  Michael Shuler, 
is this something you can take a look at?  Let me know if we can help.

A second step, imho, is to get agreement among committers that no patch that 
decrease test coverage will be accepted by any committer.   Could a PMC throw 
this question as a vote?

Finally, I used my mad data analytics skills to look at the ratio of non 
committers contributors during the last 6 months.Turns out that 60% of the 
authors are not committers.   So as mentioned in the testing thread Jason 
started, when we think about testing, I think it makes sense to consider 
opening it up to non committers.   Today the testing time ranges from 2 hours 
for unitests to 1 day for integration tests.  I’d like to explore if we can 
throw some hardware at the problem.  I’d appreciate a pointer to who I should 
talk to.

Re: splitting CQL parser & spec into separate repo

[Edward Capriolo]

On Tue, Mar 21, 2017 at 5:45 PM, Anthony Grasso 
wrote:

> This is a great idea
>
> +1 (non-binding)
>
> On 22 March 2017 at 07:04, Edward Capriolo  wrote:
>
> > On Tue, Mar 21, 2017 at 3:24 PM, Mark Dewey  wrote:
> >
> > > I can immediately think of a project I would use that in. +1
> > >
> > > On Tue, Mar 21, 2017 at 12:18 PM Jonathan Haddad 
> > > wrote:
> > >
> > > > I created CASSANDRA-13284 a few days ago with the intent of starting
> a
> > > > discussion around the topic of breaking the CQL parser out into a
> > > separate
> > > > project.  I see a few benefits to doing it and was wondering what the
> > > folks
> > > > here thought as well.
> > > >
> > > > First off, the Java CQL parser would obviously continue to be the
> > > reference
> > > > parser.  I'd love to see other languages have CQL parsers as well,
> but
> > > the
> > > > intent here isn't for the OSS C* team to be responsible for
> maintaining
> > > > that.  My vision here is simply the ability to have some high level
> > > > CQLParser.parse(statement) call that returns the parse tree, nothing
> > > more.
> > > >
> > > > It would be nice to be able to leverage that parser in other projects
> > > such
> > > > as IDEs, code gen tools, etc.  It would be outstanding to be able to
> > > create
> > > > the parser tests in such a way that they can be referenced by other
> > > parsers
> > > > in other languages.  Yay code reuse.  It also has the benefit of
> making
> > > the
> > > > codebase a little more modular and a bit easier to understand.
> > > >
> > > > Thoughts?
> > > >
> > > > Jon
> > > >
> > >
> >
> > It turns out that a similar thing was done with Hive.
> >
> > https://calcite.apache.org/
> >
> > https://calcite.apache.org/community/#apache-calcite-one-
> planner-fits-all
> >
> > The challenge is typically adoption. The elevator pitch is like:
> > "EVERYONE WILL SHARE THIS AND IT WILL BE AWESOME". Maybe this is the
> wrong
> > word, but lets just say frenemies
> > exist and they do not like control of something moving to a shared
> medium.
> > Technical issues like ANTLR 3 vs ANTRL 4 etc.
> > For something like Hive the challenge is the parser/planner needs only be
> > fast enough for analytic queries but that would not
> > be the right move for say CQL.
> >
>

I believe you could accomplish a similar goal by making a multi-module
project https://maven.apache.org/guides/mini/guide-multiple-modules.html.
Probably not as easy thanks to ant, but I think that is a better route. One
there actually are N dependent projects in the wild you can make the case
for overhead which is both technical and in ASF based.