Re: scylladb

[Dor Laor]

On Tue, Mar 14, 2017 at 7:43 AM, Eric Evans 
wrote:

> On Sun, Mar 12, 2017 at 4:01 PM, James Carman
>  wrote:
> > Does all of this Scylla talk really even belong on the Cassandra user
> > mailing list in the first place?
>
> I personally found it interesting, informative, and on-topic when it
> was about justification of the 10x performance claim, numa,
> scheduling, concurrency, etc.  At some point it got a little sales-y
> on the one side, and caremad on the other.
>

I tried to just provide accurate answers. There is a meme on twitter that
matches it:

Pi Day is just a fake holiday created by math companies to sell more math.

It's that good so I couldn't resist.


> ¯\_(ツ)_/¯
>
>
> --
> Eric Evans
> john.eric.ev...@gmail.com
>

Re: scylladb

[Eric Evans]

On Sun, Mar 12, 2017 at 4:01 PM, James Carman
 wrote:
> Does all of this Scylla talk really even belong on the Cassandra user
> mailing list in the first place?

I personally found it interesting, informative, and on-topic when it
was about justification of the 10x performance claim, numa,
scheduling, concurrency, etc.  At some point it got a little sales-y
on the one side, and caremad on the other.

¯\_(ツ)_/¯


-- 
Eric Evans
john.eric.ev...@gmail.com

Re: scylladb

[Dor Laor]

On Mon, Mar 13, 2017 at 12:17 AM, benjamin roth  wrote:

> @Dor,Jeff:
>
> I think Jeff pointed out an important fact: You cannot stop CS, swap
> binaries and start Scylla. To be honest that was AFAIR the only "Oooh :(" I
> had when reading the Scylla "marketing material".
>

If you're on 2.1.x you can. You will need to stop your running cluster and
many users cannot do that.
If you do have ability to sustain downtime, take a snapshot and thus your
data is safe.
There is a way to do it online, explained below


>
> If that worked it would be very valuable from both Scylla's and a users'
> point of view. As a user I would love to give scylla a try as soon as it
> provides all the features my application requires. But the hurdle is quite
> high. I have to create a separate scylla cluster and I have to migrate a
> lot of data and I have to manage somehow that my application can use (r+w)
> both CS + Scylla at the same time to not run any risk of data loss or dead
> end road if something goes wrong. And still: I would not be able to
>

So there are two options to do double writes, one is do it on the client
side and send to the two separate clusters.
Another option is we have a CQL-proxy small go application. You install it
on all of your C* nodes and it duplicated the CQL
traffic to remote Scylla cluster.

When you do the above you start Scylla from a C* snapshot/backup or sstable
load the previous data (works for 2.x and 3.y)(first start the double
writes). If you have a double cluster, you have zero risk if something goes
wrong.


> compare CS + Scylla for my workload totally fair as the conditions
> changed. New hardware, maybe partial dataset, probably only "test traffic".
>

50% of users run on AWS so it's easy to use the same conditions.
If you're on physical machines, just give us half the hardware and see we
cope with it - meaning
err on our side (as long as you don't use crappy hardware which will cap
us).

The advantage is that you can run in this mode weeks/months until you're
absolute sure things work fine.
Most of our users who came from C* did it.


> However, if I was able to just replace a single node in an existing
> cluster I'd have:
>

Everybody wanted it but


> 1. Superlow hurdle to give it a try: No risk, no effort
>

Yes there is risk. We considered that and even toyed with such an
implementation.
The risk is that this node will be part of the cluster and although it
supposed to behave and you'll
have replicas, something may go wrong and your cluster's RPC will suffer.

The Cassandra RPC wasn't documented as changed over the releases, I bet you
cannot mix a 2.1 with 3.4
and can only do latest x.minor -> (x+1).0
In addition, in case there is an issue, whose fault would it be?
Lastly, we have our own internal RPC with various optimizations.


> 2. Fair comparison by comparing new node against some equally equipeed old
> node in the same cluster with the same workload
>

Even if it would work, the new node won't run faster unless CL=1 which is
rare.


> 3. Easy to make a decision if to continue or not
>
> That would be totally awesome!
>
>
> 2017-03-12 23:16 GMT+01:00 Kant Kodali :
>
>> I don't think ScyallDB guys started this conversation in the first place
>> to suggest or promote "drop-in replacement". It was something that is
>> brought up by one of the Cassandra users and ScyallDB guys just clarified
>> it. They are gracious enough to share the internals in detail.
>>
>> honestly, I find it weird when I see questions like whether a question
>> belongs  to a mailing list or not especially in this case. If one doesn't
>> like it they can simply not follow the thread. I am not sure what is the
>> harm here.
>>
>>
>>
>> On Sun, Mar 12, 2017 at 2:29 PM, James Carman > > wrote:
>>
>>> Well, looking back, it appears this thread is from 2015, so apparently
>>> everyone is okay with it.
>>>
>>> Promoting a value-add product that makes using Cassandra easier/more
>>> efficient/etc would be cool, but coming to the Cassandra mailing list to
>>> promote a "drop-in replacement" (use us, not Cassandra) isn't cool, IMHO.
>>>
>>>
>>> On Sun, Mar 12, 2017 at 5:04 PM Kant Kodali  wrote:
>>>
>>> yes.
>>>
>>> On Sun, Mar 12, 2017 at 2:01 PM, James Carman <
>>> ja...@carmanconsulting.com> wrote:
>>>
>>> Does all of this Scylla talk really even belong on the Cassandra user
>>> mailing list in the first place?
>>>
>>>
>>>
>>>
>>> On Sun, Mar 12, 2017 at 4:07 PM Jeff Jirsa  wrote:
>>>
>>>
>>>
>>> On 2017-03-11 22:33 (-0700), Dor Laor  wrote:
>>> > On Sat, Mar 11, 2017 at 10:02 PM, Jeff Jirsa  wrote:
>>> > > On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote:
>>> > > > Cassanda vs Scylla is a valid comparison because they both are
>>> > > compatible. Scylla is a drop-in replacement for Cassandra.
>>> > >
>>> > > No, they aren't, and no, it isn't
>>> > >
>>> >
>>> > Jeff is 

Re: scylladb

[Avi Kivity]

Hi,


We did indeed consider support for a mixed cluster, but in the end 
decided against it, for many reasons:


 - the internode protocol is underdocumented and keeps changing, so it 
would be hard to support it, and hard to test it


 - it would limit the kind of optimizations we can do by reducing our 
freedom to change implementation details


 - would users be willing to risk their production cluster by adding a 
foreign node? many would not


 - a cluster is as fast as its slowest node, so you wouldn't see 
performance improvements until you've upgraded all of your nodes



So in the end, we went with our own internode protocol.  We do have a 
migration support tool (cqlproxy - duplicates cql requests to two 
parallel clusters), and many users just add dual-write support to their 
applications during the proof-of-concept/migration phase.



btw, you can do the stop/swap/start cycle; you just have to stop all 
nodes for it.  While that's not possible for many production clusters, 
it is possible for others.


On 03/13/2017 09:17 AM, benjamin roth wrote:

@Dor,Jeff:

I think Jeff pointed out an important fact: You cannot stop CS, swap 
binaries and start Scylla. To be honest that was AFAIR the only "Oooh 
:(" I had when reading the Scylla "marketing material".


If that worked it would be very valuable from both Scylla's and a 
users' point of view. As a user I would love to give scylla a try as 
soon as it provides all the features my application requires. But the 
hurdle is quite high. I have to create a separate scylla cluster and I 
have to migrate a lot of data and I have to manage somehow that my 
application can use (r+w) both CS + Scylla at the same time to not run 
any risk of data loss or dead end road if something goes wrong. And 
still: I would not be able to compare CS + Scylla for my workload 
totally fair as the conditions changed. New hardware, maybe partial 
dataset, probably only "test traffic".


However, if I was able to just replace a single node in an existing 
cluster I'd have:

1. Superlow hurdle to give it a try: No risk, no effort
2. Fair comparison by comparing new node against some equally equipeed 
old node in the same cluster with the same workload

3. Easy to make a decision if to continue or not

That would be totally awesome!


2017-03-12 23:16 GMT+01:00 Kant Kodali >:


I don't think ScyallDB guys started this conversation in the first
place to suggest or promote "drop-in replacement". It was
something that is brought up by one of the Cassandra users and
ScyallDB guys just clarified it. They are gracious enough to share
the internals in detail.

honestly, I find it weird when I see questions like whether a
question belongs  to a mailing list or not especially in this
case. If one doesn't like it they can simply not follow the
thread. I am not sure what is the harm here.



On Sun, Mar 12, 2017 at 2:29 PM, James Carman
>
wrote:

Well, looking back, it appears this thread is from 2015, so
apparently everyone is okay with it.

Promoting a value-add product that makes using Cassandra
easier/more efficient/etc would be cool, but coming to the
Cassandra mailing list to promote a "drop-in replacement" (use
us, not Cassandra) isn't cool, IMHO.


On Sun, Mar 12, 2017 at 5:04 PM Kant Kodali > wrote:

yes.

On Sun, Mar 12, 2017 at 2:01 PM, James Carman
> wrote:

Does all of this Scylla talk really even belong on the
Cassandra user mailing list in the first place?




On Sun, Mar 12, 2017 at 4:07 PM Jeff Jirsa
> wrote:



On 2017-03-11 22:33 (-0700), Dor Laor
> wrote:
> On Sat, Mar 11, 2017 at 10:02 PM, Jeff Jirsa
> wrote:
> > On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote:
> > > Cassanda vs Scylla is a valid comparison
because they both are
> > compatible. Scylla is a drop-in replacement
for Cassandra.
> >
> > No, they aren't, and no, it isn't
> >
>
> Jeff is angry with us for some reason. I don't
know why, it's natural that
> when  a new opponent there are objections and
the proof lies on us.

I'm not angry. When I'm angry I send emails with
  

Re: scylladb

[Dor Laor]

We came to the thread to provide technical answers about whether the
difference in performance arise from
C++ only or beyond. When the discussion included numa, we even dove deep
into the weeds. I think we provided enough answers and I respect all of the
opinions here and thus if someone has further questions they are welcome to
ask on our mailing list or privately.

Cheers,
Dor

On Mon, Mar 13, 2017 at 12:43 AM, Dor Laor  wrote:

> On Mon, Mar 13, 2017 at 12:17 AM, benjamin roth  wrote:
>
>> @Dor,Jeff:
>>
>> I think Jeff pointed out an important fact: You cannot stop CS, swap
>> binaries and start Scylla. To be honest that was AFAIR the only "Oooh :(" I
>> had when reading the Scylla "marketing material".
>>
>
> If you're on 2.1.x you can. You will need to stop your running cluster and
> many users cannot do that.
> If you do have ability to sustain downtime, take a snapshot and thus your
> data is safe.
> There is a way to do it online, explained below
>
>
>>
>> If that worked it would be very valuable from both Scylla's and a users'
>> point of view. As a user I would love to give scylla a try as soon as it
>> provides all the features my application requires. But the hurdle is quite
>> high. I have to create a separate scylla cluster and I have to migrate a
>> lot of data and I have to manage somehow that my application can use (r+w)
>> both CS + Scylla at the same time to not run any risk of data loss or dead
>> end road if something goes wrong. And still: I would not be able to
>>
>
> So there are two options to do double writes, one is do it on the client
> side and send to the two separate clusters.
> Another option is we have a CQL-proxy small go application. You install it
> on all of your C* nodes and it duplicated the CQL
> traffic to remote Scylla cluster.
>
> When you do the above you start Scylla from a C* snapshot/backup or
> sstable load the previous data (works for 2.x and 3.y)(first start the
> double writes). If you have a double cluster, you have zero risk if
> something goes wrong.
>
>
>> compare CS + Scylla for my workload totally fair as the conditions
>> changed. New hardware, maybe partial dataset, probably only "test traffic".
>>
>
> 50% of users run on AWS so it's easy to use the same conditions.
> If you're on physical machines, just give us half the hardware and see we
> cope with it - meaning
> err on our side (as long as you don't use crappy hardware which will cap
> us).
>
> The advantage is that you can run in this mode weeks/months until you're
> absolute sure things work fine.
> Most of our users who came from C* did it.
>
>
>> However, if I was able to just replace a single node in an existing
>> cluster I'd have:
>>
>
> Everybody wanted it but
>
>
>> 1. Superlow hurdle to give it a try: No risk, no effort
>>
>
> Yes there is risk. We considered that and even toyed with such an
> implementation.
> The risk is that this node will be part of the cluster and although it
> supposed to behave and you'll
> have replicas, something may go wrong and your cluster's RPC will suffer.
>
> The Cassandra RPC wasn't documented as changed over the releases, I bet
> you cannot mix a 2.1 with 3.4
> and can only do latest x.minor -> (x+1).0
> In addition, in case there is an issue, whose fault would it be?
> Lastly, we have our own internal RPC with various optimizations.
>
>
>> 2. Fair comparison by comparing new node against some equally equipeed
>> old node in the same cluster with the same workload
>>
>
> Even if it would work, the new node won't run faster unless CL=1 which is
> rare.
>
>
>> 3. Easy to make a decision if to continue or not
>>
>> That would be totally awesome!
>>
>>
>> 2017-03-12 23:16 GMT+01:00 Kant Kodali :
>>
>>> I don't think ScyallDB guys started this conversation in the first place
>>> to suggest or promote "drop-in replacement". It was something that is
>>> brought up by one of the Cassandra users and ScyallDB guys just clarified
>>> it. They are gracious enough to share the internals in detail.
>>>
>>> honestly, I find it weird when I see questions like whether a question
>>> belongs  to a mailing list or not especially in this case. If one doesn't
>>> like it they can simply not follow the thread. I am not sure what is the
>>> harm here.
>>>
>>>
>>>
>>> On Sun, Mar 12, 2017 at 2:29 PM, James Carman <
>>> ja...@carmanconsulting.com> wrote:
>>>
 Well, looking back, it appears this thread is from 2015, so apparently
 everyone is okay with it.

 Promoting a value-add product that makes using Cassandra easier/more
 efficient/etc would be cool, but coming to the Cassandra mailing list to
 promote a "drop-in replacement" (use us, not Cassandra) isn't cool, IMHO.


 On Sun, Mar 12, 2017 at 5:04 PM Kant Kodali  wrote:

 yes.

 On Sun, Mar 12, 2017 at 2:01 PM, James Carman <
 ja...@carmanconsulting.com> wrote:

 Does all of this 

Re: scylladb

[benjamin roth]

@Dor,Jeff:

I think Jeff pointed out an important fact: You cannot stop CS, swap
binaries and start Scylla. To be honest that was AFAIR the only "Oooh :(" I
had when reading the Scylla "marketing material".

If that worked it would be very valuable from both Scylla's and a users'
point of view. As a user I would love to give scylla a try as soon as it
provides all the features my application requires. But the hurdle is quite
high. I have to create a separate scylla cluster and I have to migrate a
lot of data and I have to manage somehow that my application can use (r+w)
both CS + Scylla at the same time to not run any risk of data loss or dead
end road if something goes wrong. And still: I would not be able to compare
CS + Scylla for my workload totally fair as the conditions changed. New
hardware, maybe partial dataset, probably only "test traffic".

However, if I was able to just replace a single node in an existing cluster
I'd have:
1. Superlow hurdle to give it a try: No risk, no effort
2. Fair comparison by comparing new node against some equally equipeed old
node in the same cluster with the same workload
3. Easy to make a decision if to continue or not

That would be totally awesome!


2017-03-12 23:16 GMT+01:00 Kant Kodali :

> I don't think ScyallDB guys started this conversation in the first place
> to suggest or promote "drop-in replacement". It was something that is
> brought up by one of the Cassandra users and ScyallDB guys just clarified
> it. They are gracious enough to share the internals in detail.
>
> honestly, I find it weird when I see questions like whether a question
> belongs  to a mailing list or not especially in this case. If one doesn't
> like it they can simply not follow the thread. I am not sure what is the
> harm here.
>
>
>
> On Sun, Mar 12, 2017 at 2:29 PM, James Carman 
> wrote:
>
>> Well, looking back, it appears this thread is from 2015, so apparently
>> everyone is okay with it.
>>
>> Promoting a value-add product that makes using Cassandra easier/more
>> efficient/etc would be cool, but coming to the Cassandra mailing list to
>> promote a "drop-in replacement" (use us, not Cassandra) isn't cool, IMHO.
>>
>>
>> On Sun, Mar 12, 2017 at 5:04 PM Kant Kodali  wrote:
>>
>> yes.
>>
>> On Sun, Mar 12, 2017 at 2:01 PM, James Carman > > wrote:
>>
>> Does all of this Scylla talk really even belong on the Cassandra user
>> mailing list in the first place?
>>
>>
>>
>>
>> On Sun, Mar 12, 2017 at 4:07 PM Jeff Jirsa  wrote:
>>
>>
>>
>> On 2017-03-11 22:33 (-0700), Dor Laor  wrote:
>> > On Sat, Mar 11, 2017 at 10:02 PM, Jeff Jirsa  wrote:
>> > > On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote:
>> > > > Cassanda vs Scylla is a valid comparison because they both are
>> > > compatible. Scylla is a drop-in replacement for Cassandra.
>> > >
>> > > No, they aren't, and no, it isn't
>> > >
>> >
>> > Jeff is angry with us for some reason. I don't know why, it's natural
>> that
>> > when  a new opponent there are objections and the proof lies on us.
>>
>> I'm not angry. When I'm angry I send emails with paragraphs of
>> expletives. It doesn't happen very often.
>>
>> This is an open source ASF project, it's not about fighting for market
>> share against startups who find it necessary to inflate their level of
>> compatibility to sell support contracts, it's about providing software that
>> people can use (with a license that makes it easy to use). I don't work for
>> a company that makes money selling Cassandra based solutions and you're not
>> an opponent.
>>
>> >
>> > Scylla IS a drop in replacement for C*. We support the same CQL (from
>> > version 1.7 it's cql 3.3.1, protocol v4), the same SStable format
>> (based on
>> > 2.1.8).
>>
>> Scylla doesn't even run on all of the supported operating systems, let
>> alone have feature parity or network level compatibility (which you'd
>> probably need if you REALLY want to be drop-in
>> stop-one-cassandra-node-swap-binaries-start-it-up compatible, which is
>> what your site used to claim, but obviously isn't supported). You support a
>> subset of one query language and can read and write one sstable format. You
>> do it with great supporting tech and a great engineering team, but you're
>> not compatible, and if I were your cofounder I'd ask you to focus on the
>> tech strengths and not your drop-in compatibility, so engineers who care
>> about facts don't grow to resent your public lies.
>>
>> I've used a lot of databases in my life, but I don't know that I've ever
>> had someone call me angry because I pointed out that database A wasn't
>> compatible with database B, but I guess I'll chalk it up to 2017 and the
>> year of fake news / alternative facts.
>>
>> Hugs and kisses,
>> - Jeff
>>
>>
>>
>

Re: scylladb

[Kant Kodali]

I don't think ScyallDB guys started this conversation in the first place to
suggest or promote "drop-in replacement". It was something that is brought
up by one of the Cassandra users and ScyallDB guys just clarified it. They
are gracious enough to share the internals in detail.

honestly, I find it weird when I see questions like whether a question
belongs  to a mailing list or not especially in this case. If one doesn't
like it they can simply not follow the thread. I am not sure what is the
harm here.



On Sun, Mar 12, 2017 at 2:29 PM, James Carman 
wrote:

> Well, looking back, it appears this thread is from 2015, so apparently
> everyone is okay with it.
>
> Promoting a value-add product that makes using Cassandra easier/more
> efficient/etc would be cool, but coming to the Cassandra mailing list to
> promote a "drop-in replacement" (use us, not Cassandra) isn't cool, IMHO.
>
>
> On Sun, Mar 12, 2017 at 5:04 PM Kant Kodali  wrote:
>
> yes.
>
> On Sun, Mar 12, 2017 at 2:01 PM, James Carman 
> wrote:
>
> Does all of this Scylla talk really even belong on the Cassandra user
> mailing list in the first place?
>
>
>
>
> On Sun, Mar 12, 2017 at 4:07 PM Jeff Jirsa  wrote:
>
>
>
> On 2017-03-11 22:33 (-0700), Dor Laor  wrote:
> > On Sat, Mar 11, 2017 at 10:02 PM, Jeff Jirsa  wrote:
> > > On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote:
> > > > Cassanda vs Scylla is a valid comparison because they both are
> > > compatible. Scylla is a drop-in replacement for Cassandra.
> > >
> > > No, they aren't, and no, it isn't
> > >
> >
> > Jeff is angry with us for some reason. I don't know why, it's natural
> that
> > when  a new opponent there are objections and the proof lies on us.
>
> I'm not angry. When I'm angry I send emails with paragraphs of expletives.
> It doesn't happen very often.
>
> This is an open source ASF project, it's not about fighting for market
> share against startups who find it necessary to inflate their level of
> compatibility to sell support contracts, it's about providing software that
> people can use (with a license that makes it easy to use). I don't work for
> a company that makes money selling Cassandra based solutions and you're not
> an opponent.
>
> >
> > Scylla IS a drop in replacement for C*. We support the same CQL (from
> > version 1.7 it's cql 3.3.1, protocol v4), the same SStable format (based
> on
> > 2.1.8).
>
> Scylla doesn't even run on all of the supported operating systems, let
> alone have feature parity or network level compatibility (which you'd
> probably need if you REALLY want to be drop-in 
> stop-one-cassandra-node-swap-binaries-start-it-up
> compatible, which is what your site used to claim, but obviously isn't
> supported). You support a subset of one query language and can read and
> write one sstable format. You do it with great supporting tech and a great
> engineering team, but you're not compatible, and if I were your cofounder
> I'd ask you to focus on the tech strengths and not your drop-in
> compatibility, so engineers who care about facts don't grow to resent your
> public lies.
>
> I've used a lot of databases in my life, but I don't know that I've ever
> had someone call me angry because I pointed out that database A wasn't
> compatible with database B, but I guess I'll chalk it up to 2017 and the
> year of fake news / alternative facts.
>
> Hugs and kisses,
> - Jeff
>
>
>

Re: scylladb

[Jeff Jirsa]

On 2017-03-12 14:29 (-0700), James Carman  wrote: 
> Well, looking back, it appears this thread is from 2015, so apparently
> everyone is okay with it.
> 
> Promoting a value-add product that makes using Cassandra easier/more
> efficient/etc would be cool, but coming to the Cassandra mailing list to
> promote a "drop-in replacement" (use us, not Cassandra) isn't cool, IMHO.
> 

Agreed. Questions about scylla internals belong on scylla lists. user@cassandra 
is meant for questions about Apache Cassandra - let's please not revive 2015 
threads, and let's keep mails to the list on topic for the list. 

Re: scylladb

[James Carman]

Well, looking back, it appears this thread is from 2015, so apparently
everyone is okay with it.

Promoting a value-add product that makes using Cassandra easier/more
efficient/etc would be cool, but coming to the Cassandra mailing list to
promote a "drop-in replacement" (use us, not Cassandra) isn't cool, IMHO.


On Sun, Mar 12, 2017 at 5:04 PM Kant Kodali  wrote:

yes.

On Sun, Mar 12, 2017 at 2:01 PM, James Carman 
wrote:

Does all of this Scylla talk really even belong on the Cassandra user
mailing list in the first place?




On Sun, Mar 12, 2017 at 4:07 PM Jeff Jirsa  wrote:



On 2017-03-11 22:33 (-0700), Dor Laor  wrote:
> On Sat, Mar 11, 2017 at 10:02 PM, Jeff Jirsa  wrote:
> > On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote:
> > > Cassanda vs Scylla is a valid comparison because they both are
> > compatible. Scylla is a drop-in replacement for Cassandra.
> >
> > No, they aren't, and no, it isn't
> >
>
> Jeff is angry with us for some reason. I don't know why, it's natural that
> when  a new opponent there are objections and the proof lies on us.

I'm not angry. When I'm angry I send emails with paragraphs of expletives.
It doesn't happen very often.

This is an open source ASF project, it's not about fighting for market
share against startups who find it necessary to inflate their level of
compatibility to sell support contracts, it's about providing software that
people can use (with a license that makes it easy to use). I don't work for
a company that makes money selling Cassandra based solutions and you're not
an opponent.

>
> Scylla IS a drop in replacement for C*. We support the same CQL (from
> version 1.7 it's cql 3.3.1, protocol v4), the same SStable format (based
on
> 2.1.8).

Scylla doesn't even run on all of the supported operating systems, let
alone have feature parity or network level compatibility (which you'd
probably need if you REALLY want to be drop-in
stop-one-cassandra-node-swap-binaries-start-it-up compatible, which is what
your site used to claim, but obviously isn't supported). You support a
subset of one query language and can read and write one sstable format. You
do it with great supporting tech and a great engineering team, but you're
not compatible, and if I were your cofounder I'd ask you to focus on the
tech strengths and not your drop-in compatibility, so engineers who care
about facts don't grow to resent your public lies.

I've used a lot of databases in my life, but I don't know that I've ever
had someone call me angry because I pointed out that database A wasn't
compatible with database B, but I guess I'll chalk it up to 2017 and the
year of fake news / alternative facts.

Hugs and kisses,
- Jeff

Re: scylladb

[Kant Kodali]

yes.

On Sun, Mar 12, 2017 at 2:01 PM, James Carman 
wrote:

> Does all of this Scylla talk really even belong on the Cassandra user
> mailing list in the first place?
>
>
>
>
> On Sun, Mar 12, 2017 at 4:07 PM Jeff Jirsa  wrote:
>
>
>
> On 2017-03-11 22:33 (-0700), Dor Laor  wrote:
> > On Sat, Mar 11, 2017 at 10:02 PM, Jeff Jirsa  wrote:
> > > On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote:
> > > > Cassanda vs Scylla is a valid comparison because they both are
> > > compatible. Scylla is a drop-in replacement for Cassandra.
> > >
> > > No, they aren't, and no, it isn't
> > >
> >
> > Jeff is angry with us for some reason. I don't know why, it's natural
> that
> > when  a new opponent there are objections and the proof lies on us.
>
> I'm not angry. When I'm angry I send emails with paragraphs of expletives.
> It doesn't happen very often.
>
> This is an open source ASF project, it's not about fighting for market
> share against startups who find it necessary to inflate their level of
> compatibility to sell support contracts, it's about providing software that
> people can use (with a license that makes it easy to use). I don't work for
> a company that makes money selling Cassandra based solutions and you're not
> an opponent.
>
> >
> > Scylla IS a drop in replacement for C*. We support the same CQL (from
> > version 1.7 it's cql 3.3.1, protocol v4), the same SStable format (based
> on
> > 2.1.8).
>
> Scylla doesn't even run on all of the supported operating systems, let
> alone have feature parity or network level compatibility (which you'd
> probably need if you REALLY want to be drop-in 
> stop-one-cassandra-node-swap-binaries-start-it-up
> compatible, which is what your site used to claim, but obviously isn't
> supported). You support a subset of one query language and can read and
> write one sstable format. You do it with great supporting tech and a great
> engineering team, but you're not compatible, and if I were your cofounder
> I'd ask you to focus on the tech strengths and not your drop-in
> compatibility, so engineers who care about facts don't grow to resent your
> public lies.
>
> I've used a lot of databases in my life, but I don't know that I've ever
> had someone call me angry because I pointed out that database A wasn't
> compatible with database B, but I guess I'll chalk it up to 2017 and the
> year of fake news / alternative facts.
>
> Hugs and kisses,
> - Jeff
>
>

Re: scylladb

[James Carman]

Does all of this Scylla talk really even belong on the Cassandra user
mailing list in the first place?




On Sun, Mar 12, 2017 at 4:07 PM Jeff Jirsa  wrote:



On 2017-03-11 22:33 (-0700), Dor Laor  wrote:
> On Sat, Mar 11, 2017 at 10:02 PM, Jeff Jirsa  wrote:
> > On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote:
> > > Cassanda vs Scylla is a valid comparison because they both are
> > compatible. Scylla is a drop-in replacement for Cassandra.
> >
> > No, they aren't, and no, it isn't
> >
>
> Jeff is angry with us for some reason. I don't know why, it's natural that
> when  a new opponent there are objections and the proof lies on us.

I'm not angry. When I'm angry I send emails with paragraphs of expletives.
It doesn't happen very often.

This is an open source ASF project, it's not about fighting for market
share against startups who find it necessary to inflate their level of
compatibility to sell support contracts, it's about providing software that
people can use (with a license that makes it easy to use). I don't work for
a company that makes money selling Cassandra based solutions and you're not
an opponent.

>
> Scylla IS a drop in replacement for C*. We support the same CQL (from
> version 1.7 it's cql 3.3.1, protocol v4), the same SStable format (based
on
> 2.1.8).

Scylla doesn't even run on all of the supported operating systems, let
alone have feature parity or network level compatibility (which you'd
probably need if you REALLY want to be drop-in
stop-one-cassandra-node-swap-binaries-start-it-up compatible, which is what
your site used to claim, but obviously isn't supported). You support a
subset of one query language and can read and write one sstable format. You
do it with great supporting tech and a great engineering team, but you're
not compatible, and if I were your cofounder I'd ask you to focus on the
tech strengths and not your drop-in compatibility, so engineers who care
about facts don't grow to resent your public lies.

I've used a lot of databases in my life, but I don't know that I've ever
had someone call me angry because I pointed out that database A wasn't
compatible with database B, but I guess I'll chalk it up to 2017 and the
year of fake news / alternative facts.

Hugs and kisses,
- Jeff

Re: scylladb

[Edward Capriolo]

On Sun, Mar 12, 2017 at 3:45 PM, Dor Laor <d...@scylladb.com> wrote:

> On Sun, Mar 12, 2017 at 12:11 PM, Edward Capriolo <edlinuxg...@gmail.com>
> wrote:
>
>> The simple claim that "Scylla IS a drop in replacement for C*" shows
>> that they clearly don't know as much as they think they do.
>>
>> Even if it did supposedly "support everything" it would not actually work
>> like that. For example, some things in Cassandra work "the way they work" .
>> They are not specifically defined in a unit test or a document that
>> describes how they are supposed to work. During a massive code port you can
>> not reason if the code still works the same way in all situations.
>>
>> Example, without using SEDA and using something else it definitely wont
>> work the same way when the thread pools fill up and it starts blocking,
>> dropping, whatever. There is so much implicitly undefined behavior.
>>
>
> According to your definition there is no such a thing as drop and
> replacement, doesn't it?
>
> One of our users asked us to add a protocol verb that identify Scylla as
> Scylla so they'll know which
> is which for the time they run 2 clusters.
>
> Look, if we'll claim we have all the features and when someone checks they
> see we don't have LWT then it makes us a bad service. Usually when we get
> someone (specific) interested, we map their C* usage and say what feature
> isn't yet there. So far it's just lack of those not-implemented yet
> features that hold users back. We do try to mimic the exact behaviour of C*.
>
> Clearly, I can't defend a 100% drop-in replacement. Once we implement
> someone's selected
> featureset, then we're a drop-in replacement for them and we're not a good
> match for others.
> We're not after quick wins, quite the opposite.
>
>
>> Also just for argument sake. YCSB proves nothing. Nothing. It generates
>> key-value data, and well frankly that is not the primary use case of
>> Cassandra. So again. Know what you don't know.
>>
>>
> a. We do not pretend we know it all.
> We do have a 3 year mileage with Cassandra and 2.5 with Scylla and we
> gained some knowledge... before we decided to go after the C* path, we
> considered
> to reimplement Mongo, HDFS, Kafka and few more examples and the fact
> we chose
> C* shows our appreciation to this project and not the opposite.
>
> b. YCSB is an industry standard, and that's why everybody use it.
> We don't like it at all since it doesn't have prepared statements
> (it's time that
> someone will merge this support).
> It's not a plain K/V since it's a table of 10 columns of 100b each.
> We do support wide rows and learned (the hard way) their challenge,
> especially
> with compaction, repair and streaming. The current Scylla code doesn't
> cache
> wide row beyond 10MB which isn't ideal. In 1.8 (next month) we have a
> partial
> row caching which supposed to be very good. During the past 20 months
> since
> our beta we tried to focus on good out-of-the-box experience to all
> real workloads
> and we knowingly deferred features like LWT since we wanted a good
> solid base
> before we reach feature parity. If we'll do a good job with a
> benchmark but a bad
> one in real workload, we just shot ourselves in the foot. This was the
> case around our
> beta but it was just a beta. Today we think we're in a very solid
> position. We still
> have lots to complete around repair (which is ok but not great). There
> is a work
> in progress to switch out from Merkle tree to a new algorithm and
> reduced latency
> (almost there). We have mixed feelings about anti-compaction for
> incremental repair
> but we're likely to go through this route too
>
>
>>
>>
>>
>> On Sun, Mar 12, 2017 at 2:15 PM, Jonathan Haddad <j...@jonhaddad.com>
>> wrote:
>>
>>> I don't think Jeff comes across as angry.  He's simply pointing out that
>>> ScyllaDB isn't a drop in replacement for Cassandra.  Saying that it is is
>>> very misleading.  The marketing material should really say something like
>>> "drop in replacement for some workloads" or "aims to be a drop in
>>> replacement".  As is, it doesn't support everything, so it's not a drop in.
>>>
>>>
>>> On Sat, Mar 11, 2017 at 10:34 PM Dor Laor <d...@scylladb.com> wrote:
>>>
>>>> On Sat, Mar 11, 2017 at 10:02 PM, Jeff Jirsa <jji...@gmail.com> wrote:
>>>>
>>>>
>>>>
>>>> On 2017-03-10 09:57 (-0800), Rakesh Kum

Re: scylladb

[Jeff Jirsa]

On 2017-03-11 22:33 (-0700), Dor Laor  wrote: 
> On Sat, Mar 11, 2017 at 10:02 PM, Jeff Jirsa  wrote:
> > On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote:
> > > Cassanda vs Scylla is a valid comparison because they both are
> > compatible. Scylla is a drop-in replacement for Cassandra.
> >
> > No, they aren't, and no, it isn't
> >
> 
> Jeff is angry with us for some reason. I don't know why, it's natural that
> when  a new opponent there are objections and the proof lies on us.

I'm not angry. When I'm angry I send emails with paragraphs of expletives. It 
doesn't happen very often. 

This is an open source ASF project, it's not about fighting for market share 
against startups who find it necessary to inflate their level of compatibility 
to sell support contracts, it's about providing software that people can use 
(with a license that makes it easy to use). I don't work for a company that 
makes money selling Cassandra based solutions and you're not an opponent.

> 
> Scylla IS a drop in replacement for C*. We support the same CQL (from
> version 1.7 it's cql 3.3.1, protocol v4), the same SStable format (based on
> 2.1.8). 

Scylla doesn't even run on all of the supported operating systems, let alone 
have feature parity or network level compatibility (which you'd probably need 
if you REALLY want to be drop-in 
stop-one-cassandra-node-swap-binaries-start-it-up compatible, which is what 
your site used to claim, but obviously isn't supported). You support a subset 
of one query language and can read and write one sstable format. You do it with 
great supporting tech and a great engineering team, but you're not compatible, 
and if I were your cofounder I'd ask you to focus on the tech strengths and not 
your drop-in compatibility, so engineers who care about facts don't grow to 
resent your public lies.

I've used a lot of databases in my life, but I don't know that I've ever had 
someone call me angry because I pointed out that database A wasn't compatible 
with database B, but I guess I'll chalk it up to 2017 and the year of fake news 
/ alternative facts. 

Hugs and kisses,
- Jeff

Re: scylladb

[Dor Laor]

On Sun, Mar 12, 2017 at 12:11 PM, Edward Capriolo <edlinuxg...@gmail.com>
wrote:

> The simple claim that "Scylla IS a drop in replacement for C*" shows that
> they clearly don't know as much as they think they do.
>
> Even if it did supposedly "support everything" it would not actually work
> like that. For example, some things in Cassandra work "the way they work" .
> They are not specifically defined in a unit test or a document that
> describes how they are supposed to work. During a massive code port you can
> not reason if the code still works the same way in all situations.
>
> Example, without using SEDA and using something else it definitely wont
> work the same way when the thread pools fill up and it starts blocking,
> dropping, whatever. There is so much implicitly undefined behavior.
>

According to your definition there is no such a thing as drop and
replacement, doesn't it?

One of our users asked us to add a protocol verb that identify Scylla as
Scylla so they'll know which
is which for the time they run 2 clusters.

Look, if we'll claim we have all the features and when someone checks they
see we don't have LWT then it makes us a bad service. Usually when we get
someone (specific) interested, we map their C* usage and say what feature
isn't yet there. So far it's just lack of those not-implemented yet
features that hold users back. We do try to mimic the exact behaviour of C*.

Clearly, I can't defend a 100% drop-in replacement. Once we implement
someone's selected
featureset, then we're a drop-in replacement for them and we're not a good
match for others.
We're not after quick wins, quite the opposite.


> Also just for argument sake. YCSB proves nothing. Nothing. It generates
> key-value data, and well frankly that is not the primary use case of
> Cassandra. So again. Know what you don't know.
>
>
a. We do not pretend we know it all.
We do have a 3 year mileage with Cassandra and 2.5 with Scylla and we
gained some knowledge... before we decided to go after the C* path, we
considered
to reimplement Mongo, HDFS, Kafka and few more examples and the fact we
chose
C* shows our appreciation to this project and not the opposite.

b. YCSB is an industry standard, and that's why everybody use it.
We don't like it at all since it doesn't have prepared statements (it's
time that
someone will merge this support).
It's not a plain K/V since it's a table of 10 columns of 100b each.
We do support wide rows and learned (the hard way) their challenge,
especially
with compaction, repair and streaming. The current Scylla code doesn't
cache
wide row beyond 10MB which isn't ideal. In 1.8 (next month) we have a
partial
row caching which supposed to be very good. During the past 20 months
since
our beta we tried to focus on good out-of-the-box experience to all
real workloads
and we knowingly deferred features like LWT since we wanted a good
solid base
before we reach feature parity. If we'll do a good job with a benchmark
but a bad
one in real workload, we just shot ourselves in the foot. This was the
case around our
beta but it was just a beta. Today we think we're in a very solid
position. We still
have lots to complete around repair (which is ok but not great). There
is a work
in progress to switch out from Merkle tree to a new algorithm and
reduced latency
(almost there). We have mixed feelings about anti-compaction for
incremental repair
but we're likely to go through this route too


>
>
>
> On Sun, Mar 12, 2017 at 2:15 PM, Jonathan Haddad <j...@jonhaddad.com>
> wrote:
>
>> I don't think Jeff comes across as angry.  He's simply pointing out that
>> ScyllaDB isn't a drop in replacement for Cassandra.  Saying that it is is
>> very misleading.  The marketing material should really say something like
>> "drop in replacement for some workloads" or "aims to be a drop in
>> replacement".  As is, it doesn't support everything, so it's not a drop in.
>>
>>
>> On Sat, Mar 11, 2017 at 10:34 PM Dor Laor <d...@scylladb.com> wrote:
>>
>>> On Sat, Mar 11, 2017 at 10:02 PM, Jeff Jirsa <jji...@gmail.com> wrote:
>>>
>>>
>>>
>>> On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote:
>>> > Cassanda vs Scylla is a valid comparison because they both are
>>> compatible. Scylla is a drop-in replacement for Cassandra.
>>>
>>> No, they aren't, and no, it isn't
>>>
>>>
>>> Jeff is angry with us for some reason. I don't know why, it's natural
>>> that when
>>> a new opponent there are objections and the proof lies on us.
>>> We go through great deal of doing it and we don't just throw comments
>>> without back

Re: scylladb

[Edward Capriolo]

The simple claim that "Scylla IS a drop in replacement for C*" shows that
they clearly don't know as much as they think they do.

Even if it did supposedly "support everything" it would not actually work
like that. For example, some things in Cassandra work "the way they work" .
They are not specifically defined in a unit test or a document that
describes how they are supposed to work. During a massive code port you can
not reason if the code still works the same way in all situations.

Example, without using SEDA and using something else it definitely wont
work the same way when the thread pools fill up and it starts blocking,
dropping, whatever. There is so much implicitly undefined behavior.

Also just for argument sake. YCSB proves nothing. Nothing. It generates
key-value data, and well frankly that is not the primary use case of
Cassandra. So again. Know what you don't know.





On Sun, Mar 12, 2017 at 2:15 PM, Jonathan Haddad <j...@jonhaddad.com> wrote:

> I don't think Jeff comes across as angry.  He's simply pointing out that
> ScyllaDB isn't a drop in replacement for Cassandra.  Saying that it is is
> very misleading.  The marketing material should really say something like
> "drop in replacement for some workloads" or "aims to be a drop in
> replacement".  As is, it doesn't support everything, so it's not a drop in.
>
>
> On Sat, Mar 11, 2017 at 10:34 PM Dor Laor <d...@scylladb.com> wrote:
>
>> On Sat, Mar 11, 2017 at 10:02 PM, Jeff Jirsa <jji...@gmail.com> wrote:
>>
>>
>>
>> On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote:
>> > Cassanda vs Scylla is a valid comparison because they both are
>> compatible. Scylla is a drop-in replacement for Cassandra.
>>
>> No, they aren't, and no, it isn't
>>
>>
>> Jeff is angry with us for some reason. I don't know why, it's natural
>> that when
>> a new opponent there are objections and the proof lies on us.
>> We go through great deal of doing it and we don't just throw comments
>> without backing.
>>
>> Scylla IS a drop in replacement for C*. We support the same CQL (from
>> version 1.7 it's cql 3.3.1, protocol v4), the same SStable format (based on
>> 2.1.8). In 1.7 release we support cql uploader
>> from 3.x. We will support the SStable format of 3.x natively in 3 month
>> time. Soon all of the feature set will be implemented. We always have been
>> using this page (not 100% up to date, we'll update it this week):
>> http://www.scylladb.com/technology/status/
>>
>> We add a jmx-proxy daemon in java in order to make the transition as
>> smooth as possible. Almost all the nodetool commands just work, for sure
>> all the important ones.
>> Btw: we have a RESTapi and Prometheus formats, much better than the hairy
>> jmx one.
>>
>> Spark, Kairosdb, Presto and probably Titan (we add Thrift just for legacy
>> users and we don't intend
>> to decommission an api).
>>
>> Regarding benchmarks, if someone finds a flaw in them, we'll do the best
>> to fix it.
>> Let's ignore them and just here what our users have to say:
>> http://www.scylladb.com/users/
>>
>>
>>

Re: scylladb

[Dor Laor]

On Sun, Mar 12, 2017 at 11:15 AM, Jonathan Haddad <j...@jonhaddad.com> wrote:

> I don't think Jeff comes across as angry.  He's simply pointing out that
> ScyllaDB isn't a drop in
>

Agree, I take it back, it's wasn't due to this.


> replacement for Cassandra.  Saying that it is is very misleading.  The
> marketing material should really say something like "drop in replacement
> for some workloads" or "aims to be a drop in replacement".  As is, it
> doesn't support everything, so it's not a drop in.
>
>
When we need to describe what Scylla is in 140 characters or one liner, we
use drop-in-replacement. When we talk about the details, we provide the
full details as I did above.
The code is open and we take the upstream-first approach and there is the
status page
to summarize it. If someone depends on LWT or UDF we don't have an
immediate answer.
We do have answers for the rest. The vast majority of users don't get to
use these features
and thus they can (and some did) seamlessly migrate.

For a reference sanity check, see all the databases/tools who claim SQL
ability, most of them
don't comply to the ANSI standard. As you said, our desire is to be 100%
compatible.

Btw, going back to technology discussion, while there are lots of reasons
to use C++, the only
challenge is in features like UDF/triggers which relay on JVM based code
execution. We are likely to use Lua for it initially, and later we'll
integrate it with a JVM based solution.



>
> On Sat, Mar 11, 2017 at 10:34 PM Dor Laor <d...@scylladb.com> wrote:
>
>> On Sat, Mar 11, 2017 at 10:02 PM, Jeff Jirsa <jji...@gmail.com> wrote:
>>
>>
>>
>> On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote:
>> > Cassanda vs Scylla is a valid comparison because they both are
>> compatible. Scylla is a drop-in replacement for Cassandra.
>>
>> No, they aren't, and no, it isn't
>>
>>
>> Jeff is angry with us for some reason. I don't know why, it's natural
>> that when
>> a new opponent there are objections and the proof lies on us.
>> We go through great deal of doing it and we don't just throw comments
>> without backing.
>>
>> Scylla IS a drop in replacement for C*. We support the same CQL (from
>> version 1.7 it's cql 3.3.1, protocol v4), the same SStable format (based on
>> 2.1.8). In 1.7 release we support cql uploader
>> from 3.x. We will support the SStable format of 3.x natively in 3 month
>> time. Soon all of the feature set will be implemented. We always have been
>> using this page (not 100% up to date, we'll update it this week):
>> http://www.scylladb.com/technology/status/
>>
>> We add a jmx-proxy daemon in java in order to make the transition as
>> smooth as possible. Almost all the nodetool commands just work, for sure
>> all the important ones.
>> Btw: we have a RESTapi and Prometheus formats, much better than the hairy
>> jmx one.
>>
>> Spark, Kairosdb, Presto and probably Titan (we add Thrift just for legacy
>> users and we don't intend
>> to decommission an api).
>>
>> Regarding benchmarks, if someone finds a flaw in them, we'll do the best
>> to fix it.
>> Let's ignore them and just here what our users have to say:
>> http://www.scylladb.com/users/
>>
>>
>>

Re: scylladb

[Dor Laor]

On Sun, Mar 12, 2017 at 6:40 AM, Stefan Podkowinski <s...@apache.org> wrote:

> If someone would create a benchmark showing that Cassandra is 10x faster
> than Aerospike, would that mean Cassandra is 100x faster than ScyllaDB?
>
> Joking aside, I personally don't pay a lot of attention to any published
> benchmarks and look at them as pure marketing material. What I'm interested
> in instead is to learn why exactly one solution is faster than the other
> and I have to say that Avi is doing a really good job explaining the design
> motivations behind ScyllaDB in his presentations.
>
> But the Aerospike comparison also has a good point by showing that you
> probably always will be able to find a solution that is faster for a
> certain work load. Therefor the most important step when looking for the
> fastest datastore, is to first really understand your work load
> characteristic. Unfortunately this is something people tend to skip and
> instead get lost in controversial benchmark discussions, which are more fun
> than thinking about your data model and talking to people about projected
> long term load. Because if you do, you might realize that those benchmark
> test scenarios (e.g. insert 1TB as fast as possible and measure compaction
> times) aren't actually that relevant for your application.
>
Agree, however, it allows you to realize what a real workload will suffer
from and that's why we
measured a 'read while heavily writing' result too. In addition we measured
small, medium and large datasets for read only. Still, benchmarks are not a
real workload and we always advise to use our Prometheus detailed metrics
to realize if the hardware is utilized and to understand what's the
bottleneck. Scylla implemented the CQL tracing and can run the slow query
tracing all of the time with a low performance impact



>
> On 03/10/2017 05:58 PM, Bhuvan Rawal wrote:
>
> Agreed C++ gives an added advantage to talk to underlying hardware with
> better efficiency, it sound good but can a pice of code written in C++ give
> 1000% throughput than a Java app? Is TPC design 10X more performant than
> SEDA arch?
>
> And if C/C++ is indeed that fast how can Aerospike (which is itself
> written in C) claim to be 10X faster than Scylla here
> http://www.aerospike.com/benchmarks/scylladb-initial/ ? (Combining your's
> and aerospike's benchmarks it appears that Aerospike is 100X performant
> than C* - I highly doubt that!! )
>
> For a moment lets forget about evaluating 2 different databases, one can
> observe 10X performance difference between a mistuned cassandra cluster and
> one thats tuned as per data model - there are so many Tunables in yaml as
> well as table configs.
>
> Idea is - in order to strengthen your claim, you need to provide complete
> system metrics (Disk, CPU, Network), the OPS increase starts to decay along
> with the configs used. Having plain ops per second and 99p latency is
> blackbox.
>
> Regards,
> Bhuvan
>
> On Fri, Mar 10, 2017 at 12:47 PM, Avi Kivity <a...@scylladb.com> wrote:
>
>> ScyllaDB engineer here.
>>
>> C++ is really an enabling technology here. It is directly responsible for
>> a small fraction of the gain by executing faster than Java.  But it is
>> indirectly responsible for the gain by allowing us direct control over
>> memory and threading.  Just as an example, Scylla starts by taking over
>> almost all of the machine's memory, and dynamically assigning it to
>> memtables, cache, and working memory needed to handle requests in flight.
>> Memory is statically partitioned across cores, allowing us to exploit NUMA
>> fully.  You can't do these things in Java.
>>
>> I would say the major contributors to Scylla performance are:
>>  - thread-per-core design
>>  - replacement of the page cache with a row cache
>>  - careful attention to many small details, each contributing a little,
>> but with a large overall impact
>>
>> While I'm here I can say that performance is not the only goal here, it
>> is stable and predictable performance over varying loads and during
>> maintenance operations like repair, without any special tuning.  We measure
>> the amount of CPU and I/O spent on foreground (user) and background
>> (maintenance) tasks and divide them fairly.  This work is not complete but
>> already makes operating Scylla a lot simpler.
>>
>>
>> On 03/10/2017 01:42 AM, Kant Kodali wrote:
>>
>> I dont think ScyllaDB performance is because of C++. The design decisions
>> in scylladb are indeed different from Cassandra such as getting rid of SEDA
>> and moving to TPC and so on.
>>
>> If someone thinks it is because of C++ then just show the benchmarks

Re: scylladb

[Jonathan Haddad]

I don't think Jeff comes across as angry.  He's simply pointing out that
ScyllaDB isn't a drop in replacement for Cassandra.  Saying that it is is
very misleading.  The marketing material should really say something like
"drop in replacement for some workloads" or "aims to be a drop in
replacement".  As is, it doesn't support everything, so it's not a drop in.


On Sat, Mar 11, 2017 at 10:34 PM Dor Laor <d...@scylladb.com> wrote:

> On Sat, Mar 11, 2017 at 10:02 PM, Jeff Jirsa <jji...@gmail.com> wrote:
>
>
>
> On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote:
> > Cassanda vs Scylla is a valid comparison because they both are
> compatible. Scylla is a drop-in replacement for Cassandra.
>
> No, they aren't, and no, it isn't
>
>
> Jeff is angry with us for some reason. I don't know why, it's natural that
> when
> a new opponent there are objections and the proof lies on us.
> We go through great deal of doing it and we don't just throw comments
> without backing.
>
> Scylla IS a drop in replacement for C*. We support the same CQL (from
> version 1.7 it's cql 3.3.1, protocol v4), the same SStable format (based on
> 2.1.8). In 1.7 release we support cql uploader
> from 3.x. We will support the SStable format of 3.x natively in 3 month
> time. Soon all of the feature set will be implemented. We always have been
> using this page (not 100% up to date, we'll update it this week):
> http://www.scylladb.com/technology/status/
>
> We add a jmx-proxy daemon in java in order to make the transition as
> smooth as possible. Almost all the nodetool commands just work, for sure
> all the important ones.
> Btw: we have a RESTapi and Prometheus formats, much better than the hairy
> jmx one.
>
> Spark, Kairosdb, Presto and probably Titan (we add Thrift just for legacy
> users and we don't intend
> to decommission an api).
>
> Regarding benchmarks, if someone finds a flaw in them, we'll do the best
> to fix it.
> Let's ignore them and just here what our users have to say:
> http://www.scylladb.com/users/
>
>
>

RE: scylladb

[Jacques-Henri Berthemet]

Will you support custom secondary indexes, triggers and UDF?
I checked index code but it’s just a couple of files with commented Java code. 
I’m curious to test Scylladb but our application uses LWT and custom secondary 
indexes, I understand LWT is coming (soon?).

--
Jacques-Henri Berthemet

From: sfesc...@gmail.com [mailto:sfesc...@gmail.com]
Sent: dimanche 12 mars 2017 09:23
To: user@cassandra.apache.org
Subject: Re: scylladb


On Sat, Mar 11, 2017 at 1:52 AM Avi Kivity 
<a...@scylladb.com<mailto:a...@scylladb.com>> wrote:


Lastly, why don't you test Scylla yourself?  It's pretty easy to set up, 
there's nothing to tune.

Avi

 I'll look seriously at Scylla when it is 3.0.12 compatible.

Re: scylladb

[sfesc...@gmail.com]

On Sat, Mar 11, 2017 at 1:52 AM Avi Kivity  wrote:

>
>
> Lastly, why don't you test Scylla yourself?  It's pretty easy to set up,
> there's nothing to tune.
>
> Avi
>

 I'll look seriously at Scylla when it is 3.0.12 compatible.

Re: scylladb

[Edward Capriolo]

On Sun, Mar 12, 2017 at 11:40 AM, Edward Capriolo 
wrote:

>
>
> On Sun, Mar 12, 2017 at 1:38 AM, benjamin roth  wrote:
>
>> There is no reason to be angry. This is progress. This is the circle of
>> live.
>>
>> It happens anywhere at any time.
>>
>> Am 12.03.2017 07:34 schrieb "Dor Laor" :
>>
>>> On Sat, Mar 11, 2017 at 10:02 PM, Jeff Jirsa  wrote:
>>>


 On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote:
 > Cassanda vs Scylla is a valid comparison because they both are
 compatible. Scylla is a drop-in replacement for Cassandra.

 No, they aren't, and no, it isn't

>>>
>>> Jeff is angry with us for some reason. I don't know why, it's natural
>>> that when
>>> a new opponent there are objections and the proof lies on us.
>>> We go through great deal of doing it and we don't just throw comments
>>> without backing.
>>>
>>> Scylla IS a drop in replacement for C*. We support the same CQL (from
>>> version 1.7 it's cql 3.3.1, protocol v4), the same SStable format (based on
>>> 2.1.8). In 1.7 release we support cql uploader
>>> from 3.x. We will support the SStable format of 3.x natively in 3 month
>>> time. Soon all of the feature set will be implemented. We always have been
>>> using this page (not 100% up to date, we'll update it this week):
>>> http://www.scylladb.com/technology/status/
>>>
>>> We add a jmx-proxy daemon in java in order to make the transition as
>>> smooth as possible. Almost all the nodetool commands just work, for sure
>>> all the important ones.
>>> Btw: we have a RESTapi and Prometheus formats, much better than the
>>> hairy jmx one.
>>>
>>> Spark, Kairosdb, Presto and probably Titan (we add Thrift just for
>>> legacy users and we don't intend
>>> to decommission an api).
>>>
>>> Regarding benchmarks, if someone finds a flaw in them, we'll do the best
>>> to fix it.
>>> Let's ignore them and just here what our users have to say:
>>> http://www.scylladb.com/users/
>>>
>>>
>>>
>
> Scylla is NOT a drop in replacement for Cassandra. Cassandra is a TM.
> Cassandra is NOT a certification body. You are not a certification body.
>
> "Scylla IS a drop in replacement for C*. We support the same CQL (from
> version 1.7 it's cql 3.3.1, protocol v4), the same SStable format (based on
> 2.1.8). In 1.7 release we support cql uploader
> from 3.x. We will support the SStable format of 3.x natively in 3 month
> time. Soon all of the feature set will be implemented. We always have been
> using this page (not 100% up to date, we'll update it this week):
> http://www.scylladb.com/technology/status/ "
>
> No matter how "compatible" you believe Scylla is you can not assert this
> claim.
>
>
>
Also there is no reason to say Jeff is 'angry' because he asserted his
believe in fact.

"No, they aren't, and no, it isn't"

Does not sound angry. t

Besides that your your own words prove it:

"Scylla IS a drop in replacement for C*"
"Soon all of the feature set will be implemented"

Something is NOT a "drop in replacement" when it does NOT have all the
features.

Also knowing Jeff who is very even keel, I highly doubt because he made a
short concise statement he is "angry".

That being said I am little bit angry by the shameless self promotion and
Jabbering on you seem to be doing. We get it you know about kernels and
page faults and want to talk endlessly about it.

Re: scylladb

[Edward Capriolo]

On Sun, Mar 12, 2017 at 1:38 AM, benjamin roth  wrote:

> There is no reason to be angry. This is progress. This is the circle of
> live.
>
> It happens anywhere at any time.
>
> Am 12.03.2017 07:34 schrieb "Dor Laor" :
>
>> On Sat, Mar 11, 2017 at 10:02 PM, Jeff Jirsa  wrote:
>>
>>>
>>>
>>> On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote:
>>> > Cassanda vs Scylla is a valid comparison because they both are
>>> compatible. Scylla is a drop-in replacement for Cassandra.
>>>
>>> No, they aren't, and no, it isn't
>>>
>>
>> Jeff is angry with us for some reason. I don't know why, it's natural
>> that when
>> a new opponent there are objections and the proof lies on us.
>> We go through great deal of doing it and we don't just throw comments
>> without backing.
>>
>> Scylla IS a drop in replacement for C*. We support the same CQL (from
>> version 1.7 it's cql 3.3.1, protocol v4), the same SStable format (based on
>> 2.1.8). In 1.7 release we support cql uploader
>> from 3.x. We will support the SStable format of 3.x natively in 3 month
>> time. Soon all of the feature set will be implemented. We always have been
>> using this page (not 100% up to date, we'll update it this week):
>> http://www.scylladb.com/technology/status/
>>
>> We add a jmx-proxy daemon in java in order to make the transition as
>> smooth as possible. Almost all the nodetool commands just work, for sure
>> all the important ones.
>> Btw: we have a RESTapi and Prometheus formats, much better than the hairy
>> jmx one.
>>
>> Spark, Kairosdb, Presto and probably Titan (we add Thrift just for legacy
>> users and we don't intend
>> to decommission an api).
>>
>> Regarding benchmarks, if someone finds a flaw in them, we'll do the best
>> to fix it.
>> Let's ignore them and just here what our users have to say:
>> http://www.scylladb.com/users/
>>
>>
>>

Scylla is NOT a drop in replacement for Cassandra. Cassandra is a TM.
Cassandra is NOT a certification body. You are not a certification body.

"Scylla IS a drop in replacement for C*. We support the same CQL (from
version 1.7 it's cql 3.3.1, protocol v4), the same SStable format (based on
2.1.8). In 1.7 release we support cql uploader
from 3.x. We will support the SStable format of 3.x natively in 3 month
time. Soon all of the feature set will be implemented. We always have been
using this page (not 100% up to date, we'll update it this week):
http://www.scylladb.com/technology/status/ "

No matter how "compatible" you believe Scylla is you can not assert this
claim.

Re: scylladb

[Stefan Podkowinski]

If someone would create a benchmark showing that Cassandra is 10x faster
than Aerospike, would that mean Cassandra is 100x faster than ScyllaDB?

Joking aside, I personally don't pay a lot of attention to any published
benchmarks and look at them as pure marketing material. What I'm
interested in instead is to learn why exactly one solution is faster
than the other and I have to say that Avi is doing a really good job
explaining the design motivations behind ScyllaDB in his presentations.

But the Aerospike comparison also has a good point by showing that you
probably always will be able to find a solution that is faster for a
certain work load. Therefor the most important step when looking for the
fastest datastore, is to first really understand your work load
characteristic. Unfortunately this is something people tend to skip and
instead get lost in controversial benchmark discussions, which are more
fun than thinking about your data model and talking to people about
projected long term load. Because if you do, you might realize that
those benchmark test scenarios (e.g. insert 1TB as fast as possible and
measure compaction times) aren't actually that relevant for your
application.


On 03/10/2017 05:58 PM, Bhuvan Rawal wrote:
> Agreed C++ gives an added advantage to talk to underlying hardware
> with better efficiency, it sound good but can a pice of code written
> in C++ give 1000% throughput than a Java app? Is TPC design 10X more
> performant than SEDA arch?
>
> And if C/C++ is indeed that fast how can Aerospike (which is itself
> written in C) claim to be 10X faster than Scylla
> here http://www.aerospike.com/benchmarks/scylladb-initial/ ?
> (Combining your's and aerospike's benchmarks it appears that Aerospike
> is 100X performant than C* - I highly doubt that!! )
>
> For a moment lets forget about evaluating 2 different databases, one
> can observe 10X performance difference between a mistuned cassandra
> cluster and one thats tuned as per data model - there are so many
> Tunables in yaml as well as table configs.
>
> Idea is - in order to strengthen your claim, you need to provide
> complete system metrics (Disk, CPU, Network), the OPS increase starts
> to decay along with the configs used. Having plain ops per second and
> 99p latency is blackbox.
>
> Regards,
> Bhuvan
>
> On Fri, Mar 10, 2017 at 12:47 PM, Avi Kivity <a...@scylladb.com
> <mailto:a...@scylladb.com>> wrote:
>
> ScyllaDB engineer here.
>
> C++ is really an enabling technology here. It is directly
> responsible for a small fraction of the gain by executing faster
> than Java.  But it is indirectly responsible for the gain by
> allowing us direct control over memory and threading.  Just as an
> example, Scylla starts by taking over almost all of the machine's
> memory, and dynamically assigning it to memtables, cache, and
> working memory needed to handle requests in flight.  Memory is
> statically partitioned across cores, allowing us to exploit NUMA
> fully.  You can't do these things in Java.
>
> I would say the major contributors to Scylla performance are:
>  - thread-per-core design
>  - replacement of the page cache with a row cache
>  - careful attention to many small details, each contributing a
> little, but with a large overall impact
>
> While I'm here I can say that performance is not the only goal
> here, it is stable and predictable performance over varying loads
> and during maintenance operations like repair, without any special
> tuning.  We measure the amount of CPU and I/O spent on foreground
> (user) and background (maintenance) tasks and divide them fairly. 
>     This work is not complete but already makes operating Scylla a lot
> simpler.
>
>
> On 03/10/2017 01:42 AM, Kant Kodali wrote:
>> I dont think ScyllaDB performance is because of C++. The design
>> decisions in scylladb are indeed different from Cassandra such as
>> getting rid of SEDA and moving to TPC and so on. 
>>
>> If someone thinks it is because of C++ then just show the
>> benchmarks that proves it is indeed the C++ which gave 10X
>> performance boost as ScyllaDB claims instead of stating it.
>>
>>
>> On Thu, Mar 9, 2017 at 3:22 PM, Richard L. Burton III
>> <mrbur...@gmail.com <mailto:mrbur...@gmail.com>> wrote:
>>
>> They spend an enormous amount of time focusing on
>> performance. You can expect them to continue on with their
>> optimization and keep crushing it.
>>
>>     P.S., I don't work for ScyllaDB.  
>>
>> On Thu, Mar 9, 2017 at 6:02 PM, Rakesh Kumar
>> 

Re: scylladb

[Kant Kodali]

One more thing. Pretty much every database that is written in C++ or Java
uses native kernel threads for non-blocking I/O as well. They didn't use
Seaster or Quasar but anyways I am going to read up on Seaster and see what
it really does.

On Sun, Mar 12, 2017 at 3:48 AM, Kant Kodali  wrote:

>
> If you have thread-per-core and N (logical) cores, and have M tasks
>> running concurrently where M > N, then you need a scheduler to decide which
>> of those M tasks gets to run on those N kernel threads.  Whether those M
>> tasks are user-level threads, or callbacks, or a mix of the two is
>> immaterial.  In such cases a scheduler always exists, even if it is a
>> simple FIFO queue.
>>
>
>
>> yes ofcourse scheduler is needed. But what you said is immaterial is
>> where I see the devil or say our conflict of arguments really are. Let the
>> kernel thread per core deal with callbacks rather than having to build a
>> user-level thread library and its scheduling mechanisms and the mapping
>> between them. This sounds more of an overhead in general but may work in a
>> specific case.
>>
>
>

Re: scylladb

[Kant Kodali]

re second-order entities
> in the system. If a kernel thread uses a GDT slot for TLS data, a user
> thread perhaps can only use an LDT slot for TLS data. With increasingly
> more supports available from the new processors for threading/scheduling
> (Hyperthreading, NUMA, many-core), the second order nature seriously limits
> the ability of M:N threading.
>
> On Sun, Mar 12, 2017 at 1:15 AM, Kant Kodali <k...@peernova.com> wrote:
>
>> Hi Dor,
>>
>> I will reply to this on a separate thread since there seem to be good
>> knowledge exchange on this thread!
>>
>> Thanks!
>> kant
>>
>> On Sun, Mar 12, 2017 at 12:48 AM, Dor Laor <d...@scylladb.com> wrote:
>>
>>> Hi Kant,
>>>
>>> 2.0 is scheduled around July. Do you need LWT, it will be part of it.
>>> If you need other features, they'll be inside before. That status page
>>> will get refreshed this week.
>>>
>>> About gains, it depends on the hardware. With large physical machines
>>> (or the large i3.16xl) we can show 10X and beyond. With smaller machines
>>> it may be 1.5-2X but do check the c3.2xl (4 core machine) benchmark on
>>> our
>>> site, there are lots of factors we do better.
>>>
>>> Actually the right benchmark is the otherway around and will show larger
>>> value
>>> of Scylla - you define your workload in terms of OPS and latency and
>>> then you
>>> find the right set of hardware for Scylla and Cassandra and compare.
>>> I am *sure* that the difference is substantial.
>>>
>>> Can you please elaborate on your workload and the type of machines you
>>> use?
>>> Schema and row length plus access pattern are also welcomed.
>>>
>>> Cheers,
>>> Dor
>>>
>>>
>>> On Sun, Mar 12, 2017 at 12:13 AM, Kant Kodali <k...@peernova.com> wrote:
>>>
>>>> Progress is all that matters and I do think you guys had assembled a
>>>> smart team. Looking at that status page I am willing to try ScyllaDB
>>>> whenever 2.0 version is released. we have big banks as our clients and I
>>>> would most likely replace it if I see a 10X or some significant improvement
>>>> with our workloads but in case if it only gives me 1.5-2X I wouldn't be so
>>>> inclined to go through all the work.
>>>>
>>>> On Sat, Mar 11, 2017 at 10:38 PM, benjamin roth <brs...@gmail.com>
>>>> wrote:
>>>>
>>>>> There is no reason to be angry. This is progress. This is the circle
>>>>> of live.
>>>>>
>>>>> It happens anywhere at any time.
>>>>>
>>>>> Am 12.03.2017 07:34 schrieb "Dor Laor" <d...@scylladb.com>:
>>>>>
>>>>>> On Sat, Mar 11, 2017 at 10:02 PM, Jeff Jirsa <jji...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote:
>>>>>>> > Cassanda vs Scylla is a valid comparison because they both are
>>>>>>> compatible. Scylla is a drop-in replacement for Cassandra.
>>>>>>>
>>>>>>> No, they aren't, and no, it isn't
>>>>>>>
>>>>>>
>>>>>> Jeff is angry with us for some reason. I don't know why, it's natural
>>>>>> that when
>>>>>> a new opponent there are objections and the proof lies on us.
>>>>>> We go through great deal of doing it and we don't just throw comments
>>>>>> without backing.
>>>>>>
>>>>>> Scylla IS a drop in replacement for C*. We support the same CQL (from
>>>>>> version 1.7 it's cql 3.3.1, protocol v4), the same SStable format (based 
>>>>>> on
>>>>>> 2.1.8). In 1.7 release we support cql uploader
>>>>>> from 3.x. We will support the SStable format of 3.x natively in 3
>>>>>> month time. Soon all of the feature set will be implemented. We always 
>>>>>> have
>>>>>> been using this page (not 100% up to date, we'll update it this week):
>>>>>> http://www.scylladb.com/technology/status/
>>>>>>
>>>>>> We add a jmx-proxy daemon in java in order to make the transition as
>>>>>> smooth as possible. Almost all the nodetool commands just work, for sure
>>>>>> all the important ones.
>>>>>> Btw: we have a RESTapi and Prometheus formats, much better than the
>>>>>> hairy jmx one.
>>>>>>
>>>>>> Spark, Kairosdb, Presto and probably Titan (we add Thrift just for
>>>>>> legacy users and we don't intend
>>>>>> to decommission an api).
>>>>>>
>>>>>> Regarding benchmarks, if someone finds a flaw in them, we'll do the
>>>>>> best to fix it.
>>>>>> Let's ignore them and just here what our users have to say:
>>>>>> http://www.scylladb.com/users/
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>>
>

Re: scylladb

[Kant Kodali]

> If you have thread-per-core and N (logical) cores, and have M tasks
> running concurrently where M > N, then you need a scheduler to decide which
> of those M tasks gets to run on those N kernel threads.  Whether those M
> tasks are user-level threads, or callbacks, or a mix of the two is
> immaterial.  In such cases a scheduler always exists, even if it is a
> simple FIFO queue.
>


> yes ofcourse scheduler is needed. But what you said is immaterial is where
> I see the devil or say our conflict of arguments really are. Let the kernel
> thread per core deal with callbacks rather than having to build a
> user-level thread library and its scheduling mechanisms and the mapping
> between them. This sounds more of an overhead in general but may work in a
> specific case.
>

Re: scylladb

[Avi Kivity]

If you have thread-per-core and N (logical) cores, and have M tasks 
running concurrently where M > N, then you need a scheduler to decide 
which of those M tasks gets to run on those N kernel threads.  Whether 
those M tasks are user-level threads, or callbacks, or a mix of the two 
is immaterial.  In such cases a scheduler always exists, even if it is a 
simple FIFO queue.



Scheduling happens either voluntarily (the task issues I/O) or 
involuntarily (the scheduler decides it needs to run another task to 
satisfy latency SLA), but it has to happen.  The only case where it 
doesn't need to happen is if M<=N, in which case your server will be 
underutilized whenever your task has to wait.



On 03/12/2017 12:17 PM, Kant Kodali wrote:

@Avi

I don't disagree with thread per core design and in fact I said that 
is a reasonable/good choice. But I am having a hard time seeing 
through how user level scheduling can make a significant difference 
even in Non-blocking I/O case. My question really is that if you 
already have TPC why do you need user level scheduling ? And if the 
answer is to switch between user level tasks then I am simply trying 
to say "concurrency is not parallelism" (just because one was able to 
switch between user level threads doesn't mean they are running in 
parallel underneath). Why not simple schedule those on kernel threads 
running on those cores and have a callback mechanism. Why would one 
need to deal with user level scheduling overhead and all the problems 
that comes with it. This to me just sounds like difference in the 
design paradigm but doesn't seem to add much to the performance.


Seaster sounds very similar to Quasar. And I am not seeing great 
benefits from it.





On Sun, Mar 12, 2017 at 1:48 AM, Avi Kivity <a...@scylladb.com 
<mailto:a...@scylladb.com>> wrote:


We already quantified it, the result is Scylla. Now, Scylla's
performance is only in part due to the threading model, so I can't
give you a number that quantifies how much just this aspect of the
design is worth.  Removing it (or adding it to Cassandra) is a
multi-man-year effort that I can't justify for this conversation.


If you want to continue to use kernel threads for you
applications, by all means continue to do so.  They're the right
choice for all but the most I/O intensive applications.  But for
these I/O intensive applications thread-per-core is the right
choice, regardless of the points you raise.


I encourage you to study the seastar code base [1] and
documentation [2] to see how we handled those problems. I'll also
comment a bit below.


[1] https://github.com/scylladb/seastar
<https://github.com/scylladb/seastar>

[2] http://www.seastar-project.org/ <http://www.seastar-project.org/>


On 03/12/2017 11:07 AM, Kant Kodali wrote:

@Avi

"User-level scheduling is great for high performance I/O
intensive applications like databases and file systems." This is
generally a claim made by people you want to use user-level
threads but I rarely had seen any significant performance gain.
Since you are claiming that you do. It would be great if you can
quantify that. The other day I have seen a benchmark of a Golang
server which supports user level threads/green threads natively
and it was able to handle 10K concurrent requests. Even Nginx
which is written and C and uses kernel threads can handle that
many with Non-blocking I/O. We all know concurrency is not
parallelism.

You may have to pay for something which could be any of the
following.

*Duplication of the schedulers*
M:N requires two schedulers which basically do same work, one at
user level and one in kernel. This is undesirable. It requires
frequent data communications between kernel and user space for
scheduling information transference.

Duplication takes more space in both Dcache and Icache for
scheduling than a single scheduler. It is highly undesirable if
cache misses are caused by the schedulers but the application,
because a L2 cache miss could be more expensive than a kernel
thread switch. Then the additional scheduler might become a
trouble maker! In this case, to save kernel trappings does not
justify a user-scheduler, which is more truen when the processors
are providing faster and faster kernel trapping execution.



That's not a problem, at least in my experience. The kernel
scheduler needs to schedule only one thread, and that very
infrequently. It is completely out of any hot path.



*Thread local data maintenance*
M:N has to maintain thread specific data, which are already
provided by kernel for kernel thread, such as the TLS data, error
number. To provide the same feature for user threads is not
straightforward, because, for example, the error number is
return

Re: scylladb

[Kant Kodali]

@Avi

I don't disagree with thread per core design and in fact I said that is a
reasonable/good choice. But I am having a hard time seeing through how user
level scheduling can make a significant difference even in Non-blocking I/O
case. My question really is that if you already have TPC why do you need
user level scheduling ? And if the answer is to switch between user level
tasks then I am simply trying to say "concurrency is not parallelism" (just
because one was able to switch between user level threads doesn't mean they
are running in parallel underneath). Why not simple schedule those on
kernel threads running on those cores and have a callback mechanism. Why
would one need to deal with user level scheduling overhead and all the
problems that comes with it. This to me just sounds like difference in the
design paradigm but doesn't seem to add much to the performance.

Seaster sounds very similar to Quasar. And I am not seeing great benefits
from it.




On Sun, Mar 12, 2017 at 1:48 AM, Avi Kivity <a...@scylladb.com> wrote:

> We already quantified it, the result is Scylla. Now, Scylla's performance
> is only in part due to the threading model, so I can't give you a number
> that quantifies how much just this aspect of the design is worth.  Removing
> it (or adding it to Cassandra) is a multi-man-year effort that I can't
> justify for this conversation.
>
>
> If you want to continue to use kernel threads for you applications, by all
> means continue to do so.  They're the right choice for all but the most I/O
> intensive applications.  But for these I/O intensive applications
> thread-per-core is the right choice, regardless of the points you raise.
>
>
> I encourage you to study the seastar code base [1] and documentation [2]
> to see how we handled those problems.  I'll also comment a bit below.
>
>
> [1] https://github.com/scylladb/seastar
>
> [2] http://www.seastar-project.org/
>
> On 03/12/2017 11:07 AM, Kant Kodali wrote:
>
> @Avi
>
> "User-level scheduling is great for high performance I/O intensive
> applications like databases and file systems." This is generally a claim
> made by people you want to use user-level threads but I rarely had seen any
> significant performance gain. Since you are claiming that you do. It would
> be great if you can quantify that. The other day I have seen a benchmark of
> a Golang server which supports user level threads/green threads natively
> and it was able to handle 10K concurrent requests. Even Nginx which is
> written and C and uses kernel threads can handle that many with
> Non-blocking I/O. We all know concurrency is not parallelism.
>
> You may have to pay for something which could be any of the following.
>
> *Duplication of the schedulers*
> M:N requires two schedulers which basically do same work, one at user
> level and one in kernel. This is undesirable. It requires frequent data
> communications between kernel and user space for scheduling information
> transference.
>
> Duplication takes more space in both Dcache and Icache for scheduling than
> a single scheduler. It is highly undesirable if cache misses are caused by
> the schedulers but the application, because a L2 cache miss could be more
> expensive than a kernel thread switch. Then the additional scheduler might
> become a trouble maker! In this case, to save kernel trappings does not
> justify a user-scheduler, which is more truen when the processors are
> providing faster and faster kernel trapping execution.
>
>
>
> That's not a problem, at least in my experience. The kernel scheduler
> needs to schedule only one thread, and that very infrequently. It is
> completely out of any hot path.
>
>
> *Thread local data maintenance*
> M:N has to maintain thread specific data, which are already provided by
> kernel for kernel thread, such as the TLS data, error number. To provide
> the same feature for user threads is not straightforward, because, for
> example, the error number is returned for system call failure and supported
> by kernel. User-level support degrades system performance and increases
> system complexity.
>
>
> This is also not a problem, we capture error codes in exceptions
> immediately after a system call and so we don't need to rely on TLS for
> errno.
>
>
> *System info oblivious*
> Kernel scheduler is close to underlying platform and architecture. It can
> take advantage of their features. This is difficult for user thread library
> because it's a layer at user level. User threads are second-order entities
> in the system. If a kernel thread uses a GDT slot for TLS data, a user
> thread perhaps can only use an LDT slot for TLS data. With increasingly
> more supports available from the new processors for t

Re: scylladb

[Avi Kivity]

We already quantified it, the result is Scylla. Now, Scylla's 
performance is only in part due to the threading model, so I can't give 
you a number that quantifies how much just this aspect of the design is 
worth.  Removing it (or adding it to Cassandra) is a multi-man-year 
effort that I can't justify for this conversation.



If you want to continue to use kernel threads for you applications, by 
all means continue to do so.  They're the right choice for all but the 
most I/O intensive applications.  But for these I/O intensive 
applications thread-per-core is the right choice, regardless of the 
points you raise.



I encourage you to study the seastar code base [1] and documentation [2] 
to see how we handled those problems.  I'll also comment a bit below.



[1] https://github.com/scylladb/seastar

[2] http://www.seastar-project.org/


On 03/12/2017 11:07 AM, Kant Kodali wrote:

@Avi

"User-level scheduling is great for high performance I/O intensive 
applications like databases and file systems." This is generally a 
claim made by people you want to use user-level threads but I rarely 
had seen any significant performance gain. Since you are claiming that 
you do. It would be great if you can quantify that. The other day I 
have seen a benchmark of a Golang server which supports user level 
threads/green threads natively and it was able to handle 10K 
concurrent requests. Even Nginx which is written and C and uses kernel 
threads can handle that many with Non-blocking I/O. We all know 
concurrency is not parallelism.


You may have to pay for something which could be any of the following.

*Duplication of the schedulers*
M:N requires two schedulers which basically do same work, one at user 
level and one in kernel. This is undesirable. It requires frequent 
data communications between kernel and user space for scheduling 
information transference.


Duplication takes more space in both Dcache and Icache for scheduling 
than a single scheduler. It is highly undesirable if cache misses are 
caused by the schedulers but the application, because a L2 cache miss 
could be more expensive than a kernel thread switch. Then the 
additional scheduler might become a trouble maker! In this case, to 
save kernel trappings does not justify a user-scheduler, which is more 
truen when the processors are providing faster and faster kernel 
trapping execution.



That's not a problem, at least in my experience. The kernel scheduler 
needs to schedule only one thread, and that very infrequently. It is 
completely out of any hot path.




*Thread local data maintenance*
M:N has to maintain thread specific data, which are already provided 
by kernel for kernel thread, such as the TLS data, error number. To 
provide the same feature for user threads is not straightforward, 
because, for example, the error number is returned for system call 
failure and supported by kernel. User-level support degrades system 
performance and increases system complexity.


This is also not a problem, we capture error codes in exceptions 
immediately after a system call and so we don't need to rely on TLS for 
errno.




*System info oblivious*
Kernel scheduler is close to underlying platform and architecture. It 
can take advantage of their features. This is difficult for user 
thread library because it's a layer at user level. User threads are 
second-order entities in the system. If a kernel thread uses a GDT 
slot for TLS data, a user thread perhaps can only use an LDT slot for 
TLS data. With increasingly more supports available from the new 
processors for threading/scheduling (Hyperthreading, NUMA, many-core), 
the second order nature seriously limits the ability of M:N threading.


Those are non-issues, in my experience.  In fact it's the other way 
around, the kernel scheduler cannot assume anything about the threads it 
is preempting and so has to save more state.  The threads being 
preempted also cannot assume anything about the kernel scheduler, and so 
have to use atomic read-modify-write instructions for synchronization, 
and to perform a system call whenever they need to block or wake another 
thread.






On Sun, Mar 12, 2017 at 1:05 AM, Avi Kivity <a...@scylladb.com 
<mailto:a...@scylladb.com>> wrote:


btw, for an example of how user-level tasks can be scheduled in a
way that cannot be done with kernel threads, see this pair of blog
posts:


http://www.scylladb.com/2016/04/14/io-scheduler-1/
<http://www.scylladb.com/2016/04/14/io-scheduler-1/>

http://www.scylladb.com/2016/04/29/io-scheduler-2/
<http://www.scylladb.com/2016/04/29/io-scheduler-2/>


There's simply no way to get this kind of control when you rely on
the kernel for scheduling and page cache management.  As a result
you have to overprovision your node and then you mostly
underutilize it.


On 03/12/2017 10:23 AM, Avi Kivity wrote:




On 03/12/2017 12:19 AM, Kant Kodali

Re: scylladb

[Kant Kodali]

On Sun, Mar 12, 2017 at 12:23 AM, Avi Kivity  wrote:

>
>
> On 03/12/2017 12:19 AM, Kant Kodali wrote:
>
> My response is inline.
>
> On Sat, Mar 11, 2017 at 1:43 PM, Avi Kivity  wrote:
>
>> There are several issues at play here.
>>
>> First, a database runs a large number of concurrent operations, each of
>> which only consumes a small amount of CPU. The high concurrency is need to
>> hide latency: disk latency, or the latency of contacting a remote node.
>>
>
> *Ok so you are talking about hiding I/O latency.  If all these I/O are
> non-blocking system calls then a thread per core and callback mechanism
> should suffice isn't it?*
>
>
>
> Scylla uses a mix of user-level threads and callbacks. Most of the code
> uses callbacks (fronted by a future/promise API). SSTable writers
> (memtable flush, compaction) use a user-level thread (internally
> implemented using callbacks).  The important bit is multiplexing many
> concurrent operations onto a single kernel thread.
>
>
> This means that the scheduler will need to switch contexts very often. A
>> kernel thread scheduler knows very little about the application, so it has
>> to switch a lot of context.  A user level scheduler is tightly bound to the
>> application, so it can perform the switching faster.
>>
>
> *sure but this applies in other direction as well. A user level scheduler
> has no idea about kernel level scheduler either.  There is literally no
> coordination between kernel level scheduler and user level scheduler in
> linux or any major OS. It may be possible with OS's that support scheduler
> activation(LWP's) and upcall mechanism. *
>
>
> There is no need for coordination, because the kernel scheduler has no
> scheduling decisions to make.  With one thread per core, bound to its core,
> the kernel scheduler can't make the wrong decision because it has just one
> choice.
>
>
> *Even then it is hard to say if it is all worth it (The research shows
> performance may not outweigh the complexity). Golang problem is exactly
> this if one creates 1000 go routines/green threads where each of them is
> making a blocking system call then it would create 1000 kernel threads
> underneath because it has no way to know that the kernel thread is blocked
> (no upcall). *
>
>
> All of the significant system calls we issue are through the main thread,
> either asynchronous or non-blocking.
>
> *And in non-blocking case I still don't even see a significant performance
> when compared to few kernel threads with callback mechanism.*
>
>
> We do.
>
>
> *  If you are saying user level scheduling is the Future (perhaps I would
> just let the researchers argue about it) As of today that is not case else
> languages would have had it natively instead of using third party
> frameworks or libraries. *
>
>
> User-level scheduling is great for high performance I/O intensive
> applications like databases and file systems.  It's not a general solution,
> and it involves a lot of effort to set up the infrastructure. However, for
> our use case, it was worth it.
>

*Even with I/O intensive applications it is very much debatable. The
numbers I had seen aren't convincing at all. *

>
>
>
>
>> There are also implications on the concurrency primitives in use (locks
>> etc.) -- they will be much faster for the user-level scheduler, because
>> they cooperate with the scheduler.  For example, no atomic
>> read-modify-write instructions need to be executed.
>>
>
>
>  Second, how many (kernel) threads should you run?
> * This question one will always have. If there are 10K user level threads
> that maps to only one kernel thread then they cannot exploit parallelism.
> so there is no right answer but a thread per core is a reasonable/good
> choice. *
>
>
> Only if you can multiplex many operations on top of each of those
> threads.  Otherwise, the CPUs end up underutilized.
>

*Yes thats exactly my point to your question on "how many (kernel) threads
should you run?" so I will repeat myself here.  This question one will
always have even they prefer user-level thread scheduling they still need
to know how may kernel threads they need to map to so one will end up with
same question which is how many kernel threads to create?. If there are 10K
user level threads that maps to only one kernel thread then they cannot
exploit parallelism. so there is no right answer but a thread per core is a
reasonable/good choice. *


>
>
>
>
>> If you run too few threads, then you will not be able to saturate the CPU
>> resources.  This is a common problem with Cassandra -- it's very hard to
>> get it to consume all of the CPU power on even a moderately large machine.
>> On the other hand, if you have too many threads, you will see latency rise
>> very quickly, because kernel scheduling granularity is on the order of
>> milliseconds.  User-level scheduling, because it leaves control in the hand
>> of the application, allows you to both saturate the CPU and 

Re: scylladb

[Bhuvan Rawal]

​

On Sun, Mar 12, 2017 at 2:42 PM, Bhuvan Rawal <bhu1ra...@gmail.com> wrote:

> Looking at the costs of cloud instances, it clearly appears the cost of
> CPU dictates the overall cost of the instance. Having 2X more cores
> increases cost by nearly 2X keeping other things same as can be seen below
> as an example:
>
> (C3 may have slightly better processor but not more than 10-15% peformance
> increase)
>
> Optimising for fewer CPU cycles will invariably reduce costs by a large
> factor. On a modern day machine with SSD's where data density on node can
> be high more requests can be assumed to be served from single node, things
> get CPU bound. Perhaps its because it was invented at a time when SSD's did
> not exist. If we observe closely, many of cassandra defaults are assuming
> disk is rotational - number of flush writers, concurrent compactors, etc.
> The design suggest that too (Using Sequential io as far as possible. Infact
> thats the underlying philosophy for sequential sstable flushes and
> sequential commitlog files to avoid random io). Perhaps if it was designed
> currently things may look radically different.
>
> Comparing an average hard disk - ~200 iops  vs ~40K for ssd thats approx
> 200 times increase effectively increasing expectation from processor to
> serve significantly higher ops per second.
>
> In order to extract best from a modern day node it may need significant
> changes such like below :
> https://issues.apache.org/jira/browse/CASSANDRA-10989
> Possibly going forward the number of cores per node is only going to
> increase as it has been seen for last 5-6 years. In a way thats suggesting
> a significant change in design and possibly thats what scylladb is upto.
>
> "We found that we need a cpu scheduler which takes into account the
> priority of different tasks, such as repair, compaction, streaming, read
> operations and write operations."
> From my understanding in Cassandra as well compaction threads run on low
> nice priority - not sure about repair/streaming.
> http://grokbase.com/t/cassandra/user/14a85xpce7/significant-nice-cpu-usage
>
> Regards,
>
> On Sun, Mar 12, 2017 at 2:35 PM, Avi Kivity <a...@scylladb.com> wrote:
>
>> btw, for an example of how user-level tasks can be scheduled in a way
>> that cannot be done with kernel threads, see this pair of blog posts:
>>
>>
>>   http://www.scylladb.com/2016/04/14/io-scheduler-1/
>>
>>   http://www.scylladb.com/2016/04/29/io-scheduler-2/
>>
>>
>> There's simply no way to get this kind of control when you rely on the
>> kernel for scheduling and page cache management.  As a result you have to
>> overprovision your node and then you mostly underutilize it.
>>
>> On 03/12/2017 10:23 AM, Avi Kivity wrote:
>>
>>
>>
>> On 03/12/2017 12:19 AM, Kant Kodali wrote:
>>
>> My response is inline.
>>
>> On Sat, Mar 11, 2017 at 1:43 PM, Avi Kivity <a...@scylladb.com> wrote:
>>
>>> There are several issues at play here.
>>>
>>> First, a database runs a large number of concurrent operations, each of
>>> which only consumes a small amount of CPU. The high concurrency is need to
>>> hide latency: disk latency, or the latency of contacting a remote node.
>>>
>>
>> *Ok so you are talking about hiding I/O latency.  If all these I/O are
>> non-blocking system calls then a thread per core and callback mechanism
>> should suffice isn't it?*
>>
>>
>>
>> Scylla uses a mix of user-level threads and callbacks. Most of the code
>> uses callbacks (fronted by a future/promise API). SSTable writers
>> (memtable flush, compaction) use a user-level thread (internally
>> implemented using callbacks).  The important bit is multiplexing many
>> concurrent operations onto a single kernel thread.
>>
>>
>> This means that the scheduler will need to switch contexts very often. A
>>> kernel thread scheduler knows very little about the application, so it has
>>> to switch a lot of context.  A user level scheduler is tightly bound to the
>>> application, so it can perform the switching faster.
>>>
>>
>> *sure but this applies in other direction as well. A user level scheduler
>> has no idea about kernel level scheduler either.  There is literally no
>> coordination between kernel level scheduler and user level scheduler in
>> linux or any major OS. It may be possible with OS's that support scheduler
>> activation(LWP's) and upcall mechanism. *
>>
>>
>> There is no need for coordination, because the kernel scheduler has no
>> scheduling decisio

Re: scylladb

[Bhuvan Rawal]

Looking at the costs of cloud instances, it clearly appears the cost of CPU
dictates the overall cost of the instance. Having 2X more cores increases
cost by nearly 2X keeping other things same as can be seen below as an
example:

(C3 may have slightly better processor but not more than 10-15% peformance
increase)

Optimising for fewer CPU cycles will invariably reduce costs by a large
factor. On a modern day machine with SSD's where data density on node can
be high more requests can be assumed to be served from single node, things
get CPU bound. Perhaps its because it was invented at a time when SSD's did
not exist. If we observe closely, many of cassandra defaults are assuming
disk is rotational - number of flush writers, concurrent compactors, etc.
The design suggest that too (Using Sequential io as far as possible. Infact
thats the underlying philosophy for sequential sstable flushes and
sequential commitlog files to avoid random io). Perhaps if it was designed
currently things may look radically different.

Comparing an average hard disk - ~200 iops  vs ~40K for ssd thats approx
200 times increase effectively increasing expectation from processor to
serve significantly higher ops per second.

In order to extract best from a modern day node it may need significant
changes such like below :
https://issues.apache.org/jira/browse/CASSANDRA-10989
Possibly going forward the number of cores per node is only going to
increase as it has been seen for last 5-6 years. In a way thats suggesting
a significant change in design and possibly thats what scylladb is upto.

"We found that we need a cpu scheduler which takes into account the
priority of different tasks, such as repair, compaction, streaming, read
operations and write operations."
>From my understanding in Cassandra as well compaction threads run on low
nice priority - not sure about repair/streaming.
http://grokbase.com/t/cassandra/user/14a85xpce7/significant-nice-cpu-usage

Regards,

On Sun, Mar 12, 2017 at 2:35 PM, Avi Kivity <a...@scylladb.com> wrote:

> btw, for an example of how user-level tasks can be scheduled in a way that
> cannot be done with kernel threads, see this pair of blog posts:
>
>
>   http://www.scylladb.com/2016/04/14/io-scheduler-1/
>
>   http://www.scylladb.com/2016/04/29/io-scheduler-2/
>
>
> There's simply no way to get this kind of control when you rely on the
> kernel for scheduling and page cache management.  As a result you have to
> overprovision your node and then you mostly underutilize it.
>
> On 03/12/2017 10:23 AM, Avi Kivity wrote:
>
>
>
> On 03/12/2017 12:19 AM, Kant Kodali wrote:
>
> My response is inline.
>
> On Sat, Mar 11, 2017 at 1:43 PM, Avi Kivity <a...@scylladb.com> wrote:
>
>> There are several issues at play here.
>>
>> First, a database runs a large number of concurrent operations, each of
>> which only consumes a small amount of CPU. The high concurrency is need to
>> hide latency: disk latency, or the latency of contacting a remote node.
>>
>
> *Ok so you are talking about hiding I/O latency.  If all these I/O are
> non-blocking system calls then a thread per core and callback mechanism
> should suffice isn't it?*
>
>
>
> Scylla uses a mix of user-level threads and callbacks. Most of the code
> uses callbacks (fronted by a future/promise API). SSTable writers
> (memtable flush, compaction) use a user-level thread (internally
> implemented using callbacks).  The important bit is multiplexing many
> concurrent operations onto a single kernel thread.
>
>
> This means that the scheduler will need to switch contexts very often. A
>> kernel thread scheduler knows very little about the application, so it has
>> to switch a lot of context.  A user level scheduler is tightly bound to the
>> application, so it can perform the switching faster.
>>
>
> *sure but this applies in other direction as well. A user level scheduler
> has no idea about kernel level scheduler either.  There is literally no
> coordination between kernel level scheduler and user level scheduler in
> linux or any major OS. It may be possible with OS's that support scheduler
> activation(LWP's) and upcall mechanism. *
>
>
> There is no need for coordination, because the kernel scheduler has no
> scheduling decisions to make.  With one thread per core, bound to its core,
> the kernel scheduler can't make the wrong decision because it has just one
> choice.
>
>
> *Even then it is hard to say if it is all worth it (The research shows
> performance may not outweigh the complexity). Golang problem is exactly
> this if one creates 1000 go routines/green threads where each of them is
> making a blocking system call then it would create 1000 kernel threads
> underneath because it has no way to know

Re: scylladb

[Kant Kodali]

@Avi

"User-level scheduling is great for high performance I/O intensive
applications like databases and file systems." This is generally a claim
made by people you want to use user-level threads but I rarely had seen any
significant performance gain. Since you are claiming that you do. It would
be great if you can quantify that. The other day I have seen a benchmark of
a Golang server which supports user level threads/green threads natively
and it was able to handle 10K concurrent requests. Even Nginx which is
written and C and uses kernel threads can handle that many with
Non-blocking I/O. We all know concurrency is not parallelism.

You may have to pay for something which could be any of the following.

*Duplication of the schedulers*
M:N requires two schedulers which basically do same work, one at user level
and one in kernel. This is undesirable. It requires frequent data
communications between kernel and user space for scheduling information
transference.

Duplication takes more space in both Dcache and Icache for scheduling than
a single scheduler. It is highly undesirable if cache misses are caused by
the schedulers but the application, because a L2 cache miss could be more
expensive than a kernel thread switch. Then the additional scheduler might
become a trouble maker! In this case, to save kernel trappings does not
justify a user-scheduler, which is more truen when the processors are
providing faster and faster kernel trapping execution.

*Thread local data maintenance*
M:N has to maintain thread specific data, which are already provided by
kernel for kernel thread, such as the TLS data, error number. To provide
the same feature for user threads is not straightforward, because, for
example, the error number is returned for system call failure and supported
by kernel. User-level support degrades system performance and increases
system complexity.

*System info oblivious*
Kernel scheduler is close to underlying platform and architecture. It can
take advantage of their features. This is difficult for user thread library
because it's a layer at user level. User threads are second-order entities
in the system. If a kernel thread uses a GDT slot for TLS data, a user
thread perhaps can only use an LDT slot for TLS data. With increasingly
more supports available from the new processors for threading/scheduling
(Hyperthreading, NUMA, many-core), the second order nature seriously limits
the ability of M:N threading.

On Sun, Mar 12, 2017 at 1:05 AM, Avi Kivity  wrote:

> btw, for an example of how user-level tasks can be scheduled in a way that
> cannot be done with kernel threads, see this pair of blog posts:
>
>
>   http://www.scylladb.com/2016/04/14/io-scheduler-1/
>
>   http://www.scylladb.com/2016/04/29/io-scheduler-2/
>
>
> There's simply no way to get this kind of control when you rely on the
> kernel for scheduling and page cache management.  As a result you have to
> overprovision your node and then you mostly underutilize it.
>
> On 03/12/2017 10:23 AM, Avi Kivity wrote:
>
>
>
> On 03/12/2017 12:19 AM, Kant Kodali wrote:
>
> My response is inline.
>
> On Sat, Mar 11, 2017 at 1:43 PM, Avi Kivity  wrote:
>
>> There are several issues at play here.
>>
>> First, a database runs a large number of concurrent operations, each of
>> which only consumes a small amount of CPU. The high concurrency is need to
>> hide latency: disk latency, or the latency of contacting a remote node.
>>
>
> *Ok so you are talking about hiding I/O latency.  If all these I/O are
> non-blocking system calls then a thread per core and callback mechanism
> should suffice isn't it?*
>
>
>
> Scylla uses a mix of user-level threads and callbacks. Most of the code
> uses callbacks (fronted by a future/promise API). SSTable writers
> (memtable flush, compaction) use a user-level thread (internally
> implemented using callbacks).  The important bit is multiplexing many
> concurrent operations onto a single kernel thread.
>
>
> This means that the scheduler will need to switch contexts very often. A
>> kernel thread scheduler knows very little about the application, so it has
>> to switch a lot of context.  A user level scheduler is tightly bound to the
>> application, so it can perform the switching faster.
>>
>
> *sure but this applies in other direction as well. A user level scheduler
> has no idea about kernel level scheduler either.  There is literally no
> coordination between kernel level scheduler and user level scheduler in
> linux or any major OS. It may be possible with OS's that support scheduler
> activation(LWP's) and upcall mechanism. *
>
>
> There is no need for coordination, because the kernel scheduler has no
> scheduling decisions to make.  With one thread per core, bound to its core,
> the kernel scheduler can't make the wrong decision because it has just one
> choice.
>
>
> *Even then it is hard to say if it is all worth it (The research shows
> performance may not outweigh 

Re: scylladb

[Avi Kivity]

btw, for an example of how user-level tasks can be scheduled in a way 
that cannot be done with kernel threads, see this pair of blog posts:



  http://www.scylladb.com/2016/04/14/io-scheduler-1/

  http://www.scylladb.com/2016/04/29/io-scheduler-2/


There's simply no way to get this kind of control when you rely on the 
kernel for scheduling and page cache management.  As a result you have 
to overprovision your node and then you mostly underutilize it.



On 03/12/2017 10:23 AM, Avi Kivity wrote:




On 03/12/2017 12:19 AM, Kant Kodali wrote:

My response is inline.

On Sat, Mar 11, 2017 at 1:43 PM, Avi Kivity > wrote:


There are several issues at play here.

First, a database runs a large number of concurrent operations,
each of which only consumes a small amount of CPU. The high
concurrency is need to hide latency: disk latency, or the latency
of contacting a remote node.

*Ok so you are talking about hiding I/O latency. If all these I/O are 
non-blocking system calls then a thread per core and callback 
mechanism should suffice isn't it?*


Scylla uses a mix of user-level threads and callbacks. Most of the 
code uses callbacks (fronted by a future/promise API). SSTable 
writers  (memtable flush, compaction) use a user-level thread 
(internally implemented using callbacks).  The important bit is 
multiplexing many concurrent operations onto a single kernel thread.




This means that the scheduler will need to switch contexts very
often. A kernel thread scheduler knows very little about the
application, so it has to switch a lot of context.  A user level
scheduler is tightly bound to the application, so it can perform
the switching faster.


*sure but this applies in other direction as well. A user level 
scheduler has no idea about kernel level scheduler either.  There is 
literally no coordination between kernel level scheduler and user 
level scheduler in linux or any major OS. It may be possible with 
OS's that support scheduler activation(LWP's) and upcall mechanism. *


There is no need for coordination, because the kernel scheduler has no 
scheduling decisions to make.  With one thread per core, bound to its 
core, the kernel scheduler can't make the wrong decision because it 
has just one choice.



*Even then it is hard to say if it is all worth it (The research 
shows performance may not outweigh the complexity). Golang problem is 
exactly this if one creates 1000 go routines/green threads where each 
of them is making a blocking system call then it would create 1000 
kernel threads underneath because it has no way to know that the 
kernel thread is blocked (no upcall). *


All of the significant system calls we issue are through the main 
thread, either asynchronous or non-blocking.


*And in non-blocking case I still don't even see a significant 
performance when compared to few kernel threads with callback mechanism.*


We do.

*  If you are saying user level scheduling is the Future (perhaps I 
would just let the researchers argue about it) As of today that is 
not case else languages would have had it natively instead of using 
third party frameworks or libraries.

*


User-level scheduling is great for high performance I/O intensive 
applications like databases and file systems.  It's not a general 
solution, and it involves a lot of effort to set up the 
infrastructure. However, for our use case, it was worth it.



There are also implications on the concurrency primitives in use
(locks etc.) -- they will be much faster for the user-level
scheduler, because they cooperate with the scheduler.  For
example, no atomic read-modify-write instructions need to be
executed.


 Second, how many (kernel) threads should you run?*This question 
one will always have. If there are 10K user level threads that maps 
to only one kernel thread then they cannot exploit parallelism. so 
there is no right answer but a thread per core is a reasonable/good 
choice.

*


Only if you can multiplex many operations on top of each of those 
threads.  Otherwise, the CPUs end up underutilized.



If you run too few threads, then you will not be able to saturate
the CPU resources.  This is a common problem with Cassandra --
it's very hard to get it to consume all of the CPU power on even
a moderately large machine. On the other hand, if you have too
many threads, you will see latency rise very quickly, because
kernel scheduling granularity is on the order of milliseconds. 
User-level scheduling, because it leaves control in the hand of

the application, allows you to both saturate the CPU and maintain
low latency.


F*or my workload and probably others I had seen Cassandra was 
always been CPU bound.*






Yes, but does it consume 100% of all of the cores on your machine?  
Cassandra generally doesn't (on a larger machine), and when you 
profile it, you see it 

Re: scylladb

[Avi Kivity]

On 03/12/2017 12:19 AM, Kant Kodali wrote:

My response is inline.

On Sat, Mar 11, 2017 at 1:43 PM, Avi Kivity > wrote:


There are several issues at play here.

First, a database runs a large number of concurrent operations,
each of which only consumes a small amount of CPU. The high
concurrency is need to hide latency: disk latency, or the latency
of contacting a remote node.

*Ok so you are talking about hiding I/O latency.  If all these I/O are 
non-blocking system calls then a thread per core and callback 
mechanism should suffice isn't it?*


Scylla uses a mix of user-level threads and callbacks. Most of the code 
uses callbacks (fronted by a future/promise API). SSTable writers  
(memtable flush, compaction) use a user-level thread (internally 
implemented using callbacks).  The important bit is multiplexing many 
concurrent operations onto a single kernel thread.




This means that the scheduler will need to switch contexts very
often. A kernel thread scheduler knows very little about the
application, so it has to switch a lot of context.  A user level
scheduler is tightly bound to the application, so it can perform
the switching faster.


*sure but this applies in other direction as well. A user level 
scheduler has no idea about kernel level scheduler either.  There is 
literally no coordination between kernel level scheduler and user 
level scheduler in linux or any major OS. It may be possible with OS's 
that support scheduler activation(LWP's) and upcall mechanism. *


There is no need for coordination, because the kernel scheduler has no 
scheduling decisions to make.  With one thread per core, bound to its 
core, the kernel scheduler can't make the wrong decision because it has 
just one choice.



*Even then it is hard to say if it is all worth it (The research shows 
performance may not outweigh the complexity). Golang problem is 
exactly this if one creates 1000 go routines/green threads where each 
of them is making a blocking system call then it would create 1000 
kernel threads underneath because it has no way to know that the 
kernel thread is blocked (no upcall). *


All of the significant system calls we issue are through the main 
thread, either asynchronous or non-blocking.


*And in non-blocking case I still don't even see a significant 
performance when compared to few kernel threads with callback mechanism.*


We do.

*  If you are saying user level scheduling is the Future (perhaps I 
would just let the researchers argue about it) As of today that is not 
case else languages would have had it natively instead of using third 
party frameworks or libraries.

*


User-level scheduling is great for high performance I/O intensive 
applications like databases and file systems.  It's not a general 
solution, and it involves a lot of effort to set up the infrastructure. 
However, for our use case, it was worth it.



There are also implications on the concurrency primitives in use
(locks etc.) -- they will be much faster for the user-level
scheduler, because they cooperate with the scheduler.  For
example, no atomic read-modify-write instructions need to be executed.


 Second, how many (kernel) threads should you run?*This question 
one will always have. If there are 10K user level threads that maps to 
only one kernel thread then they cannot exploit parallelism. so there 
is no right answer but a thread per core is a reasonable/good choice.

*


Only if you can multiplex many operations on top of each of those 
threads.  Otherwise, the CPUs end up underutilized.



If you run too few threads, then you will not be able to saturate
the CPU resources.  This is a common problem with Cassandra --
it's very hard to get it to consume all of the CPU power on even a
moderately large machine. On the other hand, if you have too many
threads, you will see latency rise very quickly, because kernel
scheduling granularity is on the order of milliseconds. 
User-level scheduling, because it leaves control in the hand of

the application, allows you to both saturate the CPU and maintain
low latency.


F*or my workload and probably others I had seen Cassandra was 
always been CPU bound.*






Yes, but does it consume 100% of all of the cores on your machine? 
Cassandra generally doesn't (on a larger machine), and when you profile 
it, you see it spending much of its time in atomic operations, or 
parking/unparking threads -- fighting with itself. It doesn't scale 
within the machine.  Scylla will happily utilize all of the cores that 
it is assigned (all of them by default in most configurations), and the 
bigger the machine you give it, the happier it will be.



There are other factors, like NUMA-friendliness, but in the end it
all boils down to efficiency and control.

None of this is new btw, it's pretty common in the storage world.

  

Re: scylladb

[Dor Laor]

On Sat, Mar 11, 2017 at 2:19 PM, Kant Kodali  wrote:

> My response is inline.
>
> On Sat, Mar 11, 2017 at 1:43 PM, Avi Kivity  wrote:
>
>> There are several issues at play here.
>>
>> First, a database runs a large number of concurrent operations, each of
>> which only consumes a small amount of CPU. The high concurrency is need to
>> hide latency: disk latency, or the latency of contacting a remote node.
>>
>
> *Ok so you are talking about hiding I/O latency.  If all these I/O are
> non-blocking system calls then a thread per core and callback mechanism
> should suffice isn't it?*
>

In general, yes but in practice it's more complicated.
Each such thread runs different tasks, you need a mechanism to switch
between these
tasks, this is the seastar continuation engine in our case. However, things
get more
complicated. We found that we need a cpu scheduler which takes into account
the priority
of different tasks, such as repair, compaction, streaming, read operations
and write operations.
We always prioritize foreground operations over background ones and thus
even when we
repair TBs of data, latency is still very low (this feature is coming in
Scylla 1.8)



>
>
>> This means that the scheduler will need to switch contexts very often. A
>> kernel thread scheduler knows very little about the application, so it has
>> to switch a lot of context.  A user level scheduler is tightly bound to the
>> application, so it can perform the switching faster.
>>
>
> *sure but this applies in other direction as well. A user level scheduler
> has no idea about kernel level scheduler either.  There is literally no
> coordination between kernel level scheduler and user level scheduler in
> linux or any major OS. It may be possible with OS's *
>

Correct. That's why we let the OS scheduler to run just one thread per core
and we bind the thread to the cpu. Inside, we do our own stuff with the
seastar scheduler and the OS doesn't know and doesn't care.

More below


> *that support scheduler activation(LWP's) and upcall mechanism. Even then
> it is hard to say if it is all worth it (The research shows performance may
> not outweigh the complexity). Golang problem is exactly this if one creates
> 1000 go routines/green threads where each of them is making a blocking
> system call then it would create 1000 kernel threads underneath because it
> has no way to know that the kernel thread is blocked (no upcall). And in
> non-blocking case I still don't even see a significant performance when
> compared to few kernel threads with callback mechanism.  If you are saying
> user level scheduling is the Future (perhaps I would just let the
> researchers argue about it) As of today that is not case else languages
> would have had it natively instead of using third party frameworks or
> libraries. *
>

That's why we do not run blocking system calls at all. We had to limit
ourselves to the XFS filesystem
only since the others did have got AIO support. Recently we bypassed some
of the issues which
made EXT4 to block and it may be ok with our AIO pattern.

We even write a DNS implementation that doesn't block and doesn't lock (for
us, even a library that uses spin locks under the hood is bad).

Bare in mind that the whole thing is simple to run and the user doesn't
need to know anything of this complexity.




>
>
>> There are also implications on the concurrency primitives in use (locks
>> etc.) -- they will be much faster for the user-level scheduler, because
>> they cooperate with the scheduler.  For example, no atomic
>> read-modify-write instructions need to be executed.
>>
>
>
>  Second, how many (kernel) threads should you run?* This question one
> will always have. If there are 10K user level threads that maps to only one
> kernel thread then they cannot exploit parallelism. so there is no right
> answer but a thread per core is a reasonable/good choice. *
>

+1


>
>
>> If you run too few threads, then you will not be able to saturate the CPU
>> resources.  This is a common problem with Cassandra -- it's very hard to
>> get it to consume all of the CPU power on even a moderately large machine.
>> On the other hand, if you have too many threads, you will see latency rise
>> very quickly, because kernel scheduling granularity is on the order of
>> milliseconds.  User-level scheduling, because it leaves control in the hand
>> of the application, allows you to both saturate the CPU and maintain low
>> latency.
>>
>
> F*or my workload and probably others I had seen Cassandra was always
> been CPU bound.*
>

Could be. However, try to make it CPU bound on 10 core, 20 core and more.
The more core you use, the less nodes you need and the overall overhead
decreases.


>
>> There are other factors, like NUMA-friendliness, but in the end it all
>> boils down to efficiency and control.
>>
>> None of this is new btw, it's pretty common in the storage world.
>>
>> Avi
>>
>>
>> On 03/11/2017 11:18 PM, 

Re: scylladb

[benjamin roth]

There is no reason to be angry. This is progress. This is the circle of
live.

It happens anywhere at any time.

Am 12.03.2017 07:34 schrieb "Dor Laor" :

> On Sat, Mar 11, 2017 at 10:02 PM, Jeff Jirsa  wrote:
>
>>
>>
>> On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote:
>> > Cassanda vs Scylla is a valid comparison because they both are
>> compatible. Scylla is a drop-in replacement for Cassandra.
>>
>> No, they aren't, and no, it isn't
>>
>
> Jeff is angry with us for some reason. I don't know why, it's natural that
> when
> a new opponent there are objections and the proof lies on us.
> We go through great deal of doing it and we don't just throw comments
> without backing.
>
> Scylla IS a drop in replacement for C*. We support the same CQL (from
> version 1.7 it's cql 3.3.1, protocol v4), the same SStable format (based on
> 2.1.8). In 1.7 release we support cql uploader
> from 3.x. We will support the SStable format of 3.x natively in 3 month
> time. Soon all of the feature set will be implemented. We always have been
> using this page (not 100% up to date, we'll update it this week):
> http://www.scylladb.com/technology/status/
>
> We add a jmx-proxy daemon in java in order to make the transition as
> smooth as possible. Almost all the nodetool commands just work, for sure
> all the important ones.
> Btw: we have a RESTapi and Prometheus formats, much better than the hairy
> jmx one.
>
> Spark, Kairosdb, Presto and probably Titan (we add Thrift just for legacy
> users and we don't intend
> to decommission an api).
>
> Regarding benchmarks, if someone finds a flaw in them, we'll do the best
> to fix it.
> Let's ignore them and just here what our users have to say:
> http://www.scylladb.com/users/
>
>
>

Re: scylladb

[Dor Laor]

On Sat, Mar 11, 2017 at 10:02 PM, Jeff Jirsa  wrote:

>
>
> On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote:
> > Cassanda vs Scylla is a valid comparison because they both are
> compatible. Scylla is a drop-in replacement for Cassandra.
>
> No, they aren't, and no, it isn't
>

Jeff is angry with us for some reason. I don't know why, it's natural that
when
a new opponent there are objections and the proof lies on us.
We go through great deal of doing it and we don't just throw comments
without backing.

Scylla IS a drop in replacement for C*. We support the same CQL (from
version 1.7 it's cql 3.3.1, protocol v4), the same SStable format (based on
2.1.8). In 1.7 release we support cql uploader
from 3.x. We will support the SStable format of 3.x natively in 3 month
time. Soon all of the feature set will be implemented. We always have been
using this page (not 100% up to date, we'll update it this week):
http://www.scylladb.com/technology/status/

We add a jmx-proxy daemon in java in order to make the transition as smooth
as possible. Almost all the nodetool commands just work, for sure all the
important ones.
Btw: we have a RESTapi and Prometheus formats, much better than the hairy
jmx one.

Spark, Kairosdb, Presto and probably Titan (we add Thrift just for legacy
users and we don't intend
to decommission an api).

Regarding benchmarks, if someone finds a flaw in them, we'll do the best to
fix it.
Let's ignore them and just here what our users have to say:
http://www.scylladb.com/users/

Re: scylladb

[benjamin roth]

Why?

Am 12.03.2017 07:02 schrieb "Jeff Jirsa" :

>
>
> On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote:
> > Cassanda vs Scylla is a valid comparison because they both are
> compatible. Scylla is a drop-in replacement for Cassandra.
>
> No, they aren't, and no, it isn't
>
>
>
>
>

Re: scylladb

[Jeff Jirsa]

On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote: 
> Cassanda vs Scylla is a valid comparison because they both are compatible. 
> Scylla is a drop-in replacement for Cassandra.

No, they aren't, and no, it isn't

Re: scylladb

[Edward Capriolo]

On Sat, Mar 11, 2017 at 9:41 PM, daemeon reiydelle <daeme...@gmail.com>
wrote:

> Recall that garbage collection on a busy node can occur minutes or seconds
> apart. Note that stop the world GC also happens as frequently as every
> couple of minutes on every node. Remove that and do the simple arithmetic.
>
>
> sent from my mobile
> Daemeon Reiydelle
> skype daemeon.c.m.reiydelle
> USA 415.501.0198 <(415)%20501-0198>
>
> On Mar 10, 2017 8:59 AM, "Bhuvan Rawal" <bhu1ra...@gmail.com> wrote:
>
>> Agreed C++ gives an added advantage to talk to underlying hardware with
>> better efficiency, it sound good but can a pice of code written in C++ give
>> 1000% throughput than a Java app? Is TPC design 10X more performant than
>> SEDA arch?
>>
>> And if C/C++ is indeed that fast how can Aerospike (which is itself
>> written in C) claim to be 10X faster than Scylla here
>> http://www.aerospike.com/benchmarks/scylladb-initial/ ? (Combining
>> your's and aerospike's benchmarks it appears that Aerospike is 100X
>> performant than C* - I highly doubt that!! )
>>
>> For a moment lets forget about evaluating 2 different databases, one can
>> observe 10X performance difference between a mistuned cassandra cluster and
>> one thats tuned as per data model - there are so many Tunables in yaml as
>> well as table configs.
>>
>> Idea is - in order to strengthen your claim, you need to provide complete
>> system metrics (Disk, CPU, Network), the OPS increase starts to decay along
>> with the configs used. Having plain ops per second and 99p latency is
>> blackbox.
>>
>> Regards,
>> Bhuvan
>>
>> On Fri, Mar 10, 2017 at 12:47 PM, Avi Kivity <a...@scylladb.com> wrote:
>>
>>> ScyllaDB engineer here.
>>>
>>> C++ is really an enabling technology here. It is directly responsible
>>> for a small fraction of the gain by executing faster than Java.  But it is
>>> indirectly responsible for the gain by allowing us direct control over
>>> memory and threading.  Just as an example, Scylla starts by taking over
>>> almost all of the machine's memory, and dynamically assigning it to
>>> memtables, cache, and working memory needed to handle requests in flight.
>>> Memory is statically partitioned across cores, allowing us to exploit NUMA
>>> fully.  You can't do these things in Java.
>>>
>>> I would say the major contributors to Scylla performance are:
>>>  - thread-per-core design
>>>  - replacement of the page cache with a row cache
>>>  - careful attention to many small details, each contributing a little,
>>> but with a large overall impact
>>>
>>> While I'm here I can say that performance is not the only goal here, it
>>> is stable and predictable performance over varying loads and during
>>> maintenance operations like repair, without any special tuning.  We measure
>>> the amount of CPU and I/O spent on foreground (user) and background
>>> (maintenance) tasks and divide them fairly.  This work is not complete but
>>> already makes operating Scylla a lot simpler.
>>>
>>>
>>> On 03/10/2017 01:42 AM, Kant Kodali wrote:
>>>
>>> I dont think ScyllaDB performance is because of C++. The design
>>> decisions in scylladb are indeed different from Cassandra such as getting
>>> rid of SEDA and moving to TPC and so on.
>>>
>>> If someone thinks it is because of C++ then just show the benchmarks
>>> that proves it is indeed the C++ which gave 10X performance boost as
>>> ScyllaDB claims instead of stating it.
>>>
>>>
>>> On Thu, Mar 9, 2017 at 3:22 PM, Richard L. Burton III <
>>> mrbur...@gmail.com> wrote:
>>>
>>>> They spend an enormous amount of time focusing on performance. You can
>>>> expect them to continue on with their optimization and keep crushing it.
>>>>
>>>> P.S., I don't work for ScyllaDB.
>>>>
>>>> On Thu, Mar 9, 2017 at 6:02 PM, Rakesh Kumar <
>>>> rakeshkumar...@outlook.com> wrote:
>>>>
>>>>> In all of their presentation they keep harping on the fact that
>>>>> scylladb is written in C++ and does not carry the overhead of Java.  Still
>>>>> the difference looks staggering.
>>>>> 
>>>>> From: daemeon reiydelle <daeme...@gmail.com>
>>>>> Sent: Thursday, March 9, 2017 14:21
>>>>> To: user@cassandra.apache.org
>

Re: scylladb

[daemeon reiydelle]

Recall that garbage collection on a busy node can occur minutes or seconds
apart. Note that stop the world GC also happens as frequently as every
couple of minutes on every node. Remove that and do the simple arithmetic.


sent from my mobile
Daemeon Reiydelle
skype daemeon.c.m.reiydelle
USA 415.501.0198

On Mar 10, 2017 8:59 AM, "Bhuvan Rawal" <bhu1ra...@gmail.com> wrote:

> Agreed C++ gives an added advantage to talk to underlying hardware with
> better efficiency, it sound good but can a pice of code written in C++ give
> 1000% throughput than a Java app? Is TPC design 10X more performant than
> SEDA arch?
>
> And if C/C++ is indeed that fast how can Aerospike (which is itself
> written in C) claim to be 10X faster than Scylla here
> http://www.aerospike.com/benchmarks/scylladb-initial/ ? (Combining your's
> and aerospike's benchmarks it appears that Aerospike is 100X performant
> than C* - I highly doubt that!! )
>
> For a moment lets forget about evaluating 2 different databases, one can
> observe 10X performance difference between a mistuned cassandra cluster and
> one thats tuned as per data model - there are so many Tunables in yaml as
> well as table configs.
>
> Idea is - in order to strengthen your claim, you need to provide complete
> system metrics (Disk, CPU, Network), the OPS increase starts to decay along
> with the configs used. Having plain ops per second and 99p latency is
> blackbox.
>
> Regards,
> Bhuvan
>
> On Fri, Mar 10, 2017 at 12:47 PM, Avi Kivity <a...@scylladb.com> wrote:
>
>> ScyllaDB engineer here.
>>
>> C++ is really an enabling technology here. It is directly responsible for
>> a small fraction of the gain by executing faster than Java.  But it is
>> indirectly responsible for the gain by allowing us direct control over
>> memory and threading.  Just as an example, Scylla starts by taking over
>> almost all of the machine's memory, and dynamically assigning it to
>> memtables, cache, and working memory needed to handle requests in flight.
>> Memory is statically partitioned across cores, allowing us to exploit NUMA
>> fully.  You can't do these things in Java.
>>
>> I would say the major contributors to Scylla performance are:
>>  - thread-per-core design
>>  - replacement of the page cache with a row cache
>>  - careful attention to many small details, each contributing a little,
>> but with a large overall impact
>>
>> While I'm here I can say that performance is not the only goal here, it
>> is stable and predictable performance over varying loads and during
>> maintenance operations like repair, without any special tuning.  We measure
>> the amount of CPU and I/O spent on foreground (user) and background
>> (maintenance) tasks and divide them fairly.  This work is not complete but
>> already makes operating Scylla a lot simpler.
>>
>>
>> On 03/10/2017 01:42 AM, Kant Kodali wrote:
>>
>> I dont think ScyllaDB performance is because of C++. The design decisions
>> in scylladb are indeed different from Cassandra such as getting rid of SEDA
>> and moving to TPC and so on.
>>
>> If someone thinks it is because of C++ then just show the benchmarks that
>> proves it is indeed the C++ which gave 10X performance boost as ScyllaDB
>> claims instead of stating it.
>>
>>
>> On Thu, Mar 9, 2017 at 3:22 PM, Richard L. Burton III <mrbur...@gmail.com
>> > wrote:
>>
>>> They spend an enormous amount of time focusing on performance. You can
>>> expect them to continue on with their optimization and keep crushing it.
>>>
>>> P.S., I don't work for ScyllaDB.
>>>
>>> On Thu, Mar 9, 2017 at 6:02 PM, Rakesh Kumar <rakeshkumar...@outlook.com
>>> > wrote:
>>>
>>>> In all of their presentation they keep harping on the fact that
>>>> scylladb is written in C++ and does not carry the overhead of Java.  Still
>>>> the difference looks staggering.
>>>> 
>>>> From: daemeon reiydelle <daeme...@gmail.com>
>>>> Sent: Thursday, March 9, 2017 14:21
>>>> To: user@cassandra.apache.org
>>>> Subject: Re: scylladb
>>>>
>>>> The comparison is fair, and conservative. Did substantial performance
>>>> comparisons for two clients, both results returned throughputs that were
>>>> faster than the published comparisons (15x as I recall). At that time the
>>>> client preferred to utilize a Cass COTS solution and use a caching solution
>>>> for OLA compliance.
>>>>
>>>>
>

Re: scylladb

[Kant Kodali]

My response is inline.

On Sat, Mar 11, 2017 at 1:43 PM, Avi Kivity  wrote:

> There are several issues at play here.
>
> First, a database runs a large number of concurrent operations, each of
> which only consumes a small amount of CPU. The high concurrency is need to
> hide latency: disk latency, or the latency of contacting a remote node.
>

*Ok so you are talking about hiding I/O latency.  If all these I/O are
non-blocking system calls then a thread per core and callback mechanism
should suffice isn't it?*


> This means that the scheduler will need to switch contexts very often. A
> kernel thread scheduler knows very little about the application, so it has
> to switch a lot of context.  A user level scheduler is tightly bound to the
> application, so it can perform the switching faster.
>

*sure but this applies in other direction as well. A user level scheduler
has no idea about kernel level scheduler either.  There is literally no
coordination between kernel level scheduler and user level scheduler in
linux or any major OS. It may be possible with OS's that support scheduler
activation(LWP's) and upcall mechanism. Even then it is hard to say if it
is all worth it (The research shows performance may not outweigh the
complexity). Golang problem is exactly this if one creates 1000 go
routines/green threads where each of them is making a blocking system call
then it would create 1000 kernel threads underneath because it has no way
to know that the kernel thread is blocked (no upcall). And in non-blocking
case I still don't even see a significant performance when compared to few
kernel threads with callback mechanism.  If you are saying user level
scheduling is the Future (perhaps I would just let the researchers argue
about it) As of today that is not case else languages would have had it
natively instead of using third party frameworks or libraries. *


> There are also implications on the concurrency primitives in use (locks
> etc.) -- they will be much faster for the user-level scheduler, because
> they cooperate with the scheduler.  For example, no atomic
> read-modify-write instructions need to be executed.
>


 Second, how many (kernel) threads should you run?* This question one
will always have. If there are 10K user level threads that maps to only one
kernel thread then they cannot exploit parallelism. so there is no right
answer but a thread per core is a reasonable/good choice. *


> If you run too few threads, then you will not be able to saturate the CPU
> resources.  This is a common problem with Cassandra -- it's very hard to
> get it to consume all of the CPU power on even a moderately large machine.
> On the other hand, if you have too many threads, you will see latency rise
> very quickly, because kernel scheduling granularity is on the order of
> milliseconds.  User-level scheduling, because it leaves control in the hand
> of the application, allows you to both saturate the CPU and maintain low
> latency.
>

F*or my workload and probably others I had seen Cassandra was always
been CPU bound.*

>
> There are other factors, like NUMA-friendliness, but in the end it all
> boils down to efficiency and control.
>
> None of this is new btw, it's pretty common in the storage world.
>
> Avi
>
>
> On 03/11/2017 11:18 PM, Kant Kodali wrote:
>
> Here is the Java version http://docs.paralleluniverse.co/quasar/ but I
> still don't see how user level scheduling can be beneficial (This is a well
> debated problem)? How can this add to the performance? or say why is user
> level scheduling necessary Given the Thread per core design and the
> callback mechanism?
>
> On Sat, Mar 11, 2017 at 12:51 PM, Avi Kivity  wrote:
>
>> Scylla uses a the seastar framework, which provides for both user-level
>> thread scheduling and simple run-to-completion tasks.
>>
>> Huge pages are limited to 2MB (and 1GB, but these aren't available as
>> transparent hugepages).
>>
>>
>> On 03/11/2017 10:26 PM, Kant Kodali wrote:
>>
>> @Dor
>>
>> 1) You guys have a CPU scheduler? you mean user level thread Scheduler
>> that maps user level threads to kernel level threads? I thought C++ by
>> default creates native kernel threads but sure nothing will stop someone to
>> create a user level scheduling library if that's what you are talking about?
>> 2) How can one create THP of size 1KB? According to this post
>> 
>>  it
>> looks like the valid values 2MB and 1GB.
>>
>> Thanks,
>> kant
>>
>> On Sat, Mar 11, 2017 at 11:41 AM, Avi Kivity  wrote:
>>
>>> Agreed, I'd recommend to treat benchmarks as a rough guide to see where
>>> there is potential, and follow through with your own tests.
>>>
>>> On 03/11/2017 09:37 PM, Edward Capriolo wrote:
>>>
>>>
>>> Benchmarks are great for FUDly blog posts. Real world work loads matter
>>> more. Every NoSQL 

Re: scylladb

[Avi Kivity]

There are several issues at play here.

First, a database runs a large number of concurrent operations, each of 
which only consumes a small amount of CPU. The high concurrency is need 
to hide latency: disk latency, or the latency of contacting a remote 
node. This means that the scheduler will need to switch contexts very 
often. A kernel thread scheduler knows very little about the 
application, so it has to switch a lot of context.  A user level 
scheduler is tightly bound to the application, so it can perform the 
switching faster.  There are also implications on the concurrency 
primitives in use (locks etc.) -- they will be much faster for the 
user-level scheduler, because they cooperate with the scheduler.  For 
example, no atomic read-modify-write instructions need to be executed.


Second, how many (kernel) threads should you run?  If you run too few 
threads, then you will not be able to saturate the CPU resources.  This 
is a common problem with Cassandra -- it's very hard to get it to 
consume all of the CPU power on even a moderately large machine.  On the 
other hand, if you have too many threads, you will see latency rise very 
quickly, because kernel scheduling granularity is on the order of 
milliseconds. User-level scheduling, because it leaves control in the 
hand of the application, allows you to both saturate the CPU and 
maintain low latency.


There are other factors, like NUMA-friendliness, but in the end it all 
boils down to efficiency and control.


None of this is new btw, it's pretty common in the storage world.

Avi

On 03/11/2017 11:18 PM, Kant Kodali wrote:
Here is the Java version http://docs.paralleluniverse.co/quasar/ but I 
still don't see how user level scheduling can be beneficial (This is a 
well debated problem)? How can this add to the performance? or say why 
is user level scheduling necessary Given the Thread per core design 
and the callback mechanism?


On Sat, Mar 11, 2017 at 12:51 PM, Avi Kivity > wrote:


Scylla uses a the seastar framework, which provides for both
user-level thread scheduling and simple run-to-completion tasks.

Huge pages are limited to 2MB (and 1GB, but these aren't available
as transparent hugepages).


On 03/11/2017 10:26 PM, Kant Kodali wrote:

@Dor

1) You guys have a CPU scheduler? you mean user level thread
Scheduler that maps user level threads to kernel level threads? I
thought C++ by default creates native kernel threads but sure
nothing will stop someone to create a user level scheduling
library if that's what you are talking about?
2) How can one create THP of size 1KB? According to this post


 it
looks like the valid values 2MB and 1GB.

Thanks,
kant

On Sat, Mar 11, 2017 at 11:41 AM, Avi Kivity > wrote:

Agreed, I'd recommend to treat benchmarks as a rough guide to
see where there is potential, and follow through with your
own tests.

On 03/11/2017 09:37 PM, Edward Capriolo wrote:


Benchmarks are great for FUDly blog posts. Real world work
loads matter more. Every NoSQL vendor wins their benchmarks.

Re: scylladb

[Kant Kodali]

Here is the Java version http://docs.paralleluniverse.co/quasar/ but I
still don't see how user level scheduling can be beneficial (This is a well
debated problem)? How can this add to the performance? or say why is user
level scheduling necessary Given the Thread per core design and the
callback mechanism?

On Sat, Mar 11, 2017 at 12:51 PM, Avi Kivity  wrote:

> Scylla uses a the seastar framework, which provides for both user-level
> thread scheduling and simple run-to-completion tasks.
>
> Huge pages are limited to 2MB (and 1GB, but these aren't available as
> transparent hugepages).
>
>
> On 03/11/2017 10:26 PM, Kant Kodali wrote:
>
> @Dor
>
> 1) You guys have a CPU scheduler? you mean user level thread Scheduler
> that maps user level threads to kernel level threads? I thought C++ by
> default creates native kernel threads but sure nothing will stop someone to
> create a user level scheduling library if that's what you are talking about?
> 2) How can one create THP of size 1KB? According to this post
> 
>  it
> looks like the valid values 2MB and 1GB.
>
> Thanks,
> kant
>
> On Sat, Mar 11, 2017 at 11:41 AM, Avi Kivity  wrote:
>
>> Agreed, I'd recommend to treat benchmarks as a rough guide to see where
>> there is potential, and follow through with your own tests.
>>
>> On 03/11/2017 09:37 PM, Edward Capriolo wrote:
>>
>>
>> Benchmarks are great for FUDly blog posts. Real world work loads matter
>> more. Every NoSQL vendor wins their benchmarks.
>>
>>
>>
>
>
>
>

Re: scylladb

[Avi Kivity]

Scylla uses a the seastar framework, which provides for both user-level 
thread scheduling and simple run-to-completion tasks.


Huge pages are limited to 2MB (and 1GB, but these aren't available as 
transparent hugepages).


On 03/11/2017 10:26 PM, Kant Kodali wrote:

@Dor

1) You guys have a CPU scheduler? you mean user level thread Scheduler 
that maps user level threads to kernel level threads? I thought C++ by 
default creates native kernel threads but sure nothing will stop 
someone to create a user level scheduling library if that's what you 
are talking about?
2) How can one create THP of size 1KB? According to this post 
 it 
looks like the valid values 2MB and 1GB.


Thanks,
kant

On Sat, Mar 11, 2017 at 11:41 AM, Avi Kivity > wrote:


Agreed, I'd recommend to treat benchmarks as a rough guide to see
where there is potential, and follow through with your own tests.

On 03/11/2017 09:37 PM, Edward Capriolo wrote:


Benchmarks are great for FUDly blog posts. Real world work loads
matter more. Every NoSQL vendor wins their benchmarks.

Re: scylladb

[Kant Kodali]

@Dor

1) You guys have a CPU scheduler? you mean user level thread Scheduler that
maps user level threads to kernel level threads? I thought C++ by default
creates native kernel threads but sure nothing will stop someone to create
a user level scheduling library if that's what you are talking about?
2) How can one create THP of size 1KB? According to this post

it
looks like the valid values 2MB and 1GB.

Thanks,
kant

On Sat, Mar 11, 2017 at 11:41 AM, Avi Kivity  wrote:

> Agreed, I'd recommend to treat benchmarks as a rough guide to see where
> there is potential, and follow through with your own tests.
>
> On 03/11/2017 09:37 PM, Edward Capriolo wrote:
>
>
> Benchmarks are great for FUDly blog posts. Real world work loads matter
> more. Every NoSQL vendor wins their benchmarks.
>
>
>

Re: scylladb

[Avi Kivity]

Agreed, I'd recommend to treat benchmarks as a rough guide to see where 
there is potential, and follow through with your own tests.


On 03/11/2017 09:37 PM, Edward Capriolo wrote:


Benchmarks are great for FUDly blog posts. Real world work loads 
matter more. Every NoSQL vendor wins their benchmarks.

Re: scylladb

[Avi Kivity]

Here's a test (by Samsung MSL) comparing Scylla to Cassandra 3.9:

http://www.scylladb.com/2017/02/15/scylladb-vs-cassandra-performance-benchmark-samsung/

there's a link at the end to the original report.

On 03/11/2017 09:08 PM, Bhuvan Rawal wrote:
"Lastly, why don't you test Scylla yourself?  It's pretty easy to set 
up, there's nothing to tune."
 - The details are indeed compelling to have a go ahead and test it 
for specific use case.


If it works out good it can lead to good cost cut in infra costs as 
well as having to manage less servers plus probably less time to 
bootstrap & decommission nodes!


It will also be interesting to have a benchmark with Cassandra 3 
version as well, as the new storage engine is said to have better 
performance:
https://www.datastax.com/2015/12/storage-engine-30 
<https://www.datastax.com/2015/12/storage-engine-30>


Regards,
Bhuvan

On Sat, Mar 11, 2017 at 2:59 PM, Avi Kivity <a...@scylladb.com 
<mailto:a...@scylladb.com>> wrote:


There is no magic 10X bullet.  It's a mix of multiple factors,
which can come up to less than 10X in some circumstances and more
than 10X in others, as has been reported on this thread by others.

TPC doesn't give _any_ advantage when you have just one core, and
can give more than 10X on a machine with a large number of cores. 
These are becoming more and more common, think of the recent AMD

Naples announcement; with 32 cores per socket you can have 128
logical cores in a two-socket server; or the AWS i3.16xlarge
instance with 32 cores / 64 vcpus.

You're welcome to browse our site to learn more about the
architecture, or watch this technical talk [1] I gave in QConSF
that highlights some of the techniques we use.

Of course it's possible to mistune Cassandra to give bad results,
that is why we spent a lot more time tuning Cassandra and
documenting everything than we spent on Scylla.  You can read the
report in [2], it is very detailed, and provides a wealth of
metrics like you'd expect.

I'm not going to comment about the Aerospike numbers, I haven't
studied them in detail.  And no, you can't multiply results like
that unless they were done with very similar configurations and
test harnesses.

Lastly, why don't you test Scylla yourself?  It's pretty easy to
set up, there's nothing to tune.

Avi

[1] https://www.infoq.com/presentations/scylladb
<https://www.infoq.com/presentations/scylladb>
[2]
http://www.scylladb.com/technology/cassandra-vs-scylla-benchmark-cluster-1/

<http://www.scylladb.com/technology/cassandra-vs-scylla-benchmark-cluster-1/>



On 03/10/2017 06:58 PM, Bhuvan Rawal wrote:

Agreed C++ gives an added advantage to talk to underlying
hardware with better efficiency, it sound good but can a pice of
code written in C++ give 1000% throughput than a Java app? Is TPC
design 10X more performant than SEDA arch?

And if C/C++ is indeed that fast how can Aerospike (which is
itself written in C) claim to be 10X faster than Scylla here
http://www.aerospike.com/benchmarks/scylladb-initial/
<http://www.aerospike.com/benchmarks/scylladb-initial/> ?
(Combining your's and aerospike's benchmarks it appears that
Aerospike is 100X performant than C* - I highly doubt that!! )

For a moment lets forget about evaluating 2 different databases,
one can observe 10X performance difference between a mistuned
cassandra cluster and one thats tuned as per data model - there
are so many Tunables in yaml as well as table configs.

Idea is - in order to strengthen your claim, you need to provide
complete system metrics (Disk, CPU, Network), the OPS increase
starts to decay along with the configs used. Having plain ops per
second and 99p latency is blackbox.

Regards,
Bhuvan

On Fri, Mar 10, 2017 at 12:47 PM, Avi Kivity <a...@scylladb.com
    <mailto:a...@scylladb.com>> wrote:

ScyllaDB engineer here.

C++ is really an enabling technology here. It is directly
responsible for a small fraction of the gain by executing
faster than Java.  But it is indirectly responsible for the
gain by allowing us direct control over memory and
threading.  Just as an example, Scylla starts by taking over
almost all of the machine's memory, and dynamically assigning
it to memtables, cache, and working memory needed to handle
requests in flight.  Memory is statically partitioned across
cores, allowing us to exploit NUMA fully.  You can't do these
things in Java.

I would say the major contributors to Scylla performance are:
 - thread-per-core design
 - replacement of the page cache with a row cache
 - careful attention to many small details, each contributing
a little, but

Re: scylladb

[Edward Capriolo]

On Sat, Mar 11, 2017 at 2:08 PM, Bhuvan Rawal <bhu1ra...@gmail.com> wrote:

> "Lastly, why don't you test Scylla yourself?  It's pretty easy to set up,
> there's nothing to tune."
>  - The details are indeed compelling to have a go ahead and test it for
> specific use case.
>
> If it works out good it can lead to good cost cut in infra costs as well
> as having to manage less servers plus probably less time to bootstrap &
> decommission nodes!
>
> It will also be interesting to have a benchmark with Cassandra 3 version
> as well, as the new storage engine is said to have better performance:
> https://www.datastax.com/2015/12/storage-engine-30
>
> Regards,
> Bhuvan
>
> On Sat, Mar 11, 2017 at 2:59 PM, Avi Kivity <a...@scylladb.com> wrote:
>
>> There is no magic 10X bullet.  It's a mix of multiple factors, which can
>> come up to less than 10X in some circumstances and more than 10X in others,
>> as has been reported on this thread by others.
>>
>> TPC doesn't give _any_ advantage when you have just one core, and can
>> give more than 10X on a machine with a large number of cores.  These are
>> becoming more and more common, think of the recent AMD Naples announcement;
>> with 32 cores per socket you can have 128 logical cores in a two-socket
>> server; or the AWS i3.16xlarge instance with 32 cores / 64 vcpus.
>>
>> You're welcome to browse our site to learn more about the architecture,
>> or watch this technical talk [1] I gave in QConSF that highlights some of
>> the techniques we use.
>>
>> Of course it's possible to mistune Cassandra to give bad results, that is
>> why we spent a lot more time tuning Cassandra and documenting everything
>> than we spent on Scylla.  You can read the report in [2], it is very
>> detailed, and provides a wealth of metrics like you'd expect.
>>
>> I'm not going to comment about the Aerospike numbers, I haven't studied
>> them in detail.  And no, you can't multiply results like that unless they
>> were done with very similar configurations and test harnesses.
>>
>> Lastly, why don't you test Scylla yourself?  It's pretty easy to set up,
>> there's nothing to tune.
>>
>> Avi
>>
>> [1] https://www.infoq.com/presentations/scylladb
>> [2] http://www.scylladb.com/technology/cassandra-vs-scylla-bench
>> mark-cluster-1/
>>
>>
>> On 03/10/2017 06:58 PM, Bhuvan Rawal wrote:
>>
>> Agreed C++ gives an added advantage to talk to underlying hardware with
>> better efficiency, it sound good but can a pice of code written in C++ give
>> 1000% throughput than a Java app? Is TPC design 10X more performant than
>> SEDA arch?
>>
>> And if C/C++ is indeed that fast how can Aerospike (which is itself
>> written in C) claim to be 10X faster than Scylla here
>> http://www.aerospike.com/benchmarks/scylladb-initial/ ? (Combining
>> your's and aerospike's benchmarks it appears that Aerospike is 100X
>> performant than C* - I highly doubt that!! )
>>
>> For a moment lets forget about evaluating 2 different databases, one can
>> observe 10X performance difference between a mistuned cassandra cluster and
>> one thats tuned as per data model - there are so many Tunables in yaml as
>> well as table configs.
>>
>> Idea is - in order to strengthen your claim, you need to provide complete
>> system metrics (Disk, CPU, Network), the OPS increase starts to decay along
>> with the configs used. Having plain ops per second and 99p latency is
>> blackbox.
>>
>> Regards,
>> Bhuvan
>>
>> On Fri, Mar 10, 2017 at 12:47 PM, Avi Kivity <a...@scylladb.com> wrote:
>>
>>> ScyllaDB engineer here.
>>>
>>> C++ is really an enabling technology here. It is directly responsible
>>> for a small fraction of the gain by executing faster than Java.  But it is
>>> indirectly responsible for the gain by allowing us direct control over
>>> memory and threading.  Just as an example, Scylla starts by taking over
>>> almost all of the machine's memory, and dynamically assigning it to
>>> memtables, cache, and working memory needed to handle requests in flight.
>>> Memory is statically partitioned across cores, allowing us to exploit NUMA
>>> fully.  You can't do these things in Java.
>>>
>>> I would say the major contributors to Scylla performance are:
>>>  - thread-per-core design
>>>  - replacement of the page cache with a row cache
>>>  - careful attention to many small details, each contributing a little,
>>> but with a large overall impact

Re: scylladb

[Bhuvan Rawal]

"Lastly, why don't you test Scylla yourself?  It's pretty easy to set up,
there's nothing to tune."
 - The details are indeed compelling to have a go ahead and test it for
specific use case.

If it works out good it can lead to good cost cut in infra costs as well as
having to manage less servers plus probably less time to bootstrap &
decommission nodes!

It will also be interesting to have a benchmark with Cassandra 3 version as
well, as the new storage engine is said to have better performance:
https://www.datastax.com/2015/12/storage-engine-30

Regards,
Bhuvan

On Sat, Mar 11, 2017 at 2:59 PM, Avi Kivity <a...@scylladb.com> wrote:

> There is no magic 10X bullet.  It's a mix of multiple factors, which can
> come up to less than 10X in some circumstances and more than 10X in others,
> as has been reported on this thread by others.
>
> TPC doesn't give _any_ advantage when you have just one core, and can give
> more than 10X on a machine with a large number of cores.  These are
> becoming more and more common, think of the recent AMD Naples announcement;
> with 32 cores per socket you can have 128 logical cores in a two-socket
> server; or the AWS i3.16xlarge instance with 32 cores / 64 vcpus.
>
> You're welcome to browse our site to learn more about the architecture, or
> watch this technical talk [1] I gave in QConSF that highlights some of the
> techniques we use.
>
> Of course it's possible to mistune Cassandra to give bad results, that is
> why we spent a lot more time tuning Cassandra and documenting everything
> than we spent on Scylla.  You can read the report in [2], it is very
> detailed, and provides a wealth of metrics like you'd expect.
>
> I'm not going to comment about the Aerospike numbers, I haven't studied
> them in detail.  And no, you can't multiply results like that unless they
> were done with very similar configurations and test harnesses.
>
> Lastly, why don't you test Scylla yourself?  It's pretty easy to set up,
> there's nothing to tune.
>
> Avi
>
> [1] https://www.infoq.com/presentations/scylladb
> [2] http://www.scylladb.com/technology/cassandra-vs-scylla-
> benchmark-cluster-1/
>
>
> On 03/10/2017 06:58 PM, Bhuvan Rawal wrote:
>
> Agreed C++ gives an added advantage to talk to underlying hardware with
> better efficiency, it sound good but can a pice of code written in C++ give
> 1000% throughput than a Java app? Is TPC design 10X more performant than
> SEDA arch?
>
> And if C/C++ is indeed that fast how can Aerospike (which is itself
> written in C) claim to be 10X faster than Scylla here
> http://www.aerospike.com/benchmarks/scylladb-initial/ ? (Combining your's
> and aerospike's benchmarks it appears that Aerospike is 100X performant
> than C* - I highly doubt that!! )
>
> For a moment lets forget about evaluating 2 different databases, one can
> observe 10X performance difference between a mistuned cassandra cluster and
> one thats tuned as per data model - there are so many Tunables in yaml as
> well as table configs.
>
> Idea is - in order to strengthen your claim, you need to provide complete
> system metrics (Disk, CPU, Network), the OPS increase starts to decay along
> with the configs used. Having plain ops per second and 99p latency is
> blackbox.
>
> Regards,
> Bhuvan
>
> On Fri, Mar 10, 2017 at 12:47 PM, Avi Kivity <a...@scylladb.com> wrote:
>
>> ScyllaDB engineer here.
>>
>> C++ is really an enabling technology here. It is directly responsible for
>> a small fraction of the gain by executing faster than Java.  But it is
>> indirectly responsible for the gain by allowing us direct control over
>> memory and threading.  Just as an example, Scylla starts by taking over
>> almost all of the machine's memory, and dynamically assigning it to
>> memtables, cache, and working memory needed to handle requests in flight.
>> Memory is statically partitioned across cores, allowing us to exploit NUMA
>> fully.  You can't do these things in Java.
>>
>> I would say the major contributors to Scylla performance are:
>>  - thread-per-core design
>>  - replacement of the page cache with a row cache
>>  - careful attention to many small details, each contributing a little,
>> but with a large overall impact
>>
>> While I'm here I can say that performance is not the only goal here, it
>> is stable and predictable performance over varying loads and during
>> maintenance operations like repair, without any special tuning.  We measure
>> the amount of CPU and I/O spent on foreground (user) and background
>> (maintenance) tasks and divide them fairly.  This work is not complete but
>> already makes operating Scylla a lot simpler.
>>
>

Re: scylladb

[Avi Kivity]

There is no magic 10X bullet.  It's a mix of multiple factors, which can 
come up to less than 10X in some circumstances and more than 10X in 
others, as has been reported on this thread by others.


TPC doesn't give _any_ advantage when you have just one core, and can 
give more than 10X on a machine with a large number of cores. These are 
becoming more and more common, think of the recent AMD Naples 
announcement; with 32 cores per socket you can have 128 logical cores in 
a two-socket server; or the AWS i3.16xlarge instance with 32 cores / 64 
vcpus.


You're welcome to browse our site to learn more about the architecture, 
or watch this technical talk [1] I gave in QConSF that highlights some 
of the techniques we use.


Of course it's possible to mistune Cassandra to give bad results, that 
is why we spent a lot more time tuning Cassandra and documenting 
everything than we spent on Scylla.  You can read the report in [2], it 
is very detailed, and provides a wealth of metrics like you'd expect.


I'm not going to comment about the Aerospike numbers, I haven't studied 
them in detail.  And no, you can't multiply results like that unless 
they were done with very similar configurations and test harnesses.


Lastly, why don't you test Scylla yourself?  It's pretty easy to set up, 
there's nothing to tune.


Avi

[1] https://www.infoq.com/presentations/scylladb
[2] 
http://www.scylladb.com/technology/cassandra-vs-scylla-benchmark-cluster-1/


On 03/10/2017 06:58 PM, Bhuvan Rawal wrote:
Agreed C++ gives an added advantage to talk to underlying hardware 
with better efficiency, it sound good but can a pice of code written 
in C++ give 1000% throughput than a Java app? Is TPC design 10X more 
performant than SEDA arch?


And if C/C++ is indeed that fast how can Aerospike (which is itself 
written in C) claim to be 10X faster than Scylla here 
http://www.aerospike.com/benchmarks/scylladb-initial/ ? (Combining 
your's and aerospike's benchmarks it appears that Aerospike is 100X 
performant than C* - I highly doubt that!! )


For a moment lets forget about evaluating 2 different databases, one 
can observe 10X performance difference between a mistuned cassandra 
cluster and one thats tuned as per data model - there are so many 
Tunables in yaml as well as table configs.


Idea is - in order to strengthen your claim, you need to provide 
complete system metrics (Disk, CPU, Network), the OPS increase starts 
to decay along with the configs used. Having plain ops per second and 
99p latency is blackbox.


Regards,
Bhuvan

On Fri, Mar 10, 2017 at 12:47 PM, Avi Kivity <a...@scylladb.com 
<mailto:a...@scylladb.com>> wrote:


ScyllaDB engineer here.

C++ is really an enabling technology here. It is directly
responsible for a small fraction of the gain by executing faster
than Java.  But it is indirectly responsible for the gain by
allowing us direct control over memory and threading.  Just as an
example, Scylla starts by taking over almost all of the machine's
memory, and dynamically assigning it to memtables, cache, and
working memory needed to handle requests in flight.  Memory is
statically partitioned across cores, allowing us to exploit NUMA
fully.  You can't do these things in Java.

I would say the major contributors to Scylla performance are:
 - thread-per-core design
 - replacement of the page cache with a row cache
 - careful attention to many small details, each contributing a
little, but with a large overall impact

While I'm here I can say that performance is not the only goal
here, it is stable and predictable performance over varying loads
and during maintenance operations like repair, without any special
tuning.  We measure the amount of CPU and I/O spent on foreground
(user) and background (maintenance) tasks and divide them fairly.
This work is not complete but already makes operating Scylla a lot
simpler.


On 03/10/2017 01:42 AM, Kant Kodali wrote:

I dont think ScyllaDB performance is because of C++. The design
decisions in scylladb are indeed different from Cassandra such as
getting rid of SEDA and moving to TPC and so on.

If someone thinks it is because of C++ then just show the
benchmarks that proves it is indeed the C++ which gave 10X
performance boost as ScyllaDB claims instead of stating it.


On Thu, Mar 9, 2017 at 3:22 PM, Richard L. Burton III
<mrbur...@gmail.com <mailto:mrbur...@gmail.com>> wrote:

They spend an enormous amount of time focusing on
performance. You can expect them to continue on with their
optimization and keep crushing it.

    P.S., I don't work for ScyllaDB.

On Thu, Mar 9, 2017 at 6:02 PM, Rakesh Kumar
<rakeshkumar...@outlook.com
<mailto:rakeshkumar...@outlook.com>> wrote:

In all of their presentation they keep harping on the
  

Re: scylladb

[benjamin roth]

Thanks a lot for your detailed explanation!
I am very curious about the future development of Scylladb! Especially
about mvs and lwt!

Am 11.03.2017 02:05 schrieb "Dor Laor" <d...@scylladb.com>:

> On Fri, Mar 10, 2017 at 4:45 PM, Kant Kodali <k...@peernova.com> wrote:
>
>> http://performanceterracotta.blogspot.com/2012/09/numa-java.html
>> http://docs.oracle.com/javase/7/docs/technotes/guides/vm/per
>> formance-enhancements-7.html
>> http://openjdk.java.net/jeps/163
>>
>>
> Java can exploit NUMA but it's not as a efficient as can be done in c++.
> Andrea Arcangeli is the engineer behind Linux transparent huge pages(THP),
> he
> reported to me and the idea belongs to Avi. We did it for KVM's sake but
> it was designed to any long running process like Cassandra.
> However, the entire software stack should be aware. If you get a huge page
> (2MB)
> but keep in it only 1KB you waste lots of mem. On top of this, threads
> need to
> touch their data structures and they need to be well aligned, otherwise
> the memory
> page will bounce between the different cores.
> With Cassandra it gets more complicated since there is a heap and off-heap
> data.
>
> Do programmers really track their data alignment? I doubt it.
> Do users run C* with the JVM numa options and the right Linux THP options?
> Again, I doubt.
>
> Scylla on the other side is designed for NUMA. We have 2-level sharding.
> The inner shards are transparent
> to the user and are per-core (hyper thread). Such a shard access RAM only
> within its numa node. Memory
> is bonded to each thread/numa node. We have our own malloc allocator built
> for this scheme.
>
>
>
>> If scyllaDB has efficient Secondary indexes, LWT and MV's then that is
>> something. I would be glad to see how they perform.
>>
>>
> MV will be in 1.8, we haven't measured performance yet. We did measure our
> counter implementation
> and it looks promising (4X better throughput and 4X better latency on a
> 8-core machine).
> The not-written yet LWT will kick-a** since our fully async engine is
> ideal for the larger number
> of round trips the LWT needs.
>
> This is with the Linux tcp stack, once we'll use our dpdk one, performance
> will improve further ;)
>
>
>
>>
>> On Fri, Mar 10, 2017 at 10:45 AM, Dor Laor <d...@scylladb.com> wrote:
>>
>>> Scylla isn't just about performance too.
>>>
>>> First, a disclaimer, I am a Scylla co-founder. I respect open source a
>>> lot,
>>> so you guys are welcome to shush me out of this thread. I only
>>> participate
>>> to provide value if I can (this is a thread about Scylla and our users
>>> are
>>> on our mailing list).
>>>
>>> Scylla is all about what Cassandra is plus:
>>>  - Efficient hardware utilization (scale-up, performance)
>>>  - Low tail latency
>>>  - Auto/dynamic tuning (no JVM tuning, we tune the OS ourselves, we have
>>> cpu scheduler,
>>>I/O userspace scheduler and more to come).
>>>  - SLA between compaction, repair, streaming and your r/w operations
>>>
>>> We started with a great foundation (C*) and wish to improve almost any
>>> aspect of it.
>>> Admittedly, we're way behind C* in terms of adoption. One need to start
>>> somewhere.
>>> However, users such as AppNexus run Scylla in production with 47
>>> physical nodes
>>> across 5 datacenters and their VP estimate that C* would have at least
>>> doubled the
>>> size. So this is equal for a 100-node C* cluster. Since we have the same
>>> gossip, murmur3 hash,
>>> CQL, nothing stops us to scale to 1,000 nodes. Another user (Mogujie)
>>> run 10s of TBs per node(!)
>>> in production.
>>>
>>> Also, since we try to compare Scylla and C* in a fair way, we invested a
>>> great deal of time
>>> to run C*. I can say it's not simple at all.
>>> Lastly, in a couple of months we'll reach parity in functionality with
>>> C* (counters are in 1.7 as experimental, in 1.8 counters will be stable and
>>> we'll have MV as experimental, LWT will be
>>> in the summer). We hope to collaborate with the C* community with the
>>> development of future
>>> features.
>>>
>>> Dor
>>>
>>>
>>> On Fri, Mar 10, 2017 at 10:19 AM, Jacques-Henri Berthemet <
>>> jacques-henri.berthe...@genesys.com> wrote:
>>>
>>>> Cassandra is not about pure performance, there are many other DBs that
>>>> are much faster than Ca

Re: scylladb

[Dor Laor]

On Fri, Mar 10, 2017 at 4:45 PM, Kant Kodali <k...@peernova.com> wrote:

> http://performanceterracotta.blogspot.com/2012/09/numa-java.html
> http://docs.oracle.com/javase/7/docs/technotes/guides/vm/
> performance-enhancements-7.html
> http://openjdk.java.net/jeps/163
>
>
Java can exploit NUMA but it's not as a efficient as can be done in c++.
Andrea Arcangeli is the engineer behind Linux transparent huge pages(THP),
he
reported to me and the idea belongs to Avi. We did it for KVM's sake but
it was designed to any long running process like Cassandra.
However, the entire software stack should be aware. If you get a huge page
(2MB)
but keep in it only 1KB you waste lots of mem. On top of this, threads need
to
touch their data structures and they need to be well aligned, otherwise the
memory
page will bounce between the different cores.
With Cassandra it gets more complicated since there is a heap and off-heap
data.

Do programmers really track their data alignment? I doubt it.
Do users run C* with the JVM numa options and the right Linux THP options?
Again, I doubt.

Scylla on the other side is designed for NUMA. We have 2-level sharding.
The inner shards are transparent
to the user and are per-core (hyper thread). Such a shard access RAM only
within its numa node. Memory
is bonded to each thread/numa node. We have our own malloc allocator built
for this scheme.



> If scyllaDB has efficient Secondary indexes, LWT and MV's then that is
> something. I would be glad to see how they perform.
>
>
MV will be in 1.8, we haven't measured performance yet. We did measure our
counter implementation
and it looks promising (4X better throughput and 4X better latency on a
8-core machine).
The not-written yet LWT will kick-a** since our fully async engine is ideal
for the larger number
of round trips the LWT needs.

This is with the Linux tcp stack, once we'll use our dpdk one, performance
will improve further ;)



>
> On Fri, Mar 10, 2017 at 10:45 AM, Dor Laor <d...@scylladb.com> wrote:
>
>> Scylla isn't just about performance too.
>>
>> First, a disclaimer, I am a Scylla co-founder. I respect open source a
>> lot,
>> so you guys are welcome to shush me out of this thread. I only participate
>> to provide value if I can (this is a thread about Scylla and our users are
>> on our mailing list).
>>
>> Scylla is all about what Cassandra is plus:
>>  - Efficient hardware utilization (scale-up, performance)
>>  - Low tail latency
>>  - Auto/dynamic tuning (no JVM tuning, we tune the OS ourselves, we have
>> cpu scheduler,
>>I/O userspace scheduler and more to come).
>>  - SLA between compaction, repair, streaming and your r/w operations
>>
>> We started with a great foundation (C*) and wish to improve almost any
>> aspect of it.
>> Admittedly, we're way behind C* in terms of adoption. One need to start
>> somewhere.
>> However, users such as AppNexus run Scylla in production with 47 physical
>> nodes
>> across 5 datacenters and their VP estimate that C* would have at least
>> doubled the
>> size. So this is equal for a 100-node C* cluster. Since we have the same
>> gossip, murmur3 hash,
>> CQL, nothing stops us to scale to 1,000 nodes. Another user (Mogujie) run
>> 10s of TBs per node(!)
>> in production.
>>
>> Also, since we try to compare Scylla and C* in a fair way, we invested a
>> great deal of time
>> to run C*. I can say it's not simple at all.
>> Lastly, in a couple of months we'll reach parity in functionality with C*
>> (counters are in 1.7 as experimental, in 1.8 counters will be stable and
>> we'll have MV as experimental, LWT will be
>> in the summer). We hope to collaborate with the C* community with the
>> development of future
>> features.
>>
>> Dor
>>
>>
>> On Fri, Mar 10, 2017 at 10:19 AM, Jacques-Henri Berthemet <
>> jacques-henri.berthe...@genesys.com> wrote:
>>
>>> Cassandra is not about pure performance, there are many other DBs that
>>> are much faster than Cassandra. Cassandra strength is all about
>>> scalability, performance increases in a linear way as you add more nodes.
>>> During Cassandra summit 2014 Apple said they have a 10k node cluster. The
>>> usual limiting factor is your disk write speed and latency, I don’t see how
>>> C++ changes anything in this regard unless you can cache all your data in
>>> memory.
>>>
>>>
>>>
>>> I’d be curious to know how ScyllaDB performs with a 100+ nodes cluster
>>> with PBs of data compared to Cassandra.
>>>
>>> *--*
>>>
>>> *Jacques-Henri Berthemet*
>>>
>>>
&g

Re: scylladb

[Kant Kodali]

http://performanceterracotta.blogspot.com/2012/09/numa-java.html
http://docs.oracle.com/javase/7/docs/technotes/guides/vm/performance-enhancements-7.html
http://openjdk.java.net/jeps/163

If scyllaDB has efficient Secondary indexes, LWT and MV's then that is
something. I would be glad to see how they perform.


On Fri, Mar 10, 2017 at 10:45 AM, Dor Laor <d...@scylladb.com> wrote:

> Scylla isn't just about performance too.
>
> First, a disclaimer, I am a Scylla co-founder. I respect open source a lot,
> so you guys are welcome to shush me out of this thread. I only participate
> to provide value if I can (this is a thread about Scylla and our users are
> on our mailing list).
>
> Scylla is all about what Cassandra is plus:
>  - Efficient hardware utilization (scale-up, performance)
>  - Low tail latency
>  - Auto/dynamic tuning (no JVM tuning, we tune the OS ourselves, we have
> cpu scheduler,
>I/O userspace scheduler and more to come).
>  - SLA between compaction, repair, streaming and your r/w operations
>
> We started with a great foundation (C*) and wish to improve almost any
> aspect of it.
> Admittedly, we're way behind C* in terms of adoption. One need to start
> somewhere.
> However, users such as AppNexus run Scylla in production with 47 physical
> nodes
> across 5 datacenters and their VP estimate that C* would have at least
> doubled the
> size. So this is equal for a 100-node C* cluster. Since we have the same
> gossip, murmur3 hash,
> CQL, nothing stops us to scale to 1,000 nodes. Another user (Mogujie) run
> 10s of TBs per node(!)
> in production.
>
> Also, since we try to compare Scylla and C* in a fair way, we invested a
> great deal of time
> to run C*. I can say it's not simple at all.
> Lastly, in a couple of months we'll reach parity in functionality with C*
> (counters are in 1.7 as experimental, in 1.8 counters will be stable and
> we'll have MV as experimental, LWT will be
> in the summer). We hope to collaborate with the C* community with the
> development of future
> features.
>
> Dor
>
>
> On Fri, Mar 10, 2017 at 10:19 AM, Jacques-Henri Berthemet <
> jacques-henri.berthe...@genesys.com> wrote:
>
>> Cassandra is not about pure performance, there are many other DBs that
>> are much faster than Cassandra. Cassandra strength is all about
>> scalability, performance increases in a linear way as you add more nodes.
>> During Cassandra summit 2014 Apple said they have a 10k node cluster. The
>> usual limiting factor is your disk write speed and latency, I don’t see how
>> C++ changes anything in this regard unless you can cache all your data in
>> memory.
>>
>>
>>
>> I’d be curious to know how ScyllaDB performs with a 100+ nodes cluster
>> with PBs of data compared to Cassandra.
>>
>> *--*
>>
>> *Jacques-Henri Berthemet*
>>
>>
>>
>> *From:* Rakesh Kumar [mailto:rakeshkumar...@outlook.com]
>> *Sent:* vendredi 10 mars 2017 09:58
>>
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: scylladb
>>
>>
>>
>> Cassanda vs Scylla is a valid comparison because they both are
>> compatible.  Scylla is a drop-in replacement for Cassandra.
>> Is Aerospike a drop-in replacement for Cassandra? If yes, and only if
>> yes, then the comparison is valid with Scylla.
>>
>>
>> --
>>
>> *From:* Bhuvan Rawal <bhu1ra...@gmail.com>
>> *To:* user@cassandra.apache.org
>> *Sent:* Friday, March 10, 2017 11:59 AM
>> *Subject:* Re: scylladb
>>
>>
>>
>> Agreed C++ gives an added advantage to talk to underlying hardware with
>> better efficiency, it sound good but can a pice of code written in C++ give
>> 1000% throughput than a Java app? Is TPC design 10X more performant than
>> SEDA arch?
>>
>>
>>
>> And if C/C++ is indeed that fast how can Aerospike (which is itself
>> written in C) claim to be 10X faster than Scylla here
>> http://www.aerospike.com/benchmarks/scylladb-initial/ ? (Combining
>> your's and aerospike's benchmarks it appears that Aerospike is 100X
>> performant than C* - I highly doubt that!! )
>>
>>
>>
>> For a moment lets forget about evaluating 2 different databases, one can
>> observe 10X performance difference between a mistuned cassandra cluster and
>> one thats tuned as per data model - there are so many Tunables in yaml as
>> well as table configs.
>>
>>
>>
>> Idea is - in order to strengthen your claim, you need to provide complete
>> system metrics (Disk, CPU, Network), the OPS increase starts to decay along
&

Re: scylladb

[Dor Laor]

Scylla isn't just about performance too.

First, a disclaimer, I am a Scylla co-founder. I respect open source a lot,
so you guys are welcome to shush me out of this thread. I only participate
to provide value if I can (this is a thread about Scylla and our users are
on our mailing list).

Scylla is all about what Cassandra is plus:
 - Efficient hardware utilization (scale-up, performance)
 - Low tail latency
 - Auto/dynamic tuning (no JVM tuning, we tune the OS ourselves, we have
cpu scheduler,
   I/O userspace scheduler and more to come).
 - SLA between compaction, repair, streaming and your r/w operations

We started with a great foundation (C*) and wish to improve almost any
aspect of it.
Admittedly, we're way behind C* in terms of adoption. One need to start
somewhere.
However, users such as AppNexus run Scylla in production with 47 physical
nodes
across 5 datacenters and their VP estimate that C* would have at least
doubled the
size. So this is equal for a 100-node C* cluster. Since we have the same
gossip, murmur3 hash,
CQL, nothing stops us to scale to 1,000 nodes. Another user (Mogujie) run
10s of TBs per node(!)
in production.

Also, since we try to compare Scylla and C* in a fair way, we invested a
great deal of time
to run C*. I can say it's not simple at all.
Lastly, in a couple of months we'll reach parity in functionality with C*
(counters are in 1.7 as experimental, in 1.8 counters will be stable and
we'll have MV as experimental, LWT will be
in the summer). We hope to collaborate with the C* community with the
development of future
features.

Dor


On Fri, Mar 10, 2017 at 10:19 AM, Jacques-Henri Berthemet <
jacques-henri.berthe...@genesys.com> wrote:

> Cassandra is not about pure performance, there are many other DBs that are
> much faster than Cassandra. Cassandra strength is all about scalability,
> performance increases in a linear way as you add more nodes. During
> Cassandra summit 2014 Apple said they have a 10k node cluster. The usual
> limiting factor is your disk write speed and latency, I don’t see how C++
> changes anything in this regard unless you can cache all your data in
> memory.
>
>
>
> I’d be curious to know how ScyllaDB performs with a 100+ nodes cluster
> with PBs of data compared to Cassandra.
>
> *--*
>
> *Jacques-Henri Berthemet*
>
>
>
> *From:* Rakesh Kumar [mailto:rakeshkumar...@outlook.com]
> *Sent:* vendredi 10 mars 2017 09:58
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: scylladb
>
>
>
> Cassanda vs Scylla is a valid comparison because they both are
> compatible.  Scylla is a drop-in replacement for Cassandra.
> Is Aerospike a drop-in replacement for Cassandra? If yes, and only if yes,
> then the comparison is valid with Scylla.
>
>
> --
>
> *From:* Bhuvan Rawal <bhu1ra...@gmail.com>
> *To:* user@cassandra.apache.org
> *Sent:* Friday, March 10, 2017 11:59 AM
> *Subject:* Re: scylladb
>
>
>
> Agreed C++ gives an added advantage to talk to underlying hardware with
> better efficiency, it sound good but can a pice of code written in C++ give
> 1000% throughput than a Java app? Is TPC design 10X more performant than
> SEDA arch?
>
>
>
> And if C/C++ is indeed that fast how can Aerospike (which is itself
> written in C) claim to be 10X faster than Scylla here
> http://www.aerospike.com/benchmarks/scylladb-initial/ ? (Combining your's
> and aerospike's benchmarks it appears that Aerospike is 100X performant
> than C* - I highly doubt that!! )
>
>
>
> For a moment lets forget about evaluating 2 different databases, one can
> observe 10X performance difference between a mistuned cassandra cluster and
> one thats tuned as per data model - there are so many Tunables in yaml as
> well as table configs.
>
>
>
> Idea is - in order to strengthen your claim, you need to provide complete
> system metrics (Disk, CPU, Network), the OPS increase starts to decay along
> with the configs used. Having plain ops per second and 99p latency is
> blackbox.
>
>
>
> Regards,
>
> Bhuvan
>
>
>
> On Fri, Mar 10, 2017 at 12:47 PM, Avi Kivity <a...@scylladb.com> wrote:
>
> ScyllaDB engineer here.
>
> C++ is really an enabling technology here. It is directly responsible for
> a small fraction of the gain by executing faster than Java.  But it is
> indirectly responsible for the gain by allowing us direct control over
> memory and threading.  Just as an example, Scylla starts by taking over
> almost all of the machine's memory, and dynamically assigning it to
> memtables, cache, and working memory needed to handle requests in flight.
> Memory is statically partitioned across cores, allowing us to exploit NUMA
> fully.  You can't do these things in Java.
>
> I wo

RE: scylladb

[Jacques-Henri Berthemet]

Cassandra is not about pure performance, there are many other DBs that are much 
faster than Cassandra. Cassandra strength is all about scalability, performance 
increases in a linear way as you add more nodes. During Cassandra summit 2014 
Apple said they have a 10k node cluster. The usual limiting factor is your disk 
write speed and latency, I don’t see how C++ changes anything in this regard 
unless you can cache all your data in memory.

I’d be curious to know how ScyllaDB performs with a 100+ nodes cluster with PBs 
of data compared to Cassandra.
--
Jacques-Henri Berthemet

From: Rakesh Kumar [mailto:rakeshkumar...@outlook.com]
Sent: vendredi 10 mars 2017 09:58
To: user@cassandra.apache.org
Subject: Re: scylladb

Cassanda vs Scylla is a valid comparison because they both are compatible.  
Scylla is a drop-in replacement for Cassandra.
Is Aerospike a drop-in replacement for Cassandra? If yes, and only if yes, then 
the comparison is valid with Scylla.


From: Bhuvan Rawal <bhu1ra...@gmail.com<mailto:bhu1ra...@gmail.com>>
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Sent: Friday, March 10, 2017 11:59 AM
Subject: Re: scylladb

Agreed C++ gives an added advantage to talk to underlying hardware with better 
efficiency, it sound good but can a pice of code written in C++ give 1000% 
throughput than a Java app? Is TPC design 10X more performant than SEDA arch?

And if C/C++ is indeed that fast how can Aerospike (which is itself written in 
C) claim to be 10X faster than Scylla here 
http://www.aerospike.com/benchmarks/scylladb-initial/<http://www.aerospike.com/benchmarks/scylladb-initial/>
 ? (Combining your's and aerospike's benchmarks it appears that Aerospike is 
100X performant than C* - I highly doubt that!! )

For a moment lets forget about evaluating 2 different databases, one can 
observe 10X performance difference between a mistuned cassandra cluster and one 
thats tuned as per data model - there are so many Tunables in yaml as well as 
table configs.

Idea is - in order to strengthen your claim, you need to provide complete 
system metrics (Disk, CPU, Network), the OPS increase starts to decay along 
with the configs used. Having plain ops per second and 99p latency is blackbox.

Regards,
Bhuvan

On Fri, Mar 10, 2017 at 12:47 PM, Avi Kivity 
<a...@scylladb.com<mailto:a...@scylladb.com>> wrote:
ScyllaDB engineer here.

C++ is really an enabling technology here. It is directly responsible for a 
small fraction of the gain by executing faster than Java.  But it is indirectly 
responsible for the gain by allowing us direct control over memory and 
threading.  Just as an example, Scylla starts by taking over almost all of the 
machine's memory, and dynamically assigning it to memtables, cache, and working 
memory needed to handle requests in flight.  Memory is statically partitioned 
across cores, allowing us to exploit NUMA fully.  You can't do these things in 
Java.

I would say the major contributors to Scylla performance are:
 - thread-per-core design
 - replacement of the page cache with a row cache
 - careful attention to many small details, each contributing a little, but 
with a large overall impact

While I'm here I can say that performance is not the only goal here, it is 
stable and predictable performance over varying loads and during maintenance 
operations like repair, without any special tuning.  We measure the amount of 
CPU and I/O spent on foreground (user) and background (maintenance) tasks and 
divide them fairly.  This work is not complete but already makes operating 
Scylla a lot simpler.


On 03/10/2017 01:42 AM, Kant Kodali wrote:
I dont think ScyllaDB performance is because of C++. The design decisions in 
scylladb are indeed different from Cassandra such as getting rid of SEDA and 
moving to TPC and so on.

If someone thinks it is because of C++ then just show the benchmarks that 
proves it is indeed the C++ which gave 10X performance boost as ScyllaDB claims 
instead of stating it.


On Thu, Mar 9, 2017 at 3:22 PM, Richard L. Burton III 
<mrbur...@gmail.com<mailto:mrbur...@gmail.com>> wrote:
They spend an enormous amount of time focusing on performance. You can expect 
them to continue on with their optimization and keep crushing it.

P.S., I don't work for ScyllaDB.

On Thu, Mar 9, 2017 at 6:02 PM, Rakesh Kumar 
<rakeshkumar...@outlook.com<mailto:rakeshkumar...@outlook.com>> wrote:
In all of their presentation they keep harping on the fact that scylladb is 
written in C++ and does not carry the overhead of Java.  Still the difference 
looks staggering.
__ __
From: daemeon reiydelle <daeme...@gmail.com<mailto:daeme...@gmail.com>>
Sent: Thursday, March 9, 2017 14:21
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: scylladb

The comparison is fair, and conservative. Did substanti

Re: scylladb

[Rakesh Kumar]

Cassanda vs Scylla is a valid comparison because they both are compatible.  
Scylla is a drop-in replacement for Cassandra.
Is Aerospike a drop-in replacement for Cassandra? If yes, and only if yes, then 
the comparison is valid with Scylla.



From: Bhuvan Rawal <bhu1ra...@gmail.com>
To: user@cassandra.apache.org
Sent: Friday, March 10, 2017 11:59 AM
Subject: Re: scylladb

Agreed C++ gives an added advantage to talk to underlying hardware with better 
efficiency, it sound good but can a pice of code written in C++ give 1000% 
throughput than a Java app? Is TPC design 10X more performant than SEDA arch?

And if C/C++ is indeed that fast how can Aerospike (which is itself written in 
C) claim to be 10X faster than Scylla here 
http://www.aerospike.com/benchmarks/scylladb-initial/ ? (Combining your's and 
aerospike's benchmarks it appears that Aerospike is 100X performant than C* - I 
highly doubt that!! )

For a moment lets forget about evaluating 2 different databases, one can 
observe 10X performance difference between a mistuned cassandra cluster and one 
thats tuned as per data model - there are so many Tunables in yaml as well as 
table configs.

Idea is - in order to strengthen your claim, you need to provide complete 
system metrics (Disk, CPU, Network), the OPS increase starts to decay along 
with the configs used. Having plain ops per second and 99p latency is blackbox.

Regards,
Bhuvan

On Fri, Mar 10, 2017 at 12:47 PM, Avi Kivity 
<a...@scylladb.com<mailto:a...@scylladb.com>> wrote:
ScyllaDB engineer here.

C++ is really an enabling technology here. It is directly responsible for a 
small fraction of the gain by executing faster than Java.  But it is indirectly 
responsible for the gain by allowing us direct control over memory and 
threading.  Just as an example, Scylla starts by taking over almost all of the 
machine's memory, and dynamically assigning it to memtables, cache, and working 
memory needed to handle requests in flight.  Memory is statically partitioned 
across cores, allowing us to exploit NUMA fully.  You can't do these things in 
Java.

I would say the major contributors to Scylla performance are:
 - thread-per-core design
 - replacement of the page cache with a row cache
 - careful attention to many small details, each contributing a little, but 
with a large overall impact

While I'm here I can say that performance is not the only goal here, it is 
stable and predictable performance over varying loads and during maintenance 
operations like repair, without any special tuning.  We measure the amount of 
CPU and I/O spent on foreground (user) and background (maintenance) tasks and 
divide them fairly.  This work is not complete but already makes operating 
Scylla a lot simpler.


On 03/10/2017 01:42 AM, Kant Kodali wrote:
I dont think ScyllaDB performance is because of C++. The design decisions in 
scylladb are indeed different from Cassandra such as getting rid of SEDA and 
moving to TPC and so on.

If someone thinks it is because of C++ then just show the benchmarks that 
proves it is indeed the C++ which gave 10X performance boost as ScyllaDB claims 
instead of stating it.


On Thu, Mar 9, 2017 at 3:22 PM, Richard L. Burton III 
<mrbur...@gmail.com<mailto:mrbur...@gmail.com>> wrote:
They spend an enormous amount of time focusing on performance. You can expect 
them to continue on with their optimization and keep crushing it.

P.S., I don't work for ScyllaDB.

On Thu, Mar 9, 2017 at 6:02 PM, Rakesh Kumar 
<rakeshkumar...@outlook.com<mailto:rakeshkumar...@outlook.com>> wrote:
In all of their presentation they keep harping on the fact that scylladb is 
written in C++ and does not carry the overhead of Java.  Still the difference 
looks staggering.
__ __
From: daemeon reiydelle <daeme...@gmail.com<mailto:daeme...@gmail.com>>
Sent: Thursday, March 9, 2017 14:21
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: scylladb

The comparison is fair, and conservative. Did substantial performance 
comparisons for two clients, both results returned throughputs that were faster 
than the published comparisons (15x as I recall). At that time the client 
preferred to utilize a Cass COTS solution and use a caching solution for OLA 
compliance.


...

Daemeon C.M. Reiydelle
USA (+1) 415.501.0198
London (+44) (0) 20 8144 9872

On Thu, Mar 9, 2017 at 11:04 AM, Robin Verlangen 
<ro...@us2.nl<mailto:ro...@us2.nl><mailto:robin@us2 .nl<mailto:ro...@us2.nl>>> 
wrote:
I was wondering how people feel about the comparison that's made here between 
Cassandra and ScyllaDB : http://www.scylladb.com/techno 
logy/ycsb-cassandra-scylla/#re sults-of-3-scylla-nodes-vs-30- 
cassandra-nodes<http://www.scylladb.com/technology/ycsb-cassandra-scylla/#results-of-3-scylla-nodes-vs-30-cassandra-nodes>

They are claiming a 10x impro

Re: scylladb

[Bhuvan Rawal]

Agreed C++ gives an added advantage to talk to underlying hardware with
better efficiency, it sound good but can a pice of code written in C++ give
1000% throughput than a Java app? Is TPC design 10X more performant than
SEDA arch?

And if C/C++ is indeed that fast how can Aerospike (which is itself written
in C) claim to be 10X faster than Scylla here
http://www.aerospike.com/benchmarks/scylladb-initial/ ? (Combining your's
and aerospike's benchmarks it appears that Aerospike is 100X performant
than C* - I highly doubt that!! )

For a moment lets forget about evaluating 2 different databases, one can
observe 10X performance difference between a mistuned cassandra cluster and
one thats tuned as per data model - there are so many Tunables in yaml as
well as table configs.

Idea is - in order to strengthen your claim, you need to provide complete
system metrics (Disk, CPU, Network), the OPS increase starts to decay along
with the configs used. Having plain ops per second and 99p latency is
blackbox.

Regards,
Bhuvan

On Fri, Mar 10, 2017 at 12:47 PM, Avi Kivity <a...@scylladb.com> wrote:

> ScyllaDB engineer here.
>
> C++ is really an enabling technology here. It is directly responsible for
> a small fraction of the gain by executing faster than Java.  But it is
> indirectly responsible for the gain by allowing us direct control over
> memory and threading.  Just as an example, Scylla starts by taking over
> almost all of the machine's memory, and dynamically assigning it to
> memtables, cache, and working memory needed to handle requests in flight.
> Memory is statically partitioned across cores, allowing us to exploit NUMA
> fully.  You can't do these things in Java.
>
> I would say the major contributors to Scylla performance are:
>  - thread-per-core design
>  - replacement of the page cache with a row cache
>  - careful attention to many small details, each contributing a little,
> but with a large overall impact
>
> While I'm here I can say that performance is not the only goal here, it is
> stable and predictable performance over varying loads and during
> maintenance operations like repair, without any special tuning.  We measure
> the amount of CPU and I/O spent on foreground (user) and background
> (maintenance) tasks and divide them fairly.  This work is not complete but
> already makes operating Scylla a lot simpler.
>
>
> On 03/10/2017 01:42 AM, Kant Kodali wrote:
>
> I dont think ScyllaDB performance is because of C++. The design decisions
> in scylladb are indeed different from Cassandra such as getting rid of SEDA
> and moving to TPC and so on.
>
> If someone thinks it is because of C++ then just show the benchmarks that
> proves it is indeed the C++ which gave 10X performance boost as ScyllaDB
> claims instead of stating it.
>
>
> On Thu, Mar 9, 2017 at 3:22 PM, Richard L. Burton III <mrbur...@gmail.com>
> wrote:
>
>> They spend an enormous amount of time focusing on performance. You can
>> expect them to continue on with their optimization and keep crushing it.
>>
>> P.S., I don't work for ScyllaDB.
>>
>> On Thu, Mar 9, 2017 at 6:02 PM, Rakesh Kumar <rakeshkumar...@outlook.com>
>> wrote:
>>
>>> In all of their presentation they keep harping on the fact that scylladb
>>> is written in C++ and does not carry the overhead of Java.  Still the
>>> difference looks staggering.
>>> 
>>> From: daemeon reiydelle <daeme...@gmail.com>
>>> Sent: Thursday, March 9, 2017 14:21
>>> To: user@cassandra.apache.org
>>> Subject: Re: scylladb
>>>
>>> The comparison is fair, and conservative. Did substantial performance
>>> comparisons for two clients, both results returned throughputs that were
>>> faster than the published comparisons (15x as I recall). At that time the
>>> client preferred to utilize a Cass COTS solution and use a caching solution
>>> for OLA compliance.
>>>
>>>
>>> ...
>>>
>>> Daemeon C.M. Reiydelle
>>> USA (+1) 415.501.0198 <%28%2B1%29%20415.501.0198>
>>> London (+44) (0) 20 8144 9872 <%28%2B44%29%20%280%29%2020%208144%209872>
>>>
>>> On Thu, Mar 9, 2017 at 11:04 AM, Robin Verlangen <ro...@us2.nl>> ro...@us2.nl>> wrote:
>>> I was wondering how people feel about the comparison that's made here
>>> between Cassandra and ScyllaDB : http://www.scylladb.com/techno
>>> logy/ycsb-cassandra-scylla/#results-of-3-scylla-nodes-vs-30-
>>> cassandra-nodes
>>>
>>> They are claiming a 10x improvement, is that a fair comparison or maybe
>>> a somewhat coloured view of a (micro)

Re: scylladb

[Avi Kivity]

ScyllaDB engineer here.

C++ is really an enabling technology here. It is directly responsible 
for a small fraction of the gain by executing faster than Java.  But it 
is indirectly responsible for the gain by allowing us direct control 
over memory and threading.  Just as an example, Scylla starts by taking 
over almost all of the machine's memory, and dynamically assigning it to 
memtables, cache, and working memory needed to handle requests in 
flight.  Memory is statically partitioned across cores, allowing us to 
exploit NUMA fully.  You can't do these things in Java.


I would say the major contributors to Scylla performance are:
 - thread-per-core design
 - replacement of the page cache with a row cache
 - careful attention to many small details, each contributing a little, 
but with a large overall impact


While I'm here I can say that performance is not the only goal here, it 
is stable and predictable performance over varying loads and during 
maintenance operations like repair, without any special tuning.  We 
measure the amount of CPU and I/O spent on foreground (user) and 
background (maintenance) tasks and divide them fairly. This work is not 
complete but already makes operating Scylla a lot simpler.


On 03/10/2017 01:42 AM, Kant Kodali wrote:
I dont think ScyllaDB performance is because of C++. The design 
decisions in scylladb are indeed different from Cassandra such as 
getting rid of SEDA and moving to TPC and so on.


If someone thinks it is because of C++ then just show the benchmarks 
that proves it is indeed the C++ which gave 10X performance boost as 
ScyllaDB claims instead of stating it.



On Thu, Mar 9, 2017 at 3:22 PM, Richard L. Burton III 
<mrbur...@gmail.com <mailto:mrbur...@gmail.com>> wrote:


They spend an enormous amount of time focusing on performance. You
can expect them to continue on with their optimization and keep
crushing it.

P.S., I don't work for ScyllaDB.

On Thu, Mar 9, 2017 at 6:02 PM, Rakesh Kumar
<rakeshkumar...@outlook.com <mailto:rakeshkumar...@outlook.com>>
wrote:

In all of their presentation they keep harping on the fact
that scylladb is written in C++ and does not carry the
overhead of Java.  Still the difference looks staggering.

From: daemeon reiydelle <daeme...@gmail.com
<mailto:daeme...@gmail.com>>
Sent: Thursday, March 9, 2017 14:21
To: user@cassandra.apache.org <mailto:user@cassandra.apache.org>
Subject: Re: scylladb

The comparison is fair, and conservative. Did substantial
performance comparisons for two clients, both results returned
throughputs that were faster than the published comparisons
(15x as I recall). At that time the client preferred to
utilize a Cass COTS solution and use a caching solution for
OLA compliance.


...

Daemeon C.M. Reiydelle
USA (+1) 415.501.0198 <tel:%28%2B1%29%20415.501.0198>
London (+44) (0) 20 8144 9872
<tel:%28%2B44%29%20%280%29%2020%208144%209872>

On Thu, Mar 9, 2017 at 11:04 AM, Robin Verlangen <ro...@us2.nl
<mailto:ro...@us2.nl><mailto:ro...@us2.nl
<mailto:ro...@us2.nl>>> wrote:
I was wondering how people feel about the comparison that's
made here between Cassandra and ScyllaDB :

http://www.scylladb.com/technology/ycsb-cassandra-scylla/#results-of-3-scylla-nodes-vs-30-cassandra-nodes

<http://www.scylladb.com/technology/ycsb-cassandra-scylla/#results-of-3-scylla-nodes-vs-30-cassandra-nodes>

They are claiming a 10x improvement, is that a fair comparison
or maybe a somewhat coloured view of a (micro)benchmark in a
specific setup? Any pros/cons known?

Best regards,

Robin Verlangen
Chief Data Architect

Disclaimer: The information contained in this message and
attachments is intended solely for the attention and use of
the named addressee and may be confidential. If you are not
the intended recipient, you are reminded that the information
remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If
you have received this message in error, please contact the
sender immediately and irrevocably delete this message and any
copies.

On Wed, Dec 16, 2015 at 11:52 AM, Carlos Rolo
<r...@pythian.com
<mailto:r...@pythian.com><mailto:r...@pythian.com
<mailto:r...@pythian.com>>> wrote:
No rain at all! But I almost had it running last weekend, but
stopped short of installing it. Let's see if this one is for real!

Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Re: scylladb

[Bhuvan Rawal]

I'd say the benchmark would be complete only when at the point of inflexion
the necessary system benchmarks are provided.

Looking at scylladb report it is unclear as to what system parameter was
being the bottleneck. Also an observation - its mentioned in the report
that they are using 1KB ro and probably using default compression settings
so this could be a possible bottleneck (everytime 64K object would be
picked off and decompressed even though record is 1/64th the size):
https://groups.google.com/forum/#!topic/nosql-databases/9pett319cgs
This would really cripple performance if its the case.

Tuning 99%ile would be tricky in case of java because of background GC
happening - that really has to do with how GC parameters are tuned for the
specific workload.

I believe its pertinent to evaluate cassandra defaults - 100 MB per core
new heap which is recommended the Compression size which can cause troubles.

On Fri, Mar 10, 2017 at 5:12 AM, Kant Kodali <k...@peernova.com> wrote:

> I dont think ScyllaDB performance is because of C++. The design decisions
> in scylladb are indeed different from Cassandra such as getting rid of SEDA
> and moving to TPC and so on.
>
> If someone thinks it is because of C++ then just show the benchmarks that
> proves it is indeed the C++ which gave 10X performance boost as ScyllaDB
> claims instead of stating it.
>
>
> On Thu, Mar 9, 2017 at 3:22 PM, Richard L. Burton III <mrbur...@gmail.com>
> wrote:
>
>> They spend an enormous amount of time focusing on performance. You can
>> expect them to continue on with their optimization and keep crushing it.
>>
>> P.S., I don't work for ScyllaDB.
>>
>> On Thu, Mar 9, 2017 at 6:02 PM, Rakesh Kumar <rakeshkumar...@outlook.com>
>> wrote:
>>
>>> In all of their presentation they keep harping on the fact that scylladb
>>> is written in C++ and does not carry the overhead of Java.  Still the
>>> difference looks staggering.
>>> 
>>> From: daemeon reiydelle <daeme...@gmail.com>
>>> Sent: Thursday, March 9, 2017 14:21
>>> To: user@cassandra.apache.org
>>> Subject: Re: scylladb
>>>
>>> The comparison is fair, and conservative. Did substantial performance
>>> comparisons for two clients, both results returned throughputs that were
>>> faster than the published comparisons (15x as I recall). At that time the
>>> client preferred to utilize a Cass COTS solution and use a caching solution
>>> for OLA compliance.
>>>
>>>
>>> ...
>>>
>>> Daemeon C.M. Reiydelle
>>> USA (+1) 415.501.0198
>>> London (+44) (0) 20 8144 9872
>>>
>>> On Thu, Mar 9, 2017 at 11:04 AM, Robin Verlangen <ro...@us2.nl>> ro...@us2.nl>> wrote:
>>> I was wondering how people feel about the comparison that's made here
>>> between Cassandra and ScyllaDB : http://www.scylladb.com/techno
>>> logy/ycsb-cassandra-scylla/#results-of-3-scylla-nodes-vs-30-
>>> cassandra-nodes
>>>
>>> They are claiming a 10x improvement, is that a fair comparison or maybe
>>> a somewhat coloured view of a (micro)benchmark in a specific setup? Any
>>> pros/cons known?
>>>
>>> Best regards,
>>>
>>> Robin Verlangen
>>> Chief Data Architect
>>>
>>> Disclaimer: The information contained in this message and attachments is
>>> intended solely for the attention and use of the named addressee and may be
>>> confidential. If you are not the intended recipient, you are reminded that
>>> the information remains the property of the sender. You must not use,
>>> disclose, distribute, copy, print or rely on this e-mail. If you have
>>> received this message in error, please contact the sender immediately and
>>> irrevocably delete this message and any copies.
>>>
>>> On Wed, Dec 16, 2015 at 11:52 AM, Carlos Rolo <r...@pythian.com>> r...@pythian.com>> wrote:
>>> No rain at all! But I almost had it running last weekend, but stopped
>>> short of installing it. Let's see if this one is for real!
>>>
>>> Regards,
>>>
>>> Carlos Juzarte Rolo
>>> Cassandra Consultant
>>>
>>> Pythian - Love your data
>>>
>>> rolo@pythian | Twitter: @cjrolo | Linkedin:
>>> linkedin.com/in/carlosjuzarterolo<http://linkedin.com/in/car
>>> losjuzarterolo>
>>> Mobile: +351 91 891 81 00<tel:+351%20918%20918%20100> | Tel: +1 613 565
>>> 8696 x1649<tel:+1%20613-565-8696>
>>> www.pythian.com<http://ww

Re: scylladb

[Kant Kodali]

I dont think ScyllaDB performance is because of C++. The design decisions
in scylladb are indeed different from Cassandra such as getting rid of SEDA
and moving to TPC and so on.

If someone thinks it is because of C++ then just show the benchmarks that
proves it is indeed the C++ which gave 10X performance boost as ScyllaDB
claims instead of stating it.


On Thu, Mar 9, 2017 at 3:22 PM, Richard L. Burton III <mrbur...@gmail.com>
wrote:

> They spend an enormous amount of time focusing on performance. You can
> expect them to continue on with their optimization and keep crushing it.
>
> P.S., I don't work for ScyllaDB.
>
> On Thu, Mar 9, 2017 at 6:02 PM, Rakesh Kumar <rakeshkumar...@outlook.com>
> wrote:
>
>> In all of their presentation they keep harping on the fact that scylladb
>> is written in C++ and does not carry the overhead of Java.  Still the
>> difference looks staggering.
>> 
>> From: daemeon reiydelle <daeme...@gmail.com>
>> Sent: Thursday, March 9, 2017 14:21
>> To: user@cassandra.apache.org
>> Subject: Re: scylladb
>>
>> The comparison is fair, and conservative. Did substantial performance
>> comparisons for two clients, both results returned throughputs that were
>> faster than the published comparisons (15x as I recall). At that time the
>> client preferred to utilize a Cass COTS solution and use a caching solution
>> for OLA compliance.
>>
>>
>> ...
>>
>> Daemeon C.M. Reiydelle
>> USA (+1) 415.501.0198
>> London (+44) (0) 20 8144 9872
>>
>> On Thu, Mar 9, 2017 at 11:04 AM, Robin Verlangen <ro...@us2.nl> ro...@us2.nl>> wrote:
>> I was wondering how people feel about the comparison that's made here
>> between Cassandra and ScyllaDB : http://www.scylladb.com/techno
>> logy/ycsb-cassandra-scylla/#results-of-3-scylla-nodes-vs-
>> 30-cassandra-nodes
>>
>> They are claiming a 10x improvement, is that a fair comparison or maybe a
>> somewhat coloured view of a (micro)benchmark in a specific setup? Any
>> pros/cons known?
>>
>> Best regards,
>>
>> Robin Verlangen
>> Chief Data Architect
>>
>> Disclaimer: The information contained in this message and attachments is
>> intended solely for the attention and use of the named addressee and may be
>> confidential. If you are not the intended recipient, you are reminded that
>> the information remains the property of the sender. You must not use,
>> disclose, distribute, copy, print or rely on this e-mail. If you have
>> received this message in error, please contact the sender immediately and
>> irrevocably delete this message and any copies.
>>
>> On Wed, Dec 16, 2015 at 11:52 AM, Carlos Rolo <r...@pythian.com> r...@pythian.com>> wrote:
>> No rain at all! But I almost had it running last weekend, but stopped
>> short of installing it. Let's see if this one is for real!
>>
>> Regards,
>>
>> Carlos Juzarte Rolo
>> Cassandra Consultant
>>
>> Pythian - Love your data
>>
>> rolo@pythian | Twitter: @cjrolo | Linkedin:
>> linkedin.com/in/carlosjuzarterolo<http://linkedin.com/in/car
>> losjuzarterolo>
>> Mobile: +351 91 891 81 00<tel:+351%20918%20918%20100> | Tel: +1 613 565
>> 8696 x1649<tel:+1%20613-565-8696>
>> www.pythian.com<http://www.pythian.com/>
>>
>> On Wed, Dec 16, 2015 at 12:38 AM, Dani Traphagen <
>> dani.trapha...@datastax.com<mailto:dani.trapha...@datastax.com>> wrote:
>> You'll be the first Carlos.
>>
>> [Inline image 1]
>>
>> Had any rain lately? Curious how this went, if so.
>>
>> On Thu, Nov 12, 2015 at 4:36 AM, Jack Krupansky <jack.krupan...@gmail.com
>> <mailto:jack.krupan...@gmail.com>> wrote:
>> I just did a Twitter search on scylladb and did not see any tweets about
>> actual use, so far.
>>
>>
>> -- Jack Krupansky
>>
>> On Wed, Nov 11, 2015 at 10:54 AM, Carlos Alonso <i...@mrcalonso.com
>> <mailto:i...@mrcalonso.com>> wrote:
>> Any update about this?
>>
>> @Carlos Rolo, did you tried it? Thoughts?
>>
>> Carlos Alonso | Software Engineer | @calonso<https://twitter.com/calonso>
>>
>> On 5 November 2015 at 14:07, Carlos Rolo <r...@pythian.com<mailto:rolo@
>> pythian.com>> wrote:
>> Something to do on a expected rainy weekend. Thanks for the information.
>>
>> Regards,
>>
>> Carlos Juzarte Rolo
>> Cassandra Consultant
>>
>> Pythian - Love your data
>>
>> rolo@pythian | T

Re: scylladb

[Richard L. Burton III]

They spend an enormous amount of time focusing on performance. You can
expect them to continue on with their optimization and keep crushing it.

P.S., I don't work for ScyllaDB.

On Thu, Mar 9, 2017 at 6:02 PM, Rakesh Kumar <rakeshkumar...@outlook.com>
wrote:

> In all of their presentation they keep harping on the fact that scylladb
> is written in C++ and does not carry the overhead of Java.  Still the
> difference looks staggering.
> 
> From: daemeon reiydelle <daeme...@gmail.com>
> Sent: Thursday, March 9, 2017 14:21
> To: user@cassandra.apache.org
> Subject: Re: scylladb
>
> The comparison is fair, and conservative. Did substantial performance
> comparisons for two clients, both results returned throughputs that were
> faster than the published comparisons (15x as I recall). At that time the
> client preferred to utilize a Cass COTS solution and use a caching solution
> for OLA compliance.
>
>
> ...
>
> Daemeon C.M. Reiydelle
> USA (+1) 415.501.0198
> London (+44) (0) 20 8144 9872
>
> On Thu, Mar 9, 2017 at 11:04 AM, Robin Verlangen <ro...@us2.nl ro...@us2.nl>> wrote:
> I was wondering how people feel about the comparison that's made here
> between Cassandra and ScyllaDB : http://www.scylladb.com/
> technology/ycsb-cassandra-scylla/#results-of-3-scylla-
> nodes-vs-30-cassandra-nodes
>
> They are claiming a 10x improvement, is that a fair comparison or maybe a
> somewhat coloured view of a (micro)benchmark in a specific setup? Any
> pros/cons known?
>
> Best regards,
>
> Robin Verlangen
> Chief Data Architect
>
> Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
>
> On Wed, Dec 16, 2015 at 11:52 AM, Carlos Rolo <r...@pythian.com r...@pythian.com>> wrote:
> No rain at all! But I almost had it running last weekend, but stopped
> short of installing it. Let's see if this one is for real!
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: @cjrolo | Linkedin: linkedin.com/in/
> carlosjuzarterolo<http://linkedin.com/in/carlosjuzarterolo>
> Mobile: +351 91 891 81 00<tel:+351%20918%20918%20100> | Tel: +1 613 565
> 8696 x1649<tel:+1%20613-565-8696>
> www.pythian.com<http://www.pythian.com/>
>
> On Wed, Dec 16, 2015 at 12:38 AM, Dani Traphagen <
> dani.trapha...@datastax.com<mailto:dani.trapha...@datastax.com>> wrote:
> You'll be the first Carlos.
>
> [Inline image 1]
>
> Had any rain lately? Curious how this went, if so.
>
> On Thu, Nov 12, 2015 at 4:36 AM, Jack Krupansky <jack.krupan...@gmail.com<
> mailto:jack.krupan...@gmail.com>> wrote:
> I just did a Twitter search on scylladb and did not see any tweets about
> actual use, so far.
>
>
> -- Jack Krupansky
>
> On Wed, Nov 11, 2015 at 10:54 AM, Carlos Alonso <i...@mrcalonso.com
> <mailto:i...@mrcalonso.com>> wrote:
> Any update about this?
>
> @Carlos Rolo, did you tried it? Thoughts?
>
> Carlos Alonso | Software Engineer | @calonso<https://twitter.com/calonso>
>
> On 5 November 2015 at 14:07, Carlos Rolo <r...@pythian.com<mailto:rolo@
> pythian.com>> wrote:
> Something to do on a expected rainy weekend. Thanks for the information.
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: @cjrolo | Linkedin: linkedin.com/in/
> carlosjuzarterolo<http://linkedin.com/in/carlosjuzarterolo>
> Mobile: +351 91 891 81 00<tel:%2B351%2091%20891%2081%2000> | Tel: +1 613
> 565 8696 x1649<tel:%2B1%20613%20565%208696%20x1649>
> www.pythian.com<http://www.pythian.com/>
>
> On Thu, Nov 5, 2015 at 12:07 PM, Dani Traphagen <
> dani.trapha...@datastax.com<mailto:dani.trapha...@datastax.com>> wrote:
> As of two days ago, they say they've got it @cjrolo.
>
> https://github.com/scylladb/scylla/wiki/RELEASE-Scylla-0.11-Beta
>
>
> On Thursday, November 5, 2015, Carlos Rolo <r...@pythian.com<mailto:rolo@
> pythian.com>> wrote:
> I will not try until multi-DC is implemented. More than an month has
> passed since I looked for it, so it could possibly be in place, if so I may
> take

Re: scylladb

[Rakesh Kumar]

In all of their presentation they keep harping on the fact that scylladb is 
written in C++ and does not carry the overhead of Java.  Still the difference 
looks staggering.

From: daemeon reiydelle <daeme...@gmail.com>
Sent: Thursday, March 9, 2017 14:21
To: user@cassandra.apache.org
Subject: Re: scylladb

The comparison is fair, and conservative. Did substantial performance 
comparisons for two clients, both results returned throughputs that were faster 
than the published comparisons (15x as I recall). At that time the client 
preferred to utilize a Cass COTS solution and use a caching solution for OLA 
compliance.


...

Daemeon C.M. Reiydelle
USA (+1) 415.501.0198
London (+44) (0) 20 8144 9872

On Thu, Mar 9, 2017 at 11:04 AM, Robin Verlangen 
<ro...@us2.nl<mailto:ro...@us2.nl>> wrote:
I was wondering how people feel about the comparison that's made here between 
Cassandra and ScyllaDB : 
http://www.scylladb.com/technology/ycsb-cassandra-scylla/#results-of-3-scylla-nodes-vs-30-cassandra-nodes

They are claiming a 10x improvement, is that a fair comparison or maybe a 
somewhat coloured view of a (micro)benchmark in a specific setup? Any pros/cons 
known?

Best regards,

Robin Verlangen
Chief Data Architect

Disclaimer: The information contained in this message and attachments is 
intended solely for the attention and use of the named addressee and may be 
confidential. If you are not the intended recipient, you are reminded that the 
information remains the property of the sender. You must not use, disclose, 
distribute, copy, print or rely on this e-mail. If you have received this 
message in error, please contact the sender immediately and irrevocably delete 
this message and any copies.

On Wed, Dec 16, 2015 at 11:52 AM, Carlos Rolo 
<r...@pythian.com<mailto:r...@pythian.com>> wrote:
No rain at all! But I almost had it running last weekend, but stopped short of 
installing it. Let's see if this one is for real!

Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Linkedin: 
linkedin.com/in/carlosjuzarterolo<http://linkedin.com/in/carlosjuzarterolo>
Mobile: +351 91 891 81 00<tel:+351%20918%20918%20100> | Tel: +1 613 565 8696 
x1649<tel:+1%20613-565-8696>
www.pythian.com<http://www.pythian.com/>

On Wed, Dec 16, 2015 at 12:38 AM, Dani Traphagen 
<dani.trapha...@datastax.com<mailto:dani.trapha...@datastax.com>> wrote:
You'll be the first Carlos.

[Inline image 1]

Had any rain lately? Curious how this went, if so.

On Thu, Nov 12, 2015 at 4:36 AM, Jack Krupansky 
<jack.krupan...@gmail.com<mailto:jack.krupan...@gmail.com>> wrote:
I just did a Twitter search on scylladb and did not see any tweets about actual 
use, so far.


-- Jack Krupansky

On Wed, Nov 11, 2015 at 10:54 AM, Carlos Alonso 
<i...@mrcalonso.com<mailto:i...@mrcalonso.com>> wrote:
Any update about this?

@Carlos Rolo, did you tried it? Thoughts?

Carlos Alonso | Software Engineer | @calonso<https://twitter.com/calonso>

On 5 November 2015 at 14:07, Carlos Rolo 
<r...@pythian.com<mailto:r...@pythian.com>> wrote:
Something to do on a expected rainy weekend. Thanks for the information.

Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Linkedin: 
linkedin.com/in/carlosjuzarterolo<http://linkedin.com/in/carlosjuzarterolo>
Mobile: +351 91 891 81 00<tel:%2B351%2091%20891%2081%2000> | Tel: +1 613 565 
8696 x1649<tel:%2B1%20613%20565%208696%20x1649>
www.pythian.com<http://www.pythian.com/>

On Thu, Nov 5, 2015 at 12:07 PM, Dani Traphagen 
<dani.trapha...@datastax.com<mailto:dani.trapha...@datastax.com>> wrote:
As of two days ago, they say they've got it @cjrolo.

https://github.com/scylladb/scylla/wiki/RELEASE-Scylla-0.11-Beta


On Thursday, November 5, 2015, Carlos Rolo 
<r...@pythian.com<mailto:r...@pythian.com>> wrote:
I will not try until multi-DC is implemented. More than an month has passed 
since I looked for it, so it could possibly be in place, if so I may take some 
time to test it.

Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Linkedin: 
linkedin.com/in/carlosjuzarterolo<http://linkedin.com/in/carlosjuzarterolo>
Mobile: +351 91 891 81 00<tel:%2B351%2091%20891%2081%2000> | Tel: +1 613 565 
8696 x1649<tel:%2B1%20613%20565%208696%20x1649>
www.pythian.com<http://www.pythian.com/>

On Thu, Nov 5, 2015 at 9:37 AM, Jon Haddad <jonathan.had...@gmail.com> wrote:
Nope, no one I know.  Let me know if you try it I'd love to hear your feedback.

> On Nov 5, 2015, at 9:22 AM, tommaso barbugli <tbarbu...@gmail.com> wrote:
>
> Hi guys,
>
> did anyone already try Scylladb (yet another fastest NoSQL database in tow

Re: scylladb

[Jack Krupansky]

I just did a Twitter search on scylladb and did not see any tweets about
actual use, so far.


-- Jack Krupansky

On Wed, Nov 11, 2015 at 10:54 AM, Carlos Alonso <i...@mrcalonso.com> wrote:

> Any update about this?
>
> @Carlos Rolo, did you tried it? Thoughts?
>
> Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso>
>
> On 5 November 2015 at 14:07, Carlos Rolo <r...@pythian.com> wrote:
>
>> Something to do on a expected rainy weekend. Thanks for the information.
>>
>> Regards,
>>
>> Carlos Juzarte Rolo
>> Cassandra Consultant
>>
>> Pythian - Love your data
>>
>> rolo@pythian | Twitter: @cjrolo | Linkedin: 
>> *linkedin.com/in/carlosjuzarterolo
>> <http://linkedin.com/in/carlosjuzarterolo>*
>> Mobile: +351 91 891 81 00 | Tel: +1 613 565 8696 x1649
>> www.pythian.com
>>
>> On Thu, Nov 5, 2015 at 12:07 PM, Dani Traphagen <
>> dani.trapha...@datastax.com> wrote:
>>
>>> As of two days ago, they say they've got it @cjrolo.
>>>
>>> https://github.com/scylladb/scylla/wiki/RELEASE-Scylla-0.11-Beta
>>>
>>>
>>> On Thursday, November 5, 2015, Carlos Rolo <r...@pythian.com> wrote:
>>>
>>>> I will not try until multi-DC is implemented. More than an month has
>>>> passed since I looked for it, so it could possibly be in place, if so I may
>>>> take some time to test it.
>>>>
>>>> Regards,
>>>>
>>>> Carlos Juzarte Rolo
>>>> Cassandra Consultant
>>>>
>>>> Pythian - Love your data
>>>>
>>>> rolo@pythian | Twitter: @cjrolo | Linkedin: 
>>>> *linkedin.com/in/carlosjuzarterolo
>>>> <http://linkedin.com/in/carlosjuzarterolo>*
>>>> Mobile: +351 91 891 81 00 | Tel: +1 613 565 8696 x1649
>>>> www.pythian.com
>>>>
>>>> On Thu, Nov 5, 2015 at 9:37 AM, Jon Haddad <jonathan.had...@gmail.com>
>>>> wrote:
>>>>
>>>>> Nope, no one I know.  Let me know if you try it I'd love to hear your
>>>>> feedback.
>>>>>
>>>>> > On Nov 5, 2015, at 9:22 AM, tommaso barbugli <tbarbu...@gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > Hi guys,
>>>>> >
>>>>> > did anyone already try Scylladb (yet another fastest NoSQL database
>>>>> in town) and has some thoughts/hands-on experience to share?
>>>>> >
>>>>> > Cheers,
>>>>> > Tommaso
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> Sent from mobile -- apologizes for brevity or errors.
>>>
>>
>>
>> --
>>
>>
>>
>>
>

Re: scylladb

[Dani Traphagen]

Killer, @cjrolo. Will you update via this thread?

On Wed, Nov 11, 2015 at 7:57 AM, Carlos Rolo  wrote:

> Not yet, but not far from doing it. No rain here yet! :)
>
> On a more serious tone, should be done before end of the Month.
>
> --
>
>
>
>


-- 
[image: datastax_logo.png] 

DANI TRAPHAGEN

Technical Enablement Lead | dani.trapha...@datastax.com

[image: twitter.png]  [image: linkedin.png]

Re: scylladb

[Carlos Rolo]

Sure! I have a lot of blog post on backlog to blog asap about this,
otherwise I would only share results mid 2016 :P

Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
*
Mobile: +351 91 891 81 00 | Tel: +1 613 565 8696 x1649
www.pythian.com

On Wed, Nov 11, 2015 at 4:46 PM, Dani Traphagen  wrote:

> Killer, @cjrolo. Will you update via this thread?
>
> On Wed, Nov 11, 2015 at 7:57 AM, Carlos Rolo  wrote:
>
>> Not yet, but not far from doing it. No rain here yet! :)
>>
>> On a more serious tone, should be done before end of the Month.
>>
>> --
>>
>>
>>
>>
>
>
> --
> [image: datastax_logo.png] 
>
> DANI TRAPHAGEN
>
> Technical Enablement Lead | dani.trapha...@datastax.com
>
> [image: twitter.png]  [image:
> linkedin.png] 
> 
>
>
>

-- 


--

Re: scylladb

[Carlos Rolo]

Not yet, but not far from doing it. No rain here yet! :)

On a more serious tone, should be done before end of the Month.

-- 


--

Re: scylladb

[Carlos Alonso]

Any update about this?

@Carlos Rolo, did you tried it? Thoughts?

Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso>

On 5 November 2015 at 14:07, Carlos Rolo <r...@pythian.com> wrote:

> Something to do on a expected rainy weekend. Thanks for the information.
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: @cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
> <http://linkedin.com/in/carlosjuzarterolo>*
> Mobile: +351 91 891 81 00 | Tel: +1 613 565 8696 x1649
> www.pythian.com
>
> On Thu, Nov 5, 2015 at 12:07 PM, Dani Traphagen <
> dani.trapha...@datastax.com> wrote:
>
>> As of two days ago, they say they've got it @cjrolo.
>>
>> https://github.com/scylladb/scylla/wiki/RELEASE-Scylla-0.11-Beta
>>
>>
>> On Thursday, November 5, 2015, Carlos Rolo <r...@pythian.com> wrote:
>>
>>> I will not try until multi-DC is implemented. More than an month has
>>> passed since I looked for it, so it could possibly be in place, if so I may
>>> take some time to test it.
>>>
>>> Regards,
>>>
>>> Carlos Juzarte Rolo
>>> Cassandra Consultant
>>>
>>> Pythian - Love your data
>>>
>>> rolo@pythian | Twitter: @cjrolo | Linkedin: 
>>> *linkedin.com/in/carlosjuzarterolo
>>> <http://linkedin.com/in/carlosjuzarterolo>*
>>> Mobile: +351 91 891 81 00 | Tel: +1 613 565 8696 x1649
>>> www.pythian.com
>>>
>>> On Thu, Nov 5, 2015 at 9:37 AM, Jon Haddad <jonathan.had...@gmail.com>
>>> wrote:
>>>
>>>> Nope, no one I know.  Let me know if you try it I'd love to hear your
>>>> feedback.
>>>>
>>>> > On Nov 5, 2015, at 9:22 AM, tommaso barbugli <tbarbu...@gmail.com>
>>>> wrote:
>>>> >
>>>> > Hi guys,
>>>> >
>>>> > did anyone already try Scylladb (yet another fastest NoSQL database
>>>> in town) and has some thoughts/hands-on experience to share?
>>>> >
>>>> > Cheers,
>>>> > Tommaso
>>>>
>>>>
>>>
>>> --
>>>
>>>
>>>
>>>
>>
>> --
>> Sent from mobile -- apologizes for brevity or errors.
>>
>
>
> --
>
>
>
>

scylladb

[tommaso barbugli]

Hi guys,

did anyone already try Scylladb (yet another fastest NoSQL database in
town) and has some thoughts/hands-on experience to share?

Cheers,
Tommaso

Re: scylladb

[Jon Haddad]

Nope, no one I know.  Let me know if you try it I'd love to hear your feedback.

> On Nov 5, 2015, at 9:22 AM, tommaso barbugli <tbarbu...@gmail.com> wrote:
> 
> Hi guys,
> 
> did anyone already try Scylladb (yet another fastest NoSQL database in town) 
> and has some thoughts/hands-on experience to share?
> 
> Cheers,
> Tommaso

Re: scylladb

[Carlos Rolo]

I will not try until multi-DC is implemented. More than an month has passed
since I looked for it, so it could possibly be in place, if so I may take
some time to test it.

Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
<http://linkedin.com/in/carlosjuzarterolo>*
Mobile: +351 91 891 81 00 | Tel: +1 613 565 8696 x1649
www.pythian.com

On Thu, Nov 5, 2015 at 9:37 AM, Jon Haddad <jonathan.had...@gmail.com>
wrote:

> Nope, no one I know.  Let me know if you try it I'd love to hear your
> feedback.
>
> > On Nov 5, 2015, at 9:22 AM, tommaso barbugli <tbarbu...@gmail.com>
> wrote:
> >
> > Hi guys,
> >
> > did anyone already try Scylladb (yet another fastest NoSQL database in
> town) and has some thoughts/hands-on experience to share?
> >
> > Cheers,
> > Tommaso
>
>

-- 


--

Re: scylladb

[Dani Traphagen]

As of two days ago, they say they've got it @cjrolo.

https://github.com/scylladb/scylla/wiki/RELEASE-Scylla-0.11-Beta

On Thursday, November 5, 2015, Carlos Rolo <r...@pythian.com> wrote:

> I will not try until multi-DC is implemented. More than an month has
> passed since I looked for it, so it could possibly be in place, if so I may
> take some time to test it.
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: @cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
> <http://linkedin.com/in/carlosjuzarterolo>*
> Mobile: +351 91 891 81 00 | Tel: +1 613 565 8696 x1649
> www.pythian.com
>
> On Thu, Nov 5, 2015 at 9:37 AM, Jon Haddad <jonathan.had...@gmail.com
> <javascript:_e(%7B%7D,'cvml','jonathan.had...@gmail.com');>> wrote:
>
>> Nope, no one I know.  Let me know if you try it I'd love to hear your
>> feedback.
>>
>> > On Nov 5, 2015, at 9:22 AM, tommaso barbugli <tbarbu...@gmail.com
>> <javascript:_e(%7B%7D,'cvml','tbarbu...@gmail.com');>> wrote:
>> >
>> > Hi guys,
>> >
>> > did anyone already try Scylladb (yet another fastest NoSQL database in
>> town) and has some thoughts/hands-on experience to share?
>> >
>> > Cheers,
>> > Tommaso
>>
>>
>
> --
>
>
>
>

-- 
Sent from mobile -- apologizes for brevity or errors.

Re: scylladb

[Carlos Rolo]

Something to do on a expected rainy weekend. Thanks for the information.

Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
<http://linkedin.com/in/carlosjuzarterolo>*
Mobile: +351 91 891 81 00 | Tel: +1 613 565 8696 x1649
www.pythian.com

On Thu, Nov 5, 2015 at 12:07 PM, Dani Traphagen <dani.trapha...@datastax.com
> wrote:

> As of two days ago, they say they've got it @cjrolo.
>
> https://github.com/scylladb/scylla/wiki/RELEASE-Scylla-0.11-Beta
>
>
> On Thursday, November 5, 2015, Carlos Rolo <r...@pythian.com> wrote:
>
>> I will not try until multi-DC is implemented. More than an month has
>> passed since I looked for it, so it could possibly be in place, if so I may
>> take some time to test it.
>>
>> Regards,
>>
>> Carlos Juzarte Rolo
>> Cassandra Consultant
>>
>> Pythian - Love your data
>>
>> rolo@pythian | Twitter: @cjrolo | Linkedin: 
>> *linkedin.com/in/carlosjuzarterolo
>> <http://linkedin.com/in/carlosjuzarterolo>*
>> Mobile: +351 91 891 81 00 | Tel: +1 613 565 8696 x1649
>> www.pythian.com
>>
>> On Thu, Nov 5, 2015 at 9:37 AM, Jon Haddad <jonathan.had...@gmail.com>
>> wrote:
>>
>>> Nope, no one I know.  Let me know if you try it I'd love to hear your
>>> feedback.
>>>
>>> > On Nov 5, 2015, at 9:22 AM, tommaso barbugli <tbarbu...@gmail.com>
>>> wrote:
>>> >
>>> > Hi guys,
>>> >
>>> > did anyone already try Scylladb (yet another fastest NoSQL database in
>>> town) and has some thoughts/hands-on experience to share?
>>> >
>>> > Cheers,
>>> > Tommaso
>>>
>>>
>>
>> --
>>
>>
>>
>>
>
> --
> Sent from mobile -- apologizes for brevity or errors.
>

-- 


--

Re: ScyllaDB, a new open source, Cassandra-compatible NoSQL

[Marcelo Valle (BLOOMBERG/ LONDON)]

I think there is a very important point in Scylladb - latency. 
Performance can be an important requirement, but the fact scylladb is written 
in C and uses lock free algorithms inside means it should have lower latency 
than Cassandra, which enables it's use for a wider range of applications. 
It seems like a huge milestone achieved by Cassandra community, congratulations!

From: user@cassandra.apache.org 
Subject: Re: ScyllaDB, a new open source, Cassandra-compatible NoSQL

Looking at the architecture and what scylladb does, I'm not surprised they got 
10x improvement. SeaStar skips a lot of the overhead of copying stuff and it 
gives them CPU core affinity. Anyone that's listened to Clif Click talk about 
cache misses, locks and other low level stuff would recognize the huge boost in 
performance when many of those bottlenecks are removed. Using an actor model to 
avoid locks doesn't hurt either.

On Tue, Sep 22, 2015 at 5:20 PM, Minh Do <m...@netflix.com> wrote:

First glance at their github, it looks like they re-implemented Cassandra in 
C++.  90% components in Cassandra are
in scylladb, i.e. compaction, repair, CQL, gossip, SStable.


With C++, I believe this helps performance to some extent up to a point when 
compaction has not run yet.  
Then, it will be disk IO to be the dominant factor in the performance 
measurement as the more traffics to a node the more degrading
the performance is across the cluster.

Also, they only support Thrift protocol so it won't work with Java Driver with 
the new asynchronous protocol.  I doubt their tests 
are truly a fair one.

On Tue, Sep 22, 2015 at 2:13 PM, Venkatesh Arivazhagan <venkey.a...@gmail.com> 
wrote:

I came across this article: 
zdnet.com/article/kvm-creators-open-source-fast-cassandra-drop-in-replacement-scylla/

Tzach, I would love to know/understand moree about ScyllaDB too. Also the 
benchmark seems to have only 1 DB Server. Do you have benchmark numbers where 
more than 1 DB servers were involved? :)


On Tue, Sep 22, 2015 at 1:40 PM, Sachin Nikam <skni...@gmail.com> wrote:

Tzach,
Can you point to any documentation on scylladb site which talks about how/why 
scylla db performs better than Cassandra while using the same architecture?
Regards
Sachin

On Tue, Sep 22, 2015 at 9:18 AM, Tzach Livyatan <tz...@cloudius-systems.com> 
wrote:

Hello Cassandra users,

We are pleased to announce a new member of the Cassandra Ecosystem - ScyllaDB
ScyllaDB is a new, open source, Cassandra-compatible NoSQL data store, written 
with the goal of delivering superior performance and consistent low latency.  
Today, ScyllaDB runs 1M tps per server with sub 1ms latency.

ScyllaDB  supports CQL, is compatible with Cassandra drivers, and works out of 
the box with Cassandra tools like cqlsh, Spark connector, nodetool and 
cassandra-stress. ScyllaDB is a drop-in replacement solution for the Cassandra 
server side packages.

Scylla is implemented using the new shared-nothing Seastar framework for 
extreme performance on modern multicore hardware, and the Data Plane 
Development Kit (DPDK) for high-speed low-latency networking.

Try Scylla Now - http://www.scylladb.com

We will be at Cassandra summit 2015, you are welcome to visit our booth to hear 
more and see a demo.
Avi Kivity, our CTO, will host a session on Scylla on Thursday, 1:50 PM - 2:30 
PM in rooms M1 - M3.

Regards
Tzach
scylladb


<< ideas dont deserve respect >>

Re: ScyllaDB, a new open source, Cassandra-compatible NoSQL

[Peter Lin]

Looking at the architecture and what scylladb does, I'm not surprised they
got 10x improvement. SeaStar skips a lot of the overhead of copying stuff
and it gives them CPU core affinity. Anyone that's listened to Clif Click
talk about cache misses, locks and other low level stuff would recognize
the huge boost in performance when many of those bottlenecks are removed.
Using an actor model to avoid locks doesn't hurt either.

On Tue, Sep 22, 2015 at 5:20 PM, Minh Do <m...@netflix.com> wrote:

> First glance at their github, it looks like they re-implemented Cassandra
> in C++.  90% components in Cassandra are
> in scylladb, i.e. compaction, repair, CQL, gossip, SStable.
>
>
> With C++, I believe this helps performance to some extent up to a point
> when compaction has not run yet.
> Then, it will be disk IO to be the dominant factor in the performance
> measurement as the more traffics to a node the more degrading
> the performance is across the cluster.
>
> Also, they only support Thrift protocol so it won't work with Java Driver
> with the new asynchronous protocol.  I doubt their tests
> are truly a fair one.
>
> On Tue, Sep 22, 2015 at 2:13 PM, Venkatesh Arivazhagan <
> venkey.a...@gmail.com> wrote:
>
>> I came across this article:
>> zdnet.com/article/kvm-creators-open-source-fast-cassandra-drop-in-replacement-scylla/
>>
>> Tzach, I would love to know/understand moree about ScyllaDB too. Also the
>> benchmark seems to have only 1 DB Server. Do you have benchmark numbers
>> where more than 1 DB servers were involved? :)
>>
>>
>> On Tue, Sep 22, 2015 at 1:40 PM, Sachin Nikam <skni...@gmail.com> wrote:
>>
>>> Tzach,
>>> Can you point to any documentation on scylladb site which talks about
>>> how/why scylla db performs better than Cassandra while using the same
>>> architecture?
>>> Regards
>>> Sachin
>>>
>>> On Tue, Sep 22, 2015 at 9:18 AM, Tzach Livyatan <
>>> tz...@cloudius-systems.com> wrote:
>>>
>>>> Hello Cassandra users,
>>>>
>>>> We are pleased to announce a new member of the Cassandra Ecosystem -
>>>> ScyllaDB
>>>> ScyllaDB is a new, open source, Cassandra-compatible NoSQL data store,
>>>> written with the goal of delivering superior performance and consistent low
>>>> latency.  Today, ScyllaDB runs 1M tps per server with sub 1ms latency.
>>>>
>>>> ScyllaDB  supports CQL, is compatible with Cassandra drivers, and works
>>>> out of the box with Cassandra tools like cqlsh, Spark connector, nodetool
>>>> and cassandra-stress. ScyllaDB is a drop-in replacement solution for the
>>>> Cassandra server side packages.
>>>>
>>>> Scylla is implemented using the new shared-nothing Seastar
>>>> <http://www.seastar-project.org/> framework for extreme performance on
>>>> modern multicore hardware, and the Data Plane Development Kit (DPDK) for
>>>> high-speed low-latency networking.
>>>>
>>>> Try Scylla Now - http://www.scylladb.com
>>>>
>>>> We will be at Cassandra summit 2015, you are welcome to visit our booth
>>>> to hear more and see a demo.
>>>> Avi Kivity, our CTO, will host a session on Scylla on Thursday, 1:50 PM
>>>> - 2:30 PM in rooms M1 - M3.
>>>>
>>>> Regards
>>>> Tzach
>>>> scylladb
>>>>
>>>>
>>>
>>
>

Re: ScyllaDB, a new open source, Cassandra-compatible NoSQL

[Sachin Nikam]

Tzach,
Can you point to any documentation on scylladb site which talks about
how/why scylla db performs better than Cassandra while using the same
architecture?
Regards
Sachin

On Tue, Sep 22, 2015 at 9:18 AM, Tzach Livyatan <tz...@cloudius-systems.com>
wrote:

> Hello Cassandra users,
>
> We are pleased to announce a new member of the Cassandra Ecosystem -
> ScyllaDB
> ScyllaDB is a new, open source, Cassandra-compatible NoSQL data store,
> written with the goal of delivering superior performance and consistent low
> latency.  Today, ScyllaDB runs 1M tps per server with sub 1ms latency.
>
> ScyllaDB  supports CQL, is compatible with Cassandra drivers, and works
> out of the box with Cassandra tools like cqlsh, Spark connector, nodetool
> and cassandra-stress. ScyllaDB is a drop-in replacement solution for the
> Cassandra server side packages.
>
> Scylla is implemented using the new shared-nothing Seastar
> <http://www.seastar-project.org/> framework for extreme performance on
> modern multicore hardware, and the Data Plane Development Kit (DPDK) for
> high-speed low-latency networking.
>
> Try Scylla Now - http://www.scylladb.com
>
> We will be at Cassandra summit 2015, you are welcome to visit our booth to
> hear more and see a demo.
> Avi Kivity, our CTO, will host a session on Scylla on Thursday, 1:50 PM -
> 2:30 PM in rooms M1 - M3.
>
> Regards
> Tzach
> scylladb
>
>

Re: ScyllaDB, a new open source, Cassandra-compatible NoSQL

[Minh Do]

First glance at their github, it looks like they re-implemented Cassandra
in C++.  90% components in Cassandra are
in scylladb, i.e. compaction, repair, CQL, gossip, SStable.


With C++, I believe this helps performance to some extent up to a point
when compaction has not run yet.
Then, it will be disk IO to be the dominant factor in the performance
measurement as the more traffics to a node the more degrading
the performance is across the cluster.

Also, they only support Thrift protocol so it won't work with Java Driver
with the new asynchronous protocol.  I doubt their tests
are truly a fair one.

On Tue, Sep 22, 2015 at 2:13 PM, Venkatesh Arivazhagan <
venkey.a...@gmail.com> wrote:

> I came across this article:
> zdnet.com/article/kvm-creators-open-source-fast-cassandra-drop-in-replacement-scylla/
>
> Tzach, I would love to know/understand moree about ScyllaDB too. Also the
> benchmark seems to have only 1 DB Server. Do you have benchmark numbers
> where more than 1 DB servers were involved? :)
>
>
> On Tue, Sep 22, 2015 at 1:40 PM, Sachin Nikam <skni...@gmail.com> wrote:
>
>> Tzach,
>> Can you point to any documentation on scylladb site which talks about
>> how/why scylla db performs better than Cassandra while using the same
>> architecture?
>> Regards
>> Sachin
>>
>> On Tue, Sep 22, 2015 at 9:18 AM, Tzach Livyatan <
>> tz...@cloudius-systems.com> wrote:
>>
>>> Hello Cassandra users,
>>>
>>> We are pleased to announce a new member of the Cassandra Ecosystem -
>>> ScyllaDB
>>> ScyllaDB is a new, open source, Cassandra-compatible NoSQL data store,
>>> written with the goal of delivering superior performance and consistent low
>>> latency.  Today, ScyllaDB runs 1M tps per server with sub 1ms latency.
>>>
>>> ScyllaDB  supports CQL, is compatible with Cassandra drivers, and works
>>> out of the box with Cassandra tools like cqlsh, Spark connector, nodetool
>>> and cassandra-stress. ScyllaDB is a drop-in replacement solution for the
>>> Cassandra server side packages.
>>>
>>> Scylla is implemented using the new shared-nothing Seastar
>>> <http://www.seastar-project.org/> framework for extreme performance on
>>> modern multicore hardware, and the Data Plane Development Kit (DPDK) for
>>> high-speed low-latency networking.
>>>
>>> Try Scylla Now - http://www.scylladb.com
>>>
>>> We will be at Cassandra summit 2015, you are welcome to visit our booth
>>> to hear more and see a demo.
>>> Avi Kivity, our CTO, will host a session on Scylla on Thursday, 1:50 PM
>>> - 2:30 PM in rooms M1 - M3.
>>>
>>> Regards
>>> Tzach
>>> scylladb
>>>
>>>
>>
>

Re: ScyllaDB, a new open source, Cassandra-compatible NoSQL

[Venkatesh Arivazhagan]

I came across this article:
zdnet.com/article/kvm-creators-open-source-fast-cassandra-drop-in-replacement-scylla/

Tzach, I would love to know/understand moree about ScyllaDB too. Also the
benchmark seems to have only 1 DB Server. Do you have benchmark numbers
where more than 1 DB servers were involved? :)


On Tue, Sep 22, 2015 at 1:40 PM, Sachin Nikam <skni...@gmail.com> wrote:

> Tzach,
> Can you point to any documentation on scylladb site which talks about
> how/why scylla db performs better than Cassandra while using the same
> architecture?
> Regards
> Sachin
>
> On Tue, Sep 22, 2015 at 9:18 AM, Tzach Livyatan <
> tz...@cloudius-systems.com> wrote:
>
>> Hello Cassandra users,
>>
>> We are pleased to announce a new member of the Cassandra Ecosystem -
>> ScyllaDB
>> ScyllaDB is a new, open source, Cassandra-compatible NoSQL data store,
>> written with the goal of delivering superior performance and consistent low
>> latency.  Today, ScyllaDB runs 1M tps per server with sub 1ms latency.
>>
>> ScyllaDB  supports CQL, is compatible with Cassandra drivers, and works
>> out of the box with Cassandra tools like cqlsh, Spark connector, nodetool
>> and cassandra-stress. ScyllaDB is a drop-in replacement solution for the
>> Cassandra server side packages.
>>
>> Scylla is implemented using the new shared-nothing Seastar
>> <http://www.seastar-project.org/> framework for extreme performance on
>> modern multicore hardware, and the Data Plane Development Kit (DPDK) for
>> high-speed low-latency networking.
>>
>> Try Scylla Now - http://www.scylladb.com
>>
>> We will be at Cassandra summit 2015, you are welcome to visit our booth
>> to hear more and see a demo.
>> Avi Kivity, our CTO, will host a session on Scylla on Thursday, 1:50 PM -
>> 2:30 PM in rooms M1 - M3.
>>
>> Regards
>> Tzach
>> scylladb
>>
>>
>

Re: ScyllaDB, a new open source, Cassandra-compatible NoSQL

[Tzach Livyatan]

On Wed, Sep 23, 2015 at 12:20 AM, Minh Do <m...@netflix.com> wrote:

> First glance at their github, it looks like they re-implemented Cassandra
> in C++.  90% components in Cassandra are
> in scylladb, i.e. compaction, repair, CQL, gossip, SStable.
>

True

>
>
> With C++, I believe this helps performance to some extent up to a point
> when compaction has not run yet.
> Then, it will be disk IO to be the dominant factor in the performance
> measurement as the more traffics to a node the more degrading
> the performance is across the cluster.
>
Also, they only support Thrift protocol so it won't work with Java Driver
> with the new asynchronous protocol.  I doubt their tests
> are truly a fair one.
>

Scylla currently only support CQL
For more info, I suggest to continue the discussion at the new Scylla list
https://groups.google.com/forum/#!forum/scylladb-users



>
> On Tue, Sep 22, 2015 at 2:13 PM, Venkatesh Arivazhagan <
> venkey.a...@gmail.com> wrote:
>
>> I came across this article:
>> zdnet.com/article/kvm-creators-open-source-fast-cassandra-drop-in-replacement-scylla/
>>
>> Tzach, I would love to know/understand moree about ScyllaDB too. Also the
>> benchmark seems to have only 1 DB Server. Do you have benchmark numbers
>> where more than 1 DB servers were involved? :)
>>
>>
>> On Tue, Sep 22, 2015 at 1:40 PM, Sachin Nikam <skni...@gmail.com> wrote:
>>
>>> Tzach,
>>> Can you point to any documentation on scylladb site which talks about
>>> how/why scylla db performs better than Cassandra while using the same
>>> architecture?
>>> Regards
>>> Sachin
>>>
>>> On Tue, Sep 22, 2015 at 9:18 AM, Tzach Livyatan <
>>> tz...@cloudius-systems.com> wrote:
>>>
>>>> Hello Cassandra users,
>>>>
>>>> We are pleased to announce a new member of the Cassandra Ecosystem -
>>>> ScyllaDB
>>>> ScyllaDB is a new, open source, Cassandra-compatible NoSQL data store,
>>>> written with the goal of delivering superior performance and consistent low
>>>> latency.  Today, ScyllaDB runs 1M tps per server with sub 1ms latency.
>>>>
>>>> ScyllaDB  supports CQL, is compatible with Cassandra drivers, and works
>>>> out of the box with Cassandra tools like cqlsh, Spark connector, nodetool
>>>> and cassandra-stress. ScyllaDB is a drop-in replacement solution for the
>>>> Cassandra server side packages.
>>>>
>>>> Scylla is implemented using the new shared-nothing Seastar
>>>> <http://www.seastar-project.org/> framework for extreme performance on
>>>> modern multicore hardware, and the Data Plane Development Kit (DPDK) for
>>>> high-speed low-latency networking.
>>>>
>>>> Try Scylla Now - http://www.scylladb.com
>>>>
>>>> We will be at Cassandra summit 2015, you are welcome to visit our booth
>>>> to hear more and see a demo.
>>>> Avi Kivity, our CTO, will host a session on Scylla on Thursday, 1:50 PM
>>>> - 2:30 PM in rooms M1 - M3.
>>>>
>>>> Regards
>>>> Tzach
>>>> scylladb
>>>>
>>>>
>>>
>>
>

Re: ScyllaDB, a new open source, Cassandra-compatible NoSQL

[Tzach Livyatan]

On Wed, Sep 23, 2015 at 12:13 AM, Venkatesh Arivazhagan <
venkey.a...@gmail.com> wrote:

> I came across this article:
> zdnet.com/article/kvm-creators-open-source-fast-cassandra-drop-in-replacement-scylla/
>
> Tzach, I would love to know/understand moree about ScyllaDB too. Also the
> benchmark seems to have only 1 DB Server. Do you have benchmark numbers
> where more than 1 DB servers were involved? :)
>
Benchmark specs are here
http://www.scylladb.com/technology/cassandra-vs-scylla-benchmark/
Yes, it is one server.
More benchmarks, with more servers, will be publish soon.



>
>
> On Tue, Sep 22, 2015 at 1:40 PM, Sachin Nikam <skni...@gmail.com> wrote:
>
>> Tzach,
>> Can you point to any documentation on scylladb site which talks about
>> how/why scylla db performs better than Cassandra while using the same
>> architecture?
>> Regards
>> Sachin
>>
>> On Tue, Sep 22, 2015 at 9:18 AM, Tzach Livyatan <
>> tz...@cloudius-systems.com> wrote:
>>
>>> Hello Cassandra users,
>>>
>>> We are pleased to announce a new member of the Cassandra Ecosystem -
>>> ScyllaDB
>>> ScyllaDB is a new, open source, Cassandra-compatible NoSQL data store,
>>> written with the goal of delivering superior performance and consistent low
>>> latency.  Today, ScyllaDB runs 1M tps per server with sub 1ms latency.
>>>
>>> ScyllaDB  supports CQL, is compatible with Cassandra drivers, and works
>>> out of the box with Cassandra tools like cqlsh, Spark connector, nodetool
>>> and cassandra-stress. ScyllaDB is a drop-in replacement solution for the
>>> Cassandra server side packages.
>>>
>>> Scylla is implemented using the new shared-nothing Seastar
>>> <http://www.seastar-project.org/> framework for extreme performance on
>>> modern multicore hardware, and the Data Plane Development Kit (DPDK) for
>>> high-speed low-latency networking.
>>>
>>> Try Scylla Now - http://www.scylladb.com
>>>
>>> We will be at Cassandra summit 2015, you are welcome to visit our booth
>>> to hear more and see a demo.
>>> Avi Kivity, our CTO, will host a session on Scylla on Thursday, 1:50 PM
>>> - 2:30 PM in rooms M1 - M3.
>>>
>>> Regards
>>> Tzach
>>> scylladb
>>>
>>>
>>
>

Re: ScyllaDB, a new open source, Cassandra-compatible NoSQL

[Peter Lin]

very interesting. I'm glad to see someone building a drop in replacement
for Cassandra.

On Tue, Sep 22, 2015 at 5:40 PM, Tzach Livyatan <tz...@cloudius-systems.com>
wrote:

> Hi Sachin
>
> On Tue, Sep 22, 2015 at 11:40 PM, Sachin Nikam <skni...@gmail.com> wrote:
>
>> Tzach,
>> Can you point to any documentation on scylladb site which talks about
>> how/why scylla db performs better than Cassandra while using the same
>> architecture?
>>
> see here
> http://www.scylladb.com/technology/architecture/
>
>
>> Regards
>> Sachin
>>
>> On Tue, Sep 22, 2015 at 9:18 AM, Tzach Livyatan <
>> tz...@cloudius-systems.com> wrote:
>>
>>> Hello Cassandra users,
>>>
>>> We are pleased to announce a new member of the Cassandra Ecosystem -
>>> ScyllaDB
>>> ScyllaDB is a new, open source, Cassandra-compatible NoSQL data store,
>>> written with the goal of delivering superior performance and consistent low
>>> latency.  Today, ScyllaDB runs 1M tps per server with sub 1ms latency.
>>>
>>> ScyllaDB  supports CQL, is compatible with Cassandra drivers, and works
>>> out of the box with Cassandra tools like cqlsh, Spark connector, nodetool
>>> and cassandra-stress. ScyllaDB is a drop-in replacement solution for the
>>> Cassandra server side packages.
>>>
>>> Scylla is implemented using the new shared-nothing Seastar
>>> <http://www.seastar-project.org/> framework for extreme performance on
>>> modern multicore hardware, and the Data Plane Development Kit (DPDK) for
>>> high-speed low-latency networking.
>>>
>>> Try Scylla Now - http://www.scylladb.com
>>>
>>> We will be at Cassandra summit 2015, you are welcome to visit our booth
>>> to hear more and see a demo.
>>> Avi Kivity, our CTO, will host a session on Scylla on Thursday, 1:50 PM
>>> - 2:30 PM in rooms M1 - M3.
>>>
>>> Regards
>>> Tzach
>>> scylladb
>>>
>>>
>>
>

Re: ScyllaDB, a new open source, Cassandra-compatible NoSQL

[Tzach Livyatan]

Hi Sachin

On Tue, Sep 22, 2015 at 11:40 PM, Sachin Nikam <skni...@gmail.com> wrote:

> Tzach,
> Can you point to any documentation on scylladb site which talks about
> how/why scylla db performs better than Cassandra while using the same
> architecture?
>
see here
http://www.scylladb.com/technology/architecture/


> Regards
> Sachin
>
> On Tue, Sep 22, 2015 at 9:18 AM, Tzach Livyatan <
> tz...@cloudius-systems.com> wrote:
>
>> Hello Cassandra users,
>>
>> We are pleased to announce a new member of the Cassandra Ecosystem -
>> ScyllaDB
>> ScyllaDB is a new, open source, Cassandra-compatible NoSQL data store,
>> written with the goal of delivering superior performance and consistent low
>> latency.  Today, ScyllaDB runs 1M tps per server with sub 1ms latency.
>>
>> ScyllaDB  supports CQL, is compatible with Cassandra drivers, and works
>> out of the box with Cassandra tools like cqlsh, Spark connector, nodetool
>> and cassandra-stress. ScyllaDB is a drop-in replacement solution for the
>> Cassandra server side packages.
>>
>> Scylla is implemented using the new shared-nothing Seastar
>> <http://www.seastar-project.org/> framework for extreme performance on
>> modern multicore hardware, and the Data Plane Development Kit (DPDK) for
>> high-speed low-latency networking.
>>
>> Try Scylla Now - http://www.scylladb.com
>>
>> We will be at Cassandra summit 2015, you are welcome to visit our booth
>> to hear more and see a demo.
>> Avi Kivity, our CTO, will host a session on Scylla on Thursday, 1:50 PM -
>> 2:30 PM in rooms M1 - M3.
>>
>> Regards
>> Tzach
>> scylladb
>>
>>
>

ScyllaDB, a new open source, Cassandra-compatible NoSQL

[Tzach Livyatan]

Hello Cassandra users,

We are pleased to announce a new member of the Cassandra Ecosystem -
ScyllaDB
ScyllaDB is a new, open source, Cassandra-compatible NoSQL data store,
written with the goal of delivering superior performance and consistent low
latency.  Today, ScyllaDB runs 1M tps per server with sub 1ms latency.

ScyllaDB  supports CQL, is compatible with Cassandra drivers, and works out
of the box with Cassandra tools like cqlsh, Spark connector, nodetool and
cassandra-stress. ScyllaDB is a drop-in replacement solution for the
Cassandra server side packages.

Scylla is implemented using the new shared-nothing Seastar
<http://www.seastar-project.org/> framework for extreme performance on
modern multicore hardware, and the Data Plane Development Kit (DPDK) for
high-speed low-latency networking.

Try Scylla Now - http://www.scylladb.com

We will be at Cassandra summit 2015, you are welcome to visit our booth to
hear more and see a demo.
Avi Kivity, our CTO, will host a session on Scylla on Thursday, 1:50 PM -
2:30 PM in rooms M1 - M3.

Regards
Tzach
scylladb