Re: Cassandra table limitation

[Kai Wang]

With small data size and unknown access pattern, any particular reason to
choose C*? It sounds like a relational database fits better.

On Tue, Apr 5, 2016 at 11:40 PM, jason zhao yang <
zhaoyangsingap...@gmail.com> wrote:

> Hi Jack,
>
> Thanks for the reply.
>
> Each tenant will has around 50-100 tables for their applications. probably
> log collection, probably account table, it's not fixed and depends on
> tenants' need.
>
> There will be a team in charge of helping tenant to do data modeling and
> access patterns. Tenants will not directly admin on the cluster, we will
> take care.
>
> Yes, multi-cluster is a solution. But the cost will be quite high, because
> each tenant's data is far less than the capacity of a 3 node cluster. So I
> want to put multiple tenants into one clusters.
>
>
>
> Jack Krupansky 于2016年4月6日周三 上午10:41写道：
>
>> What is the nature of these tenants? Are they each creating their own
>> data models? Is there one central authority that will approve of all data
>> models and who can adjust the cluster configuration to support those models?
>>
>> Generally speaking, multi-tenancy is an anti-pattern for Cassandra and
>> for most servers. The proper way to do multitenancy is to not do it at all,
>> and to use separate machines or at least separate virtual machines.
>>
>> In particular, there needs to be a central authority managing a Cassandra
>> cluster to assure its smooth operation. If each tenant is going in their
>> own directions, then nobody will be in charge and capable of assuring that
>> everybody is on the same page.
>>
>> Again, it depends on the nature of these tenants and how much control the
>> cluster administrator has over them.
>>
>> Think of a Cassandra cluster as managing the data for either a single
>> application or a collection of applications which share the same data. If
>> there are multiple applications that don't share the same data, then they
>> absolutely should be on separate clusters.
>>
>>
>> -- Jack Krupansky
>>
>> On Tue, Apr 5, 2016 at 5:40 PM, Kai Wang  wrote:
>>
>>> Once a while the question about table count rises in this list. The most
>>> recent is
>>> https://groups.google.com/forum/#!topic/nosql-databases/IblAhiLUXdk
>>>
>>> In short C* is not designed to scale with the table count. For one each
>>> table/CF has some fixed memory footprint on *ALL* nodes. The consensus is
>>> you shouldn't have more than "a few hundreds" of tables.
>>>
>>> On Mon, Apr 4, 2016 at 10:17 AM, jason zhao yang <
>>> zhaoyangsingap...@gmail.com> wrote:
>>>
 Hi,

 This is Jason.

 Currently, I am using C* 2.1.10, I want to ask what's the optimal
 number of tables I should create in one cluster?

 My use case is that I will prepare a keyspace for each of my tenant,
 and every tenant will create tables they needed. Assume each tenant created
 50 tables with normal workload (half read, half write).   so how many
 number of tenants I can support in one cluster?

 I know there are a few issues related to large number of tables.
 * frequent GC
 * frequent flush due to insufficient memory
 * large latency when modifying table schema
 * large amount of tombstones during creating table

 Is there any other issues with large number of tables? Using a 32GB
 instance, I can easily create 4000 tables with off-heap-memtable.

 BTW, Is this table limitation solved in 3.X?

 Thank you very much.


>>>
>>

Re: cross DC data sync starts without rebuilding nodes on new DC

[Vijay Patil]

Thanks Alain and Sean, your detailed explanation answers my question.

Yes, nodetool status reflects new DC and nodetool netstats says not "No
Streams".
My all writes going to old DC with local_quorum. Yes this new data might be
getting synced into new DC (repair was not running anywhere).
I will proceed with rebuilding nodes on new DC.

Thanks,
Vijay

On 5 April 2016 at 18:56, Alain RODRIGUEZ  wrote:

> Hi Vijay.
>
> After completing step 7.a (which is altering keyspace with desired
>> replication factor for new DC) data automatically starts syncing from
>> existing DC to new DC
>>
>
> My guess: what you are seeing is not data syncing. Well it is, but not old
> data being streamed but new writes being replicated. As soon as you set the
> RF for the new DC, it starts accepting writes.
>
> Some background:
> Using a Local_X consistency level means the operation to copy data to all
> the DC won't happen, it means coordinator won't wait for ack from other DC
> nodes, but write should reach all the DC set in the keyspace configuration.
> So as soon as you say I want X copies of the data on the new Datacenter,
> new data start to be replicated there.
>
> To check:
>
> Are you writing in your original DC?
> Is the output of 'nodetool netstats' saying 'No streams' as I expect?
>
> When rebuilding run this command again and you should see streams.
>
> Any idea why it's happening?
>> If this is the desired behaviour then what's the purpose of rebuilding
>> each node on new DC (step 7.b)?
>>
>
> So basically, the rebuild allows the new cluster to have the *old* /
> *existing* data streamed from an other DC. We use rebuild instead of
> auto_bootstrap to avoid nodes trying to stream data as soon as they are
> added to the new DC because we want to add *all* the nodes, to have ranges
> distributed evenly before starting streaming to stream just the correct
> amount of data from the DC of our choice.
>
> C*heers,
> ---
> Alain Rodriguez - al...@thelastpickle.com
> France
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
>
> 2016-04-05 8:26 GMT+02:00 Vijay Patil :
>
>> Hi,
>>
>> I have configured new DC as per instructions at below link.
>>
>> https://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html
>>
>> After completing step 7.a (which is altering keyspace with desired
>> replication factor for new DC) data automatically starts syncing from
>> existing DC to new DC, which is not expected because auto_bootstrap is
>> false on all nodes (existing as well as new DC).
>>
>> Any idea why it's happening?
>> If this is the desired behaviour then what's the purpose of rebuilding
>> each node on new DC (step 7.b)?
>>
>> Cassandra version is 2.0.17 on all nodes in both DC's and I am using
>> GossipingPropertyFileSnitch.
>>
>> Regards,
>> Vijay
>>
>
>

Re: Cassandra table limitation

[jason zhao yang]

Thanks Kai,

One approach discussed in that post is about disabling slab allocation.
What are the consequences except for lower GC performance?


Kai Wang 于2016年4月6日周三 上午5:40写道：

> Once a while the question about table count rises in this list. The most
> recent is
> https://groups.google.com/forum/#!topic/nosql-databases/IblAhiLUXdk
>
> In short C* is not designed to scale with the table count. For one each
> table/CF has some fixed memory footprint on *ALL* nodes. The consensus is
> you shouldn't have more than "a few hundreds" of tables.
>
> On Mon, Apr 4, 2016 at 10:17 AM, jason zhao yang <
> zhaoyangsingap...@gmail.com> wrote:
>
>> Hi,
>>
>> This is Jason.
>>
>> Currently, I am using C* 2.1.10, I want to ask what's the optimal number
>> of tables I should create in one cluster?
>>
>> My use case is that I will prepare a keyspace for each of my tenant, and
>> every tenant will create tables they needed. Assume each tenant created 50
>> tables with normal workload (half read, half write).   so how many number
>> of tenants I can support in one cluster?
>>
>> I know there are a few issues related to large number of tables.
>> * frequent GC
>> * frequent flush due to insufficient memory
>> * large latency when modifying table schema
>> * large amount of tombstones during creating table
>>
>> Is there any other issues with large number of tables? Using a 32GB
>> instance, I can easily create 4000 tables with off-heap-memtable.
>>
>> BTW, Is this table limitation solved in 3.X?
>>
>> Thank you very much.
>>
>>
>

Re: Cassandra table limitation

[Jack Krupansky]

What is the nature of these tenants? Are they each creating their own data
models? Is there one central authority that will approve of all data models
and who can adjust the cluster configuration to support those models?

Generally speaking, multi-tenancy is an anti-pattern for Cassandra and for
most servers. The proper way to do multitenancy is to not do it at all, and
to use separate machines or at least separate virtual machines.

In particular, there needs to be a central authority managing a Cassandra
cluster to assure its smooth operation. If each tenant is going in their
own directions, then nobody will be in charge and capable of assuring that
everybody is on the same page.

Again, it depends on the nature of these tenants and how much control the
cluster administrator has over them.

Think of a Cassandra cluster as managing the data for either a single
application or a collection of applications which share the same data. If
there are multiple applications that don't share the same data, then they
absolutely should be on separate clusters.


-- Jack Krupansky

On Tue, Apr 5, 2016 at 5:40 PM, Kai Wang  wrote:

> Once a while the question about table count rises in this list. The most
> recent is
> https://groups.google.com/forum/#!topic/nosql-databases/IblAhiLUXdk
>
> In short C* is not designed to scale with the table count. For one each
> table/CF has some fixed memory footprint on *ALL* nodes. The consensus is
> you shouldn't have more than "a few hundreds" of tables.
>
> On Mon, Apr 4, 2016 at 10:17 AM, jason zhao yang <
> zhaoyangsingap...@gmail.com> wrote:
>
>> Hi,
>>
>> This is Jason.
>>
>> Currently, I am using C* 2.1.10, I want to ask what's the optimal number
>> of tables I should create in one cluster?
>>
>> My use case is that I will prepare a keyspace for each of my tenant, and
>> every tenant will create tables they needed. Assume each tenant created 50
>> tables with normal workload (half read, half write).   so how many number
>> of tenants I can support in one cluster?
>>
>> I know there are a few issues related to large number of tables.
>> * frequent GC
>> * frequent flush due to insufficient memory
>> * large latency when modifying table schema
>> * large amount of tombstones during creating table
>>
>> Is there any other issues with large number of tables? Using a 32GB
>> instance, I can easily create 4000 tables with off-heap-memtable.
>>
>> BTW, Is this table limitation solved in 3.X?
>>
>> Thank you very much.
>>
>>
>

Re: Is it possible to achieve "sticky" request routing?

[Steve Robenalt]

Hi Micky,

The only issues I've seen personally with dropped updates due to even small
amounts of time skew were ones I was able to fix by the judicious use of
BatchStatements. This was particularly true with counter fields in the 2.0
release (i.e. before counters were fixed), but would also happen with
different columns updated in separate statements. I'm not sure what your
circumstances are for the lost updates, so I'm not sure if these will help.
I'm only pointing it out because it was effective for my cases.

Steve


On Tue, Apr 5, 2016 at 3:21 PM, Mukil Kesavan 
wrote:

> John and Steve,
>
> We use QUORUM consistency for both READS and WRITES. So we won't have the
> problem of having to pick the right server. The real reason we have this
> requirement is because we run in a sometimes overloaded virtualized
> environment that causes the servers to have non-trivial time drifts (tens
> of milliseconds or a few seconds, even with NTP, which catches up slowly).
> This particular client was seeing some updates being dropped silently by
> Cassandra when it hit a server with an older local timestamp. They were
> relying on server side timestamp generation.
>
> So they were exploring "sticky" routing as an option since the likelihood
> of having monotonically increasing timestamps is higher if the client's
> requests always go to a single server. They are aware of the problem of
> disconnecting and reconnecting to a new server and have an application
> level solution for this. They are also testing using client side timestamps
> as well. We're just trying to explore all our options and their pros and
> cons.
>
> Thanks!
>
> On Tue, Apr 5, 2016 at 1:31 PM, Jonathan Haddad  wrote:
>
>> Yep - Steve hit the nail on the head.  The odds of hitting the right
>> server with "sticky routing" goes down as your cluster size increases.  You
>> end up adding extra network hops instead of using token aware routing.
>>
>> Unless you're trying to do a coordinator tier (and you're not, according
>> to your original post), this is a pretty bad idea and I'd advise you to
>> push back on that requirement.
>>
>> On Tue, Apr 5, 2016 at 12:47 PM Steve Robenalt 
>> wrote:
>>
>>> Aside from Jon's "why" question, I would point out that this only really
>>> works because you are running a 3 node cluster with RF=3. If your cluster
>>> is going to grow, you can't guarantee that any one server would have all
>>> records. I'd be pretty hesitant to put an invisible constraint like that on
>>> a cluster unless you're pretty sure it'll only ever be 3 nodes.
>>>
>>> On Tue, Apr 5, 2016 at 9:34 AM, Jonathan Haddad 
>>> wrote:
>>>
 Why is this a requirement?  Honestly I don't know why you would do this.


 On Sat, Apr 2, 2016 at 8:06 PM Mukil Kesavan 
 wrote:

> Hello,
>
> We currently have 3 Cassandra servers running in a single datacenter
> with a replication factor of 3 for our keyspace. We also use the
> SimpleSnitch wiith DynamicSnitching enabled by default. Our load balancing
> policy is TokenAwareLoadBalancingPolicy with RoundRobinPolicy as the 
> child.
> This overall configuration results in our client requests spreading 
> equally
> across our 3 servers.
>
> However, we have a new requirement where we need to restrict a
> client's requests to a single server and only go to the other servers on
> failure of the previous server. This particular use case does not have 
> high
> request traffic.
>
> Looking at the documentation the options we have seem to be:
>
> 1. Play with the snitching (e.g. place each server into its own DC or
> Rack) to ensure that requests always go to one server and failover to the
> others if required. I understand that this may also affect replica
> placement and we may need to run nodetool repair. So this is not our most
> preferred option.
>
> 2. Write a new load balancing policy that also uses the
> HostStateListener for tracking host up and down messages, that essentially
> accomplishes "sticky" request routing with failover to other nodes.
>
> Is option 2 the only clean way of accomplishing our requirement?
>
> Thanks,
> Micky
>

>>>
>>>
>>> --
>>> Steve Robenalt
>>> Software Architect
>>> sroben...@highwire.org 
>>> (office/cell): 916-505-1785
>>>
>>> HighWire Press, Inc.
>>> 425 Broadway St, Redwood City, CA 94063
>>> www.highwire.org
>>>
>>> Technology for Scholarly Communication
>>>
>>
>


-- 
Steve Robenalt
Software Architect
sroben...@highwire.org 
(office/cell): 916-505-1785

HighWire Press, Inc.
425 Broadway St, Redwood City, CA 94063
www.highwire.org

Technology for Scholarly Communication

Re: Is it possible to achieve "sticky" request routing?

[Mukil Kesavan]

John and Steve,

We use QUORUM consistency for both READS and WRITES. So we won't have the
problem of having to pick the right server. The real reason we have this
requirement is because we run in a sometimes overloaded virtualized
environment that causes the servers to have non-trivial time drifts (tens
of milliseconds or a few seconds, even with NTP, which catches up slowly).
This particular client was seeing some updates being dropped silently by
Cassandra when it hit a server with an older local timestamp. They were
relying on server side timestamp generation.

So they were exploring "sticky" routing as an option since the likelihood
of having monotonically increasing timestamps is higher if the client's
requests always go to a single server. They are aware of the problem of
disconnecting and reconnecting to a new server and have an application
level solution for this. They are also testing using client side timestamps
as well. We're just trying to explore all our options and their pros and
cons.

Thanks!

On Tue, Apr 5, 2016 at 1:31 PM, Jonathan Haddad  wrote:

> Yep - Steve hit the nail on the head.  The odds of hitting the right
> server with "sticky routing" goes down as your cluster size increases.  You
> end up adding extra network hops instead of using token aware routing.
>
> Unless you're trying to do a coordinator tier (and you're not, according
> to your original post), this is a pretty bad idea and I'd advise you to
> push back on that requirement.
>
> On Tue, Apr 5, 2016 at 12:47 PM Steve Robenalt 
> wrote:
>
>> Aside from Jon's "why" question, I would point out that this only really
>> works because you are running a 3 node cluster with RF=3. If your cluster
>> is going to grow, you can't guarantee that any one server would have all
>> records. I'd be pretty hesitant to put an invisible constraint like that on
>> a cluster unless you're pretty sure it'll only ever be 3 nodes.
>>
>> On Tue, Apr 5, 2016 at 9:34 AM, Jonathan Haddad 
>> wrote:
>>
>>> Why is this a requirement?  Honestly I don't know why you would do this.
>>>
>>>
>>> On Sat, Apr 2, 2016 at 8:06 PM Mukil Kesavan 
>>> wrote:
>>>
 Hello,

 We currently have 3 Cassandra servers running in a single datacenter
 with a replication factor of 3 for our keyspace. We also use the
 SimpleSnitch wiith DynamicSnitching enabled by default. Our load balancing
 policy is TokenAwareLoadBalancingPolicy with RoundRobinPolicy as the child.
 This overall configuration results in our client requests spreading equally
 across our 3 servers.

 However, we have a new requirement where we need to restrict a client's
 requests to a single server and only go to the other servers on failure of
 the previous server. This particular use case does not have high request
 traffic.

 Looking at the documentation the options we have seem to be:

 1. Play with the snitching (e.g. place each server into its own DC or
 Rack) to ensure that requests always go to one server and failover to the
 others if required. I understand that this may also affect replica
 placement and we may need to run nodetool repair. So this is not our most
 preferred option.

 2. Write a new load balancing policy that also uses the
 HostStateListener for tracking host up and down messages, that essentially
 accomplishes "sticky" request routing with failover to other nodes.

 Is option 2 the only clean way of accomplishing our requirement?

 Thanks,
 Micky

>>>
>>
>>
>> --
>> Steve Robenalt
>> Software Architect
>> sroben...@highwire.org 
>> (office/cell): 916-505-1785
>>
>> HighWire Press, Inc.
>> 425 Broadway St, Redwood City, CA 94063
>> www.highwire.org
>>
>> Technology for Scholarly Communication
>>
>

Re: Is it possible to achieve "sticky" request routing?

[Jonathan Haddad]

Jim, that's not what he asked. He asked for the equivalent of a load
balancer with sticky sessions.


On Tue, Apr 5, 2016 at 2:24 PM Jim Ancona  wrote:

> Jon and Steve:
>
> I don't understand your point. The TokenAwareLoadBalancer identifies the
> nodes in the cluster that own the data for a particular token and route
> requests to one of them. As I understand it, the OP wants to send requests
> for a particular token to the same node every time (assuming it's
> available). How does that fail in a large cluster?
>
> Jim
>
> On Tue, Apr 5, 2016 at 4:31 PM, Jonathan Haddad  wrote:
>
>> Yep - Steve hit the nail on the head.  The odds of hitting the right
>> server with "sticky routing" goes down as your cluster size increases.  You
>> end up adding extra network hops instead of using token aware routing.
>>
>> Unless you're trying to do a coordinator tier (and you're not, according
>> to your original post), this is a pretty bad idea and I'd advise you to
>> push back on that requirement.
>>
>> On Tue, Apr 5, 2016 at 12:47 PM Steve Robenalt 
>> wrote:
>>
>>> Aside from Jon's "why" question, I would point out that this only really
>>> works because you are running a 3 node cluster with RF=3. If your cluster
>>> is going to grow, you can't guarantee that any one server would have all
>>> records. I'd be pretty hesitant to put an invisible constraint like that on
>>> a cluster unless you're pretty sure it'll only ever be 3 nodes.
>>>
>>> On Tue, Apr 5, 2016 at 9:34 AM, Jonathan Haddad 
>>> wrote:
>>>
 Why is this a requirement?  Honestly I don't know why you would do this.


 On Sat, Apr 2, 2016 at 8:06 PM Mukil Kesavan 
 wrote:

> Hello,
>
> We currently have 3 Cassandra servers running in a single datacenter
> with a replication factor of 3 for our keyspace. We also use the
> SimpleSnitch wiith DynamicSnitching enabled by default. Our load balancing
> policy is TokenAwareLoadBalancingPolicy with RoundRobinPolicy as the 
> child.
> This overall configuration results in our client requests spreading 
> equally
> across our 3 servers.
>
> However, we have a new requirement where we need to restrict a
> client's requests to a single server and only go to the other servers on
> failure of the previous server. This particular use case does not have 
> high
> request traffic.
>
> Looking at the documentation the options we have seem to be:
>
> 1. Play with the snitching (e.g. place each server into its own DC or
> Rack) to ensure that requests always go to one server and failover to the
> others if required. I understand that this may also affect replica
> placement and we may need to run nodetool repair. So this is not our most
> preferred option.
>
> 2. Write a new load balancing policy that also uses the
> HostStateListener for tracking host up and down messages, that essentially
> accomplishes "sticky" request routing with failover to other nodes.
>
> Is option 2 the only clean way of accomplishing our requirement?
>
> Thanks,
> Micky
>

>>>
>>>
>>> --
>>> Steve Robenalt
>>> Software Architect
>>> sroben...@highwire.org 
>>> (office/cell): 916-505-1785
>>>
>>> HighWire Press, Inc.
>>> 425 Broadway St, Redwood City, CA 94063
>>> www.highwire.org
>>>
>>> Technology for Scholarly Communication
>>>
>>
>

Re: Cassandra table limitation

[Kai Wang]

Once a while the question about table count rises in this list. The most
recent is
https://groups.google.com/forum/#!topic/nosql-databases/IblAhiLUXdk

In short C* is not designed to scale with the table count. For one each
table/CF has some fixed memory footprint on *ALL* nodes. The consensus is
you shouldn't have more than "a few hundreds" of tables.

On Mon, Apr 4, 2016 at 10:17 AM, jason zhao yang <
zhaoyangsingap...@gmail.com> wrote:

> Hi,
>
> This is Jason.
>
> Currently, I am using C* 2.1.10, I want to ask what's the optimal number
> of tables I should create in one cluster?
>
> My use case is that I will prepare a keyspace for each of my tenant, and
> every tenant will create tables they needed. Assume each tenant created 50
> tables with normal workload (half read, half write).   so how many number
> of tenants I can support in one cluster?
>
> I know there are a few issues related to large number of tables.
> * frequent GC
> * frequent flush due to insufficient memory
> * large latency when modifying table schema
> * large amount of tombstones during creating table
>
> Is there any other issues with large number of tables? Using a 32GB
> instance, I can easily create 4000 tables with off-heap-memtable.
>
> BTW, Is this table limitation solved in 3.X?
>
> Thank you very much.
>
>

Re: Is it possible to achieve "sticky" request routing?

[Jim Ancona]

Jon and Steve:

I don't understand your point. The TokenAwareLoadBalancer identifies the
nodes in the cluster that own the data for a particular token and route
requests to one of them. As I understand it, the OP wants to send requests
for a particular token to the same node every time (assuming it's
available). How does that fail in a large cluster?

Jim

On Tue, Apr 5, 2016 at 4:31 PM, Jonathan Haddad  wrote:

> Yep - Steve hit the nail on the head.  The odds of hitting the right
> server with "sticky routing" goes down as your cluster size increases.  You
> end up adding extra network hops instead of using token aware routing.
>
> Unless you're trying to do a coordinator tier (and you're not, according
> to your original post), this is a pretty bad idea and I'd advise you to
> push back on that requirement.
>
> On Tue, Apr 5, 2016 at 12:47 PM Steve Robenalt 
> wrote:
>
>> Aside from Jon's "why" question, I would point out that this only really
>> works because you are running a 3 node cluster with RF=3. If your cluster
>> is going to grow, you can't guarantee that any one server would have all
>> records. I'd be pretty hesitant to put an invisible constraint like that on
>> a cluster unless you're pretty sure it'll only ever be 3 nodes.
>>
>> On Tue, Apr 5, 2016 at 9:34 AM, Jonathan Haddad 
>> wrote:
>>
>>> Why is this a requirement?  Honestly I don't know why you would do this.
>>>
>>>
>>> On Sat, Apr 2, 2016 at 8:06 PM Mukil Kesavan 
>>> wrote:
>>>
 Hello,

 We currently have 3 Cassandra servers running in a single datacenter
 with a replication factor of 3 for our keyspace. We also use the
 SimpleSnitch wiith DynamicSnitching enabled by default. Our load balancing
 policy is TokenAwareLoadBalancingPolicy with RoundRobinPolicy as the child.
 This overall configuration results in our client requests spreading equally
 across our 3 servers.

 However, we have a new requirement where we need to restrict a client's
 requests to a single server and only go to the other servers on failure of
 the previous server. This particular use case does not have high request
 traffic.

 Looking at the documentation the options we have seem to be:

 1. Play with the snitching (e.g. place each server into its own DC or
 Rack) to ensure that requests always go to one server and failover to the
 others if required. I understand that this may also affect replica
 placement and we may need to run nodetool repair. So this is not our most
 preferred option.

 2. Write a new load balancing policy that also uses the
 HostStateListener for tracking host up and down messages, that essentially
 accomplishes "sticky" request routing with failover to other nodes.

 Is option 2 the only clean way of accomplishing our requirement?

 Thanks,
 Micky

>>>
>>
>>
>> --
>> Steve Robenalt
>> Software Architect
>> sroben...@highwire.org 
>> (office/cell): 916-505-1785
>>
>> HighWire Press, Inc.
>> 425 Broadway St, Redwood City, CA 94063
>> www.highwire.org
>>
>> Technology for Scholarly Communication
>>
>

Re: Is it possible to achieve "sticky" request routing?

[Steve Robenalt]

Aside from Jon's "why" question, I would point out that this only really
works because you are running a 3 node cluster with RF=3. If your cluster
is going to grow, you can't guarantee that any one server would have all
records. I'd be pretty hesitant to put an invisible constraint like that on
a cluster unless you're pretty sure it'll only ever be 3 nodes.

On Tue, Apr 5, 2016 at 9:34 AM, Jonathan Haddad  wrote:

> Why is this a requirement?  Honestly I don't know why you would do this.
>
>
> On Sat, Apr 2, 2016 at 8:06 PM Mukil Kesavan 
> wrote:
>
>> Hello,
>>
>> We currently have 3 Cassandra servers running in a single datacenter with
>> a replication factor of 3 for our keyspace. We also use the SimpleSnitch
>> wiith DynamicSnitching enabled by default. Our load balancing policy is
>> TokenAwareLoadBalancingPolicy with RoundRobinPolicy as the child. This
>> overall configuration results in our client requests spreading equally
>> across our 3 servers.
>>
>> However, we have a new requirement where we need to restrict a client's
>> requests to a single server and only go to the other servers on failure of
>> the previous server. This particular use case does not have high request
>> traffic.
>>
>> Looking at the documentation the options we have seem to be:
>>
>> 1. Play with the snitching (e.g. place each server into its own DC or
>> Rack) to ensure that requests always go to one server and failover to the
>> others if required. I understand that this may also affect replica
>> placement and we may need to run nodetool repair. So this is not our most
>> preferred option.
>>
>> 2. Write a new load balancing policy that also uses the HostStateListener
>> for tracking host up and down messages, that essentially accomplishes
>> "sticky" request routing with failover to other nodes.
>>
>> Is option 2 the only clean way of accomplishing our requirement?
>>
>> Thanks,
>> Micky
>>
>


-- 
Steve Robenalt
Software Architect
sroben...@highwire.org 
(office/cell): 916-505-1785

HighWire Press, Inc.
425 Broadway St, Redwood City, CA 94063
www.highwire.org

Technology for Scholarly Communication

Re: Is it possible to achieve "sticky" request routing?

[David King]

Many C* versions back I did this by writing a custom snitch. This was to 
maximise use of the row cache but had the effect of pinning requests for a key 
to a given server. It sounds like you want to do the same thing but ignoring 
the key. In more modern reversions I believe this may be done as a load 
balancing policy.

I don't have the code handy but it was only a few dozen lines of Java, mostly 
boilerplate. IIRC the interface method just asks you to sort a list of IP 
addresses in your order of preference.

Note that this only really does anything for the ONE consistency level. In any 
other level you'll end up hitting all 3 servers anyway.


> On 02 Apr 2016, at 20:06, Mukil Kesavan  wrote:
> 
> Hello,
> 
> We currently have 3 Cassandra servers running in a single datacenter with a 
> replication factor of 3 for our keyspace. We also use the SimpleSnitch wiith 
> DynamicSnitching enabled by default. Our load balancing policy is 
> TokenAwareLoadBalancingPolicy with RoundRobinPolicy as the child. This 
> overall configuration results in our client requests spreading equally across 
> our 3 servers.
> 
> However, we have a new requirement where we need to restrict a client's 
> requests to a single server and only go to the other servers on failure of 
> the previous server. This particular use case does not have high request 
> traffic.
> 
> Looking at the documentation the options we have seem to be:
> 
> 1. Play with the snitching (e.g. place each server into its own DC or Rack) 
> to ensure that requests always go to one server and failover to the others if 
> required. I understand that this may also affect replica placement and we may 
> need to run nodetool repair. So this is not our most preferred option.
> 
> 2. Write a new load balancing policy that also uses the HostStateListener for 
> tracking host up and down messages, that essentially accomplishes "sticky" 
> request routing with failover to other nodes.
> 
> Is option 2 the only clean way of accomplishing our requirement?
> 
> Thanks,
> Micky



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Is it possible to achieve "sticky" request routing?

[Jonathan Haddad]

Why is this a requirement?  Honestly I don't know why you would do this.

On Sat, Apr 2, 2016 at 8:06 PM Mukil Kesavan 
wrote:

> Hello,
>
> We currently have 3 Cassandra servers running in a single datacenter with
> a replication factor of 3 for our keyspace. We also use the SimpleSnitch
> wiith DynamicSnitching enabled by default. Our load balancing policy is
> TokenAwareLoadBalancingPolicy with RoundRobinPolicy as the child. This
> overall configuration results in our client requests spreading equally
> across our 3 servers.
>
> However, we have a new requirement where we need to restrict a client's
> requests to a single server and only go to the other servers on failure of
> the previous server. This particular use case does not have high request
> traffic.
>
> Looking at the documentation the options we have seem to be:
>
> 1. Play with the snitching (e.g. place each server into its own DC or
> Rack) to ensure that requests always go to one server and failover to the
> others if required. I understand that this may also affect replica
> placement and we may need to run nodetool repair. So this is not our most
> preferred option.
>
> 2. Write a new load balancing policy that also uses the HostStateListener
> for tracking host up and down messages, that essentially accomplishes
> "sticky" request routing with failover to other nodes.
>
> Is option 2 the only clean way of accomplishing our requirement?
>
> Thanks,
> Micky
>

Re : Optimum value for native_transport_max_concurrent_connections and native_transport_max_concurrent_connections_per_ip

[sai krishnam raju potturi]

hi;
  we are upgrading our cluster from apache-cassandra 2.0.14 to 2.0.17. We
have been facing SYN flooding issue (port 9042) in our current version of
Cassandra at times. We are hoping to tackle the SYN flooding issues with
the following attributes in the YAML file for 2.0.17

native_transport_max_concurrent_connections

native_transport_max_concurrent_connections_per_ip


Are there any observed limitations for the above mentioned attributes.
During the peak hours each node serves around 1500 connections. Please
suggest optimal values for the mentioned attributes.


thanks
Sai Potturi

Re: cross DC data sync starts without rebuilding nodes on new DC

[Alain RODRIGUEZ]

Hi Vijay.

After completing step 7.a (which is altering keyspace with desired
> replication factor for new DC) data automatically starts syncing from
> existing DC to new DC
>

My guess: what you are seeing is not data syncing. Well it is, but not old
data being streamed but new writes being replicated. As soon as you set the
RF for the new DC, it starts accepting writes.

Some background:
Using a Local_X consistency level means the operation to copy data to all
the DC won't happen, it means coordinator won't wait for ack from other DC
nodes, but write should reach all the DC set in the keyspace configuration.
So as soon as you say I want X copies of the data on the new Datacenter,
new data start to be replicated there.

To check:

Are you writing in your original DC?
Is the output of 'nodetool netstats' saying 'No streams' as I expect?

When rebuilding run this command again and you should see streams.

Any idea why it's happening?
> If this is the desired behaviour then what's the purpose of rebuilding
> each node on new DC (step 7.b)?
>

So basically, the rebuild allows the new cluster to have the *old* /
*existing* data streamed from an other DC. We use rebuild instead of
auto_bootstrap to avoid nodes trying to stream data as soon as they are
added to the new DC because we want to add *all* the nodes, to have ranges
distributed evenly before starting streaming to stream just the correct
amount of data from the DC of our choice.

C*heers,
---
Alain Rodriguez - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com


2016-04-05 8:26 GMT+02:00 Vijay Patil :

> Hi,
>
> I have configured new DC as per instructions at below link.
>
> https://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html
>
> After completing step 7.a (which is altering keyspace with desired
> replication factor for new DC) data automatically starts syncing from
> existing DC to new DC, which is not expected because auto_bootstrap is
> false on all nodes (existing as well as new DC).
>
> Any idea why it's happening?
> If this is the desired behaviour then what's the purpose of rebuilding
> each node on new DC (step 7.b)?
>
> Cassandra version is 2.0.17 on all nodes in both DC's and I am using
> GossipingPropertyFileSnitch.
>
> Regards,
> Vijay
>

RE: cross DC data sync starts without rebuilding nodes on new DC

[SEAN_R_DURITY]

What do you mean by “automatically starts syncing?” Are you seeing streaming of 
existing data or just the addition of new, incoming data? Do you have repairs 
running as part of your automated maintenance, perhaps?

I would expect that new, incoming data would be replicated to the new DC after 
the “alter table.” But I would not expect any streaming of existing data unless 
repairs are running somewhere.

Also, does your nodetool status reflect the new DC?

Sean Durity

From: Vijay Patil [mailto:vijay2110.t...@gmail.com]
Sent: Tuesday, April 05, 2016 2:27 AM
To: user@cassandra.apache.org
Subject: cross DC data sync starts without rebuilding nodes on new DC

Hi,

I have configured new DC as per instructions at below link.
https://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html

After completing step 7.a (which is altering keyspace with desired replication 
factor for new DC) data automatically starts syncing from existing DC to new 
DC, which is not expected because auto_bootstrap is false on all nodes 
(existing as well as new DC).

Any idea why it's happening?
If this is the desired behaviour then what's the purpose of rebuilding each 
node on new DC (step 7.b)?

Cassandra version is 2.0.17 on all nodes in both DC's and I am using 
GossipingPropertyFileSnitch.

Regards,
Vijay



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: Clustering key and secondary index behavior changed between 2.0.11 and 3.3.0

[Sylvain Lebresne]

I'm surprised this would have fall through the cracks but that certainly
look like a regression (a bug). If you can reproduce on 3.0.4 (just to make
sure we haven't fixed it recently), then please open a ticket in
https://issues.apache.org/jira/browse/CASSANDRA/ with your repro steps.

On Tue, Apr 5, 2016 at 10:47 AM, julien muller 
wrote:

> Hello,
>
> I noticed the following change in behavior while migrating from 2.0.11:
> Elements of the clustering key seems to not be secondary indexable anymore.
> Anyone could give me an insight about this issue? Or point me to documents
> relative to this evolution?
>
> There is a sample table with some data:
> CREATE TABLE table1 (
> name text,
> class int,
> inter text,
> power int,
> PRIMARY KEY (name, class, inter)
> ) WITH CLUSTERING ORDER BY (class DESC, inter ASC);
> INSERT INTO table1 (name, class, inter, power) VALUES ('R1',1, 'int1',13);
> INSERT INTO table1 (name, class, inter, power) VALUES ('R1',2, 'int1',18);
> INSERT INTO table1 (name, class, inter, power) VALUES ('R1',3, 'int1',37);
> INSERT INTO table1 (name, class, inter, power) VALUES ('R1',4, 'int1',49);
>
> In version 2.0.11, I used to have a secondary index on inter, that allowed
> me to make fast queries on this table:
> CREATE INDEX table1_inter ON table1 (inter);
> SELECT * FROM table1 where name='R1' AND class>0 AND class<4 AND
> inter='int1' ALLOW FILTERING;
>
> While testing on 3.3.0, I get the following message:
> *Cluste*
> *ring column "inter" cannot be restricted (preceding column "class" is
> restricted by a non-EQ relation)*
> It seems to only be considered as a key and the index and ALLOW FILTERING
> are not taken into account anymore (as it was in 2.0.11).
>
> Further tests confused me, as I found an ugly workaround: If I duplicate
> the column inter as a regular column, I can simply query it with the
> secondary index and no ALLOW FILTERING. It looks like the behavior I would
> anticipate and do not understand why it does not work on inter only because
> it is a clustering key.
>
> Any insight highly appreciated!
>
> Julien
>

Clustering key and secondary index behavior changed between 2.0.11 and 3.3.0

[julien muller]

Hello,

I noticed the following change in behavior while migrating from 2.0.11:
Elements of the clustering key seems to not be secondary indexable anymore.
Anyone could give me an insight about this issue? Or point me to documents
relative to this evolution?

There is a sample table with some data:
CREATE TABLE table1 (
name text,
class int,
inter text,
power int,
PRIMARY KEY (name, class, inter)
) WITH CLUSTERING ORDER BY (class DESC, inter ASC);
INSERT INTO table1 (name, class, inter, power) VALUES ('R1',1, 'int1',13);
INSERT INTO table1 (name, class, inter, power) VALUES ('R1',2, 'int1',18);
INSERT INTO table1 (name, class, inter, power) VALUES ('R1',3, 'int1',37);
INSERT INTO table1 (name, class, inter, power) VALUES ('R1',4, 'int1',49);

In version 2.0.11, I used to have a secondary index on inter, that allowed
me to make fast queries on this table:
CREATE INDEX table1_inter ON table1 (inter);
SELECT * FROM table1 where name='R1' AND class>0 AND class<4 AND inter='int1'
ALLOW FILTERING;

While testing on 3.3.0, I get the following message:
*Cluste*
*ring column "inter" cannot be restricted (preceding column "class" is
restricted by a non-EQ relation)*
It seems to only be considered as a key and the index and ALLOW FILTERING
are not taken into account anymore (as it was in 2.0.11).

Further tests confused me, as I found an ugly workaround: If I duplicate
the column inter as a regular column, I can simply query it with the
secondary index and no ALLOW FILTERING. It looks like the behavior I would
anticipate and do not understand why it does not work on inter only because
it is a clustering key.

Any insight highly appreciated!

Julien

RE: all the nost are not reacheable when running massive deletes

[Paco Trujillo]

Hi daemeon

We have check network and it is ok, in fact the nodes are connecting between 
themselves with a dedicated network.

From: daemeon reiydelle [mailto:daeme...@gmail.com]
Sent: maandag 4 april 2016 18:42
To: user@cassandra.apache.org
Subject: Re: all the nost are not reacheable when running massive deletes


Network issues. Could be jumbo frames not consistent or other.

sent from my mobile

sent from my mobile
Daemeon C.M. Reiydelle
USA 415.501.0198
London +44.0.20.8144.9872
On Apr 4, 2016 5:34 AM, "Paco Trujillo" 
> wrote:
Hi everyone

We are having problems with our cluster (7 nodes version 2.0.17) when running 
“massive deletes” on one of the nodes (via cql command line). At the beginning 
everything is fine, but after a while we start getting constant 
NoHostAvailableException using the datastax driver:

Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All 
host(s) tried for query failed (tried: 
/172.31.7.243:9042 
(com.datastax.driver.core.exceptions.DriverException: Timeout while trying to 
acquire available connection (you may want to increase the driver number of 
per-host connections)), /172.31.7.245:9042 
(com.datastax.driver.core.exceptions.DriverException: Timeout while trying to 
acquire available connection (you may want to increase the driver number of 
per-host connections)), /172.31.7.246:9042 
(com.datastax.driver.core.exceptions.DriverException: Timeout while trying to 
acquire available connection (you may want to increase the driver number of 
per-host connections)), /172.31.7.247:9042, 
/172.31.7.232:9042, 
/172.31.7.233:9042, 
/172.31.7.244:9042 [only showing errors of first 3 
hosts, use getErrors() for more details])


All the nodes are running:

UN  172.31.7.244  152.21 GB  256 14.5%  
58abea69-e7ba-4e57-9609-24f3673a7e58  RAC1
UN  172.31.7.245  168.4 GB   256 14.5%  
bc11b4f0-cf96-4ca5-9a3e-33cc2b92a752  RAC1
UN  172.31.7.246  177.71 GB  256 13.7%  
8dc7bb3d-38f7-49b9-b8db-a622cc80346c  RAC1
UN  172.31.7.247  158.57 GB  256 14.1%  
94022081-a563-4042-81ab-75ffe4d13194  RAC1
UN  172.31.7.243  176.83 GB  256 14.6%  
0dda3410-db58-42f2-9351-068bdf68f530  RAC1
UN  172.31.7.233  159 GB 256 13.6%  
01e013fb-2f57-44fb-b3c5-fd89d705bfdd  RAC1
UN  172.31.7.232  166.05 GB  256 15.0%  4d009603-faa9-4add-b3a2-fe24ec16a7c1

but two of them have high cpu load, especially the 232 because I am running a 
lot of deletes using cqlsh in that node.

I know that deletes generate tombstones, but with 7 nodes in the cluster I do 
not think is normal that all the host are not accesible.

We have a replication factor of 3 and for the deletes I am not using any 
consistency (so it is using the default ONE).

I check the nodes which a lot of CPU (near 96%) and th gc activity remains on 
1.6% (using only 3 GB from the 10 which have assigned). But looking at the 
thread pool stats, the mutation stages pending column grows without stop, could 
be that the problem?

I cannot find the reason that originates the timeouts. I already have increased 
the timeouts, but It do not think that is a solution because the timeouts 
indicated another type of error. Anyone have a tip to try to determine where is 
the problem?

Thanks in advance

RE: all the nost are not reacheable when running massive deletes

[Paco Trujillo]

Hi Alain


-  Over use the cluster was one thing which I was thinking about, and I 
have requested two new nodes (anyway it was something already planned). But the 
pattern of nodes with high CPU load is only visible in 1 or two of the nodes, 
the rest are working correctly. That made me think that adding two new nodes 
maybe will not help.


-  Run the deletes at slower at constant path sounds good and 
definitely I will try that. Anyway I have similar errors during the weekly 
repair, even without the deletes running.



-  Our cluster is inhouse one, each machine ois only use as a Cassandra 
node.



-  Logs are quite normal, even when the timeouts start to appear on the 
client.



-  The update of Cassandra is a good point but I am afraid that if I 
start the updates right now the timeouts problems will appear again. During an 
update compactions are executed? If it is not I think is safe to update the 
cluster.

Thanks for your comments

From: Alain RODRIGUEZ [mailto:arodr...@gmail.com]
Sent: maandag 4 april 2016 18:35
To: user@cassandra.apache.org
Subject: Re: all the nost are not reacheable when running massive deletes

Hola Paco,

the mutation stages pending column grows without stop, could be that the problem

CPU (near 96%)

Yes, basically I think you are over using this cluster.

but two of them have high cpu load, especially the 232 because I am running a 
lot of deletes using cqlsh in that node.

Solutions would be to run delete at a slower & constant path, against all the 
nodes, using a balancing policy or adding capacity if all the nodes are facing 
the issue and you can't slow deletes. You should also have a look at iowait and 
steal, see if CPU are really used 100% or masking an other issue. (disk not 
answering fast enough or hardware / shared instance issue). I had some noisy 
neighbours at some point while using Cassandra on AWS.

 I cannot find the reason that originates the timeouts.

I don't see it that weird while being overusing some/all the nodes.

I already have increased the timeouts, but It do not think that is a solution 
because the timeouts indicated another type of error

Any relevant logs in Cassandra nodes (other than dropped mutations INFO)?

7 nodes version 2.0.17

Note: Be aware that this Cassandra version is quite old and no longer 
supported. Plus you might face issues that were solved already. I know that 
upgrading is not straight forward, but 2.0 --> 2.1 brings an amazing set of 
optimisations and some fixes too. You should try it out :-).

C*heers,
---
Alain Rodriguez - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com


2016-04-04 14:33 GMT+02:00 Paco Trujillo 
>:
Hi everyone

We are having problems with our cluster (7 nodes version 2.0.17) when running 
“massive deletes” on one of the nodes (via cql command line). At the beginning 
everything is fine, but after a while we start getting constant 
NoHostAvailableException using the datastax driver:

Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All 
host(s) tried for query failed (tried: 
/172.31.7.243:9042 
(com.datastax.driver.core.exceptions.DriverException: Timeout while trying to 
acquire available connection (you may want to increase the driver number of 
per-host connections)), /172.31.7.245:9042 
(com.datastax.driver.core.exceptions.DriverException: Timeout while trying to 
acquire available connection (you may want to increase the driver number of 
per-host connections)), /172.31.7.246:9042 
(com.datastax.driver.core.exceptions.DriverException: Timeout while trying to 
acquire available connection (you may want to increase the driver number of 
per-host connections)), /172.31.7.247:9042, 
/172.31.7.232:9042, 
/172.31.7.233:9042, 
/172.31.7.244:9042 [only showing errors of first 3 
hosts, use getErrors() for more details])


All the nodes are running:

UN  172.31.7.244  152.21 GB  256 14.5%  
58abea69-e7ba-4e57-9609-24f3673a7e58  RAC1
UN  172.31.7.245  168.4 GB   256 14.5%  
bc11b4f0-cf96-4ca5-9a3e-33cc2b92a752  RAC1
UN  172.31.7.246  177.71 GB  256 13.7%  
8dc7bb3d-38f7-49b9-b8db-a622cc80346c  RAC1
UN  172.31.7.247  158.57 GB  256 14.1%  
94022081-a563-4042-81ab-75ffe4d13194  RAC1
UN  172.31.7.243  176.83 GB  256 14.6%  
0dda3410-db58-42f2-9351-068bdf68f530  RAC1
UN  172.31.7.233  159 GB 256 13.6%  
01e013fb-2f57-44fb-b3c5-fd89d705bfdd  RAC1
UN  172.31.7.232  166.05 GB  256 15.0%  4d009603-faa9-4add-b3a2-fe24ec16a7c1

but two of them have high cpu load, especially the 232 because I am running a 
lot of deletes using cqlsh in that node.

I know that deletes 

cross DC data sync starts without rebuilding nodes on new DC

[Vijay Patil]

Hi,

I have configured new DC as per instructions at below link.
https://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html

After completing step 7.a (which is altering keyspace with desired
replication factor for new DC) data automatically starts syncing from
existing DC to new DC, which is not expected because auto_bootstrap is
false on all nodes (existing as well as new DC).

Any idea why it's happening?
If this is the desired behaviour then what's the purpose of rebuilding each
node on new DC (step 7.b)?

Cassandra version is 2.0.17 on all nodes in both DC's and I am using
GossipingPropertyFileSnitch.

Regards,
Vijay