Re: Is it possible to achieve "sticky" request routing?
John and Steve, We use QUORUM consistency for both READS and WRITES. So we won't have the problem of having to pick the right server. The real reason we have this requirement is because we run in a sometimes overloaded virtualized environment that causes the servers to have non-trivial time drifts (tens of milliseconds or a few seconds, even with NTP, which catches up slowly). This particular client was seeing some updates being dropped silently by Cassandra when it hit a server with an older local timestamp. They were relying on server side timestamp generation. So they were exploring "sticky" routing as an option since the likelihood of having monotonically increasing timestamps is higher if the client's requests always go to a single server. They are aware of the problem of disconnecting and reconnecting to a new server and have an application level solution for this. They are also testing using client side timestamps as well. We're just trying to explore all our options and their pros and cons. Thanks! On Tue, Apr 5, 2016 at 1:31 PM, Jonathan Haddad <j...@jonhaddad.com> wrote: > Yep - Steve hit the nail on the head. The odds of hitting the right > server with "sticky routing" goes down as your cluster size increases. You > end up adding extra network hops instead of using token aware routing. > > Unless you're trying to do a coordinator tier (and you're not, according > to your original post), this is a pretty bad idea and I'd advise you to > push back on that requirement. > > On Tue, Apr 5, 2016 at 12:47 PM Steve Robenalt <sroben...@highwire.org> > wrote: > >> Aside from Jon's "why" question, I would point out that this only really >> works because you are running a 3 node cluster with RF=3. If your cluster >> is going to grow, you can't guarantee that any one server would have all >> records. I'd be pretty hesitant to put an invisible constraint like that on >> a cluster unless you're pretty sure it'll only ever be 3 nodes. >> >> On Tue, Apr 5, 2016 at 9:34 AM, Jonathan Haddad <j...@jonhaddad.com> >> wrote: >> >>> Why is this a requirement? Honestly I don't know why you would do this. >>> >>> >>> On Sat, Apr 2, 2016 at 8:06 PM Mukil Kesavan <weirdbluelig...@gmail.com> >>> wrote: >>> >>>> Hello, >>>> >>>> We currently have 3 Cassandra servers running in a single datacenter >>>> with a replication factor of 3 for our keyspace. We also use the >>>> SimpleSnitch wiith DynamicSnitching enabled by default. Our load balancing >>>> policy is TokenAwareLoadBalancingPolicy with RoundRobinPolicy as the child. >>>> This overall configuration results in our client requests spreading equally >>>> across our 3 servers. >>>> >>>> However, we have a new requirement where we need to restrict a client's >>>> requests to a single server and only go to the other servers on failure of >>>> the previous server. This particular use case does not have high request >>>> traffic. >>>> >>>> Looking at the documentation the options we have seem to be: >>>> >>>> 1. Play with the snitching (e.g. place each server into its own DC or >>>> Rack) to ensure that requests always go to one server and failover to the >>>> others if required. I understand that this may also affect replica >>>> placement and we may need to run nodetool repair. So this is not our most >>>> preferred option. >>>> >>>> 2. Write a new load balancing policy that also uses the >>>> HostStateListener for tracking host up and down messages, that essentially >>>> accomplishes "sticky" request routing with failover to other nodes. >>>> >>>> Is option 2 the only clean way of accomplishing our requirement? >>>> >>>> Thanks, >>>> Micky >>>> >>> >> >> >> -- >> Steve Robenalt >> Software Architect >> sroben...@highwire.org <bza...@highwire.org> >> (office/cell): 916-505-1785 >> >> HighWire Press, Inc. >> 425 Broadway St, Redwood City, CA 94063 >> www.highwire.org >> >> Technology for Scholarly Communication >> >
Is it possible to achieve "sticky" request routing?
Hello, We currently have 3 Cassandra servers running in a single datacenter with a replication factor of 3 for our keyspace. We also use the SimpleSnitch wiith DynamicSnitching enabled by default. Our load balancing policy is TokenAwareLoadBalancingPolicy with RoundRobinPolicy as the child. This overall configuration results in our client requests spreading equally across our 3 servers. However, we have a new requirement where we need to restrict a client's requests to a single server and only go to the other servers on failure of the previous server. This particular use case does not have high request traffic. Looking at the documentation the options we have seem to be: 1. Play with the snitching (e.g. place each server into its own DC or Rack) to ensure that requests always go to one server and failover to the others if required. I understand that this may also affect replica placement and we may need to run nodetool repair. So this is not our most preferred option. 2. Write a new load balancing policy that also uses the HostStateListener for tracking host up and down messages, that essentially accomplishes "sticky" request routing with failover to other nodes. Is option 2 the only clean way of accomplishing our requirement? Thanks, Micky
Re: NTP Synchronization Setup Changes
Thanks for all the useful comments. On Thu, Mar 31, 2016 at 11:22 PM, Brice Dutheil <brice.duth...@gmail.com> wrote: > Hi another tip, make sure the OS doesn't come with pre-configured NTP > synchronisation services. We had a proper NTP setup, but we missed a > service that came with CentOS that synced to a low stratum NTP server. > > -- Brice > > > > > On Thu, Mar 31, 2016 at 10:00 AM -0700, "Eric Evans" <eev...@wikimedia.org > > wrote: > > On Wed, Mar 30, 2016 at 8:07 PM, Mukil Kesavan wrote: >> > Are there any issues if this causes a huge time correction on the cassandra >> > cluster? I know that NTP gradually corrects the time on all the servers. I >> > just wanted to understand if there were any corner cases that will cause us >> > to lose data/schema updates when this happens. In particular, we seem to be >> > having some issues around missing secondary indices at the moment (not all >> > but some). >> >> As a thought experiment, imagine every scenario where it matters to >> have one write occur after another (an update followed by a delete is >> a good example). Now imagine having your clock yanked backward to >> correct for drift between the first such operation and the second. >> >> I would strongly recommend you come up with a stable NTP setup. >> >> >> -- >> Eric evanseev...@wikimedia.org >> >>
NTP Synchronization Setup Changes
Hi, We run a 3 server cassandra cluster that is initially NTP synced to a single physical server over LAN. This server does not have connectivity to the internet for a few hours to sometimes even days. In this state we perform some schema operations and reads/writes with QUORUM consistency. Later on, the physical server has connectivity to the internet and we synchronize its time to an external NTP server on the internet. Are there any issues if this causes a huge time correction on the cassandra cluster? I know that NTP gradually corrects the time on all the servers. I just wanted to understand if there were any corner cases that will cause us to lose data/schema updates when this happens. In particular, we seem to be having some issues around missing secondary indices at the moment (not all but some). Also, for our situation where we have to work with cassandra for a while without internet connectivity, what is the preferred NTP architecture/steps that have worked for you in the field? Thanks, Micky
Are write-write conflicts possible in Cassandra atomic batches when updating multiple tables?
Hello, The documentation on atomic batches says: "For example, there is no batch isolation. Clients are able to read the first updated rows from the batch, while other rows are still being updated on the server." However does a write-write conflict scenario arise when I'm updating a column in two different tables as part of a batch? Reference: https://en.wikipedia.org/wiki/Write%E2%80%93write_conflict For example: Batch 1: Table1.column = x, Table2.column = x Batch 2: Table1.column = y, Table2.column = y Result (write-write conflict case): Table1.column = x, Table2.column = y Thanks!
