Re: One big giant cluster or several smaller ones?
I think 1 cluster per large table should be preferred, rather than per application. Example, what if there is a large application that requires several big tables, each many 10s of tera-bytes in size? Is it still recommended to have 1 cluster for that app ? On Fri, Nov 12, 2021 at 2:01 PM Jeff Jirsa wrote: > Oh sorry - a cluster per application makes sense. Sharding within an > application makes sense to avoid very very very large clusters (think: > ~thousand nodes). 1 cluster per app/use case. > > On Fri, Nov 12, 2021 at 1:39 PM S G wrote: > >> Thanks Jeff. >> Any side-effect on the client config from small clusters perspective? >> >> Like several smaller clusters means more CassandraClient objects on the >> client side but I guess number of connections shall remain the same as >> number of physical nodes will most likely remain the same only. So I think >> client side would not see any major issue. >> >> >> On Fri, Nov 12, 2021 at 11:46 AM Jeff Jirsa wrote: >> >>> Most people are better served building multiple clusters and spending >>> their engineering time optimizing for maintaining multiple clusters, vs >>> spending their engineering time learning how to work around the sharp edges >>> that make large shared clusters hard. >>> >>> Large multi-tenant clusters give you less waste and a bit more >>> elasticity (one tenant can burst and use spare capacity that would >>> typically be left for the other tenants). However, one bad use case / table >>> can ruin everything (one bad read that generates GC hits all use cases), >>> and eventually certain mechanisms/subsystems dont scale past certain points >>> (e.g. schema - large schemas and large clusters are much harder than small >>> schemas and small clusters) >>> >>> >>> >>> >>> On Fri, Nov 12, 2021 at 11:31 AM S G wrote: >>> >>>> Hello, >>>> >>>> Is there any case where we would prefer one big giant cluster (with >>>> multiple large tables) over several smaller clusters? >>>> Apart from some management overhead of multiple Cassandra Clients, it >>>> seems several smaller clusters are always better than a big one: >>>> >>>>1. Avoids SPOF for all tables >>>>2. Helps debugging (less noise from all tables in the logs) >>>>3. Traffic spikes on one table do not affect others if they are in >>>>different tables. >>>>4. We can scale tables independently of each other - so colder data >>>>can be in a smaller cluster (more data/node) while hotter data can be >>>> on a >>>>bigger cluster (less data/node) >>>> >>>> >>>> It does not mean that every table should be in its own cluster. >>>> But large ones can be moved to their own dedicated clusters (like those >>>> more than a few terabytes). >>>> And smaller ones can be clubbed together in one or few clusters. >>>> >>>> Please share any recommendations for the above from actual production >>>> experiences. >>>> Thanks for helping ! >>>> >>>>
Re: One big giant cluster or several smaller ones?
Thanks Jeff. Any side-effect on the client config from small clusters perspective? Like several smaller clusters means more CassandraClient objects on the client side but I guess number of connections shall remain the same as number of physical nodes will most likely remain the same only. So I think client side would not see any major issue. On Fri, Nov 12, 2021 at 11:46 AM Jeff Jirsa wrote: > Most people are better served building multiple clusters and spending > their engineering time optimizing for maintaining multiple clusters, vs > spending their engineering time learning how to work around the sharp edges > that make large shared clusters hard. > > Large multi-tenant clusters give you less waste and a bit more elasticity > (one tenant can burst and use spare capacity that would typically be left > for the other tenants). However, one bad use case / table can ruin > everything (one bad read that generates GC hits all use cases), and > eventually certain mechanisms/subsystems dont scale past certain points > (e.g. schema - large schemas and large clusters are much harder than small > schemas and small clusters) > > > > > On Fri, Nov 12, 2021 at 11:31 AM S G wrote: > >> Hello, >> >> Is there any case where we would prefer one big giant cluster (with >> multiple large tables) over several smaller clusters? >> Apart from some management overhead of multiple Cassandra Clients, it >> seems several smaller clusters are always better than a big one: >> >>1. Avoids SPOF for all tables >>2. Helps debugging (less noise from all tables in the logs) >>3. Traffic spikes on one table do not affect others if they are in >>different tables. >>4. We can scale tables independently of each other - so colder data >>can be in a smaller cluster (more data/node) while hotter data can be on a >>bigger cluster (less data/node) >> >> >> It does not mean that every table should be in its own cluster. >> But large ones can be moved to their own dedicated clusters (like those >> more than a few terabytes). >> And smaller ones can be clubbed together in one or few clusters. >> >> Please share any recommendations for the above from actual production >> experiences. >> Thanks for helping ! >> >>
One big giant cluster or several smaller ones?
Hello, Is there any case where we would prefer one big giant cluster (with multiple large tables) over several smaller clusters? Apart from some management overhead of multiple Cassandra Clients, it seems several smaller clusters are always better than a big one: 1. Avoids SPOF for all tables 2. Helps debugging (less noise from all tables in the logs) 3. Traffic spikes on one table do not affect others if they are in different tables. 4. We can scale tables independently of each other - so colder data can be in a smaller cluster (more data/node) while hotter data can be on a bigger cluster (less data/node) It does not mean that every table should be in its own cluster. But large ones can be moved to their own dedicated clusters (like those more than a few terabytes). And smaller ones can be clubbed together in one or few clusters. Please share any recommendations for the above from actual production experiences. Thanks for helping !
How to find traffic profile per client on a Cassandra server?
Hello, We recently faced an issue recently where the read traffic on a big Cassandra cluster shot up several times (think more than 20 times). However, the client team denies sending any huge load and they have their own traffic graphs to prove the same. Assuming the client team's graphs are correct, how do we know the source of traffic ? Slow query logging is enabled, but it only logs queries after a certain threshold, so not very helpful. Secondly, we do not know when the incidence will re-occur. So how do we solve such a problem and put some monitoring in place that shows the source of such huge spikes when it happens next time. Thinking of trying *lsof -i *and *netstat -tn* commands in a per-minute cron on each server but they only show connections from clients, not how many requests in those connections. Any suggestions on how to go about this? Thanks !
Re: Single node slowing down queries in a large cluster
Also, for the percentile based speculative retry, how big of a time-period is used to calculate the percentile? If it is only a few seconds, then the latency will increase very quickly when server performance degrades. But if it is upto a few minutes (or it is configurable), then its percentile will not shoot up in proportion to server's degrading health and hence might be very good to use. Can someone share how big of a time-period is used to calculate the percentile? On Sun, Oct 17, 2021 at 1:59 PM S G wrote: > > "The harder thing to solve is a bad coordinator node slowing down all > reads coordinated by that node" > I think this is the root of the problem and since all nodes act as > coordinator nodes, so it guaranteed that if any 1 node slows down (High GC, > Segment Merging etc), it will slow down 1/N queries in the cluster (N = > ring size). > > Speculative retry seems like a good option (non-percentile based) if it > also mandates the selection of a different server in the retry. > > Is any kind of speculative retry turned on by default ? > > > > On Wed, Oct 13, 2021 at 2:33 PM Jeff Jirsa wrote: > >> Some random notes, not necessarily going to help you, but: >> - You probably have vnodes enable, which means one bad node is PROBABLY a >> replica of almost every other node, so the fanout here is worse than it >> should be, and >> - You probably have speculative retry on the table set to a percentile. >> As the host gets slow, the percentiles change, and speculative retry stop >> being useful, so you end up timing out queries >> >> If you change speculative retry to use the MIN(Xms, p99) syntax, with X >> set on your real workload, you can likely force it to speculate sooner when >> that one host gets sick. >> >> The harder thing to solve is a bad coordinator node slowing down all >> reads coordinated by that node. Retry at the client level to work around >> that tends to be effective. >> >> >> >> On Wed, Oct 13, 2021 at 2:22 PM S G wrote: >> >>> Hello, >>> >>> We have frequently seen that a single bad node running slow can affect >>> the latencies of the entire cluster (especially for queries where the slow >>> node was acting as a coordinator). >>> >>> >>> Is there any suggestion to avoid this behavior? >>> >>> Like something on the client side to not query that bad node or >>> something on the bad node that redirects its query to other healthy >>> coordinators? >>> >>> >>> Thanks, >>> >>> >>>
Re: Single node slowing down queries in a large cluster
"The harder thing to solve is a bad coordinator node slowing down all reads coordinated by that node" I think this is the root of the problem and since all nodes act as coordinator nodes, so it guaranteed that if any 1 node slows down (High GC, Segment Merging etc), it will slow down 1/N queries in the cluster (N = ring size). Speculative retry seems like a good option (non-percentile based) if it also mandates the selection of a different server in the retry. Is any kind of speculative retry turned on by default ? On Wed, Oct 13, 2021 at 2:33 PM Jeff Jirsa wrote: > Some random notes, not necessarily going to help you, but: > - You probably have vnodes enable, which means one bad node is PROBABLY a > replica of almost every other node, so the fanout here is worse than it > should be, and > - You probably have speculative retry on the table set to a percentile. As > the host gets slow, the percentiles change, and speculative retry stop > being useful, so you end up timing out queries > > If you change speculative retry to use the MIN(Xms, p99) syntax, with X > set on your real workload, you can likely force it to speculate sooner when > that one host gets sick. > > The harder thing to solve is a bad coordinator node slowing down all reads > coordinated by that node. Retry at the client level to work around that > tends to be effective. > > > > On Wed, Oct 13, 2021 at 2:22 PM S G wrote: > >> Hello, >> >> We have frequently seen that a single bad node running slow can affect >> the latencies of the entire cluster (especially for queries where the slow >> node was acting as a coordinator). >> >> >> Is there any suggestion to avoid this behavior? >> >> Like something on the client side to not query that bad node or something >> on the bad node that redirects its query to other healthy coordinators? >> >> >> Thanks, >> >> >>
Re: Stop long running queries in Cassandra 3.11.x or Cassandra 4.x
I see. Thanks Jeff ! On Wed, Oct 13, 2021 at 2:25 PM Jeff Jirsa wrote: > Convention in the yaml is default being visible commented out. > > > On Wed, Oct 13, 2021 at 2:17 PM S G wrote: > >> ok, the link given has the value commented, so I was a bit confused. >> But then https://github.com/apache/cassandra/search?q=cross_node_timeout >> shows that default value is indeed true. >> Thanks for the help, >> >> On Wed, Oct 13, 2021 at 11:26 AM Jeff Jirsa wrote: >> >>> The default is true: >>> >>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L1000 >>> >>> There is no equivalent to `alter system kill session`, because it is >>> assumed that any query has a short, finite life in the order of seconds. >>> >>> >>> >>> On Wed, Oct 13, 2021 at 11:10 AM S G wrote: >>> >>>> Hello, >>>> >>>> Does anyone know about the default being turned off for this setting? >>>> It seems like a good one to be turned on - why have replicas process >>>> something for which coordinator has already sent the timeout to client? >>>> >>>> Thanks >>>> >>>> On Tue, Oct 12, 2021 at 11:06 AM S G wrote: >>>> >>>>> Thanks Bowen. >>>>> Any idea why is cross_node_timeout commented out by default? That >>>>> seems like a good option to enable even as per the documentation: >>>>> # If disabled, replicas will assume that requests >>>>> # were forwarded to them instantly by the coordinator, which means that >>>>> # under overload conditions we will waste that much extra time >>>>> processing >>>>> # already-timed-out requests. >>>>> >>>>> Also, taking an example from Oracle kind of RDBMS systems, is there a >>>>> command like the following that can be fired from an external script to >>>>> kill a long running query on each node: >>>>> >>>>> alter system kill session >>>>> >>>>> >>>>> >>>>> >>>>> On Tue, Oct 12, 2021 at 10:49 AM Bowen Song wrote: >>>>> >>>>>> That will depend on whether you have cross_node_timeout enabled. >>>>>> However, I have to point out that set timeout to 15ms is perhaps not a >>>>>> good >>>>>> idea, the JVM GC can easily cause a lots of timeouts. >>>>>> On 12/10/2021 18:20, S G wrote: >>>>>> >>>>>> ok, when a coordinator node sends timeout to the client, does it mean >>>>>> all the replica nodes have stopped processing that specific query too? >>>>>> Or is it just the coordinator node that has stopped waiting for the >>>>>> replicas to return response? >>>>>> >>>>>> On Tue, Oct 12, 2021 at 10:12 AM Jeff Jirsa wrote: >>>>>> >>>>>>> It sends an exception to the client, it doesnt sever the connection. >>>>>>> >>>>>>> >>>>>>> On Tue, Oct 12, 2021 at 10:06 AM S G >>>>>>> wrote: >>>>>>> >>>>>>>> Do the timeout values only kill the connection with the client or >>>>>>>> send error to the client? >>>>>>>> Or do they also kill the corresponding query execution happening on >>>>>>>> the Cassandra servers (co-ordinator, replicas etc) ? >>>>>>>> >>>>>>>> On Tue, Oct 12, 2021 at 10:00 AM Jeff Jirsa >>>>>>>> wrote: >>>>>>>> >>>>>>>>> The read and write timeout values do this today. >>>>>>>>> >>>>>>>>> >>>>>>>>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L920-L943 >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Oct 12, 2021 at 9:53 AM S G >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> Is there a way to stop long running queries in Cassandra >>>>>>>>>> (versions 3.11.x or 4.x) ? >>>>>>>>>> The use-case is to have some kind of a circuit breaker based on >>>>>>>>>> query-time that has exceeded the client's SLAs. >>>>>>>>>> Example: If server response is useless to the client after 10 ms, >>>>>>>>>> then we could >>>>>>>>>> have a *query_killing_timeout* set to 15 ms (where additional 5ms >>>>>>>>>> allows for some buffer). >>>>>>>>>> And when that much time has elapsed, Cassandra will kill the >>>>>>>>>> query execution automatically. >>>>>>>>>> >>>>>>>>>> If this is not possible in Cassandra currently, any chance we can >>>>>>>>>> do it outside of Cassandra, like >>>>>>>>>> a shell script that monitors such long running queries (through >>>>>>>>>> users table etc) and kills the >>>>>>>>>> OS-thread responsible for that query (Looks unsafe though as that >>>>>>>>>> might leave the DB in an inconsistent state) ? >>>>>>>>>> >>>>>>>>>> We are trying this as a proactive measure to safeguard our >>>>>>>>>> clusters from any rogue queries fired accidentally or maliciously. >>>>>>>>>> >>>>>>>>>> Thanks ! >>>>>>>>>> >>>>>>>>>>
Re: Stop long running queries in Cassandra 3.11.x or Cassandra 4.x
ok, the link given has the value commented, so I was a bit confused. But then https://github.com/apache/cassandra/search?q=cross_node_timeout shows that default value is indeed true. Thanks for the help, On Wed, Oct 13, 2021 at 11:26 AM Jeff Jirsa wrote: > The default is true: > > https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L1000 > > There is no equivalent to `alter system kill session`, because it is > assumed that any query has a short, finite life in the order of seconds. > > > > On Wed, Oct 13, 2021 at 11:10 AM S G wrote: > >> Hello, >> >> Does anyone know about the default being turned off for this setting? >> It seems like a good one to be turned on - why have replicas process >> something for which coordinator has already sent the timeout to client? >> >> Thanks >> >> On Tue, Oct 12, 2021 at 11:06 AM S G wrote: >> >>> Thanks Bowen. >>> Any idea why is cross_node_timeout commented out by default? That seems >>> like a good option to enable even as per the documentation: >>> # If disabled, replicas will assume that requests >>> # were forwarded to them instantly by the coordinator, which means that >>> # under overload conditions we will waste that much extra time processing >>> # already-timed-out requests. >>> >>> Also, taking an example from Oracle kind of RDBMS systems, is there a >>> command like the following that can be fired from an external script to >>> kill a long running query on each node: >>> >>> alter system kill session >>> >>> >>> >>> >>> On Tue, Oct 12, 2021 at 10:49 AM Bowen Song wrote: >>> >>>> That will depend on whether you have cross_node_timeout enabled. >>>> However, I have to point out that set timeout to 15ms is perhaps not a good >>>> idea, the JVM GC can easily cause a lots of timeouts. >>>> On 12/10/2021 18:20, S G wrote: >>>> >>>> ok, when a coordinator node sends timeout to the client, does it mean >>>> all the replica nodes have stopped processing that specific query too? >>>> Or is it just the coordinator node that has stopped waiting for the >>>> replicas to return response? >>>> >>>> On Tue, Oct 12, 2021 at 10:12 AM Jeff Jirsa wrote: >>>> >>>>> It sends an exception to the client, it doesnt sever the connection. >>>>> >>>>> >>>>> On Tue, Oct 12, 2021 at 10:06 AM S G >>>>> wrote: >>>>> >>>>>> Do the timeout values only kill the connection with the client or >>>>>> send error to the client? >>>>>> Or do they also kill the corresponding query execution happening on >>>>>> the Cassandra servers (co-ordinator, replicas etc) ? >>>>>> >>>>>> On Tue, Oct 12, 2021 at 10:00 AM Jeff Jirsa wrote: >>>>>> >>>>>>> The read and write timeout values do this today. >>>>>>> >>>>>>> >>>>>>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L920-L943 >>>>>>> >>>>>>> >>>>>>> On Tue, Oct 12, 2021 at 9:53 AM S G >>>>>>> wrote: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> Is there a way to stop long running queries in Cassandra (versions >>>>>>>> 3.11.x or 4.x) ? >>>>>>>> The use-case is to have some kind of a circuit breaker based on >>>>>>>> query-time that has exceeded the client's SLAs. >>>>>>>> Example: If server response is useless to the client after 10 ms, >>>>>>>> then we could >>>>>>>> have a *query_killing_timeout* set to 15 ms (where additional 5ms >>>>>>>> allows for some buffer). >>>>>>>> And when that much time has elapsed, Cassandra will kill the query >>>>>>>> execution automatically. >>>>>>>> >>>>>>>> If this is not possible in Cassandra currently, any chance we can >>>>>>>> do it outside of Cassandra, like >>>>>>>> a shell script that monitors such long running queries (through >>>>>>>> users table etc) and kills the >>>>>>>> OS-thread responsible for that query (Looks unsafe though as that >>>>>>>> might leave the DB in an inconsistent state) ? >>>>>>>> >>>>>>>> We are trying this as a proactive measure to safeguard our clusters >>>>>>>> from any rogue queries fired accidentally or maliciously. >>>>>>>> >>>>>>>> Thanks ! >>>>>>>> >>>>>>>>
Single node slowing down queries in a large cluster
Hello, We have frequently seen that a single bad node running slow can affect the latencies of the entire cluster (especially for queries where the slow node was acting as a coordinator). Is there any suggestion to avoid this behavior? Like something on the client side to not query that bad node or something on the bad node that redirects its query to other healthy coordinators? Thanks,
Re: Stop long running queries in Cassandra 3.11.x or Cassandra 4.x
Hello, Does anyone know about the default being turned off for this setting? It seems like a good one to be turned on - why have replicas process something for which coordinator has already sent the timeout to client? Thanks On Tue, Oct 12, 2021 at 11:06 AM S G wrote: > Thanks Bowen. > Any idea why is cross_node_timeout commented out by default? That seems > like a good option to enable even as per the documentation: > # If disabled, replicas will assume that requests > # were forwarded to them instantly by the coordinator, which means that > # under overload conditions we will waste that much extra time processing > # already-timed-out requests. > > Also, taking an example from Oracle kind of RDBMS systems, is there a > command like the following that can be fired from an external script to > kill a long running query on each node: > > alter system kill session > > > > > On Tue, Oct 12, 2021 at 10:49 AM Bowen Song wrote: > >> That will depend on whether you have cross_node_timeout enabled. However, >> I have to point out that set timeout to 15ms is perhaps not a good idea, >> the JVM GC can easily cause a lots of timeouts. >> On 12/10/2021 18:20, S G wrote: >> >> ok, when a coordinator node sends timeout to the client, does it mean all >> the replica nodes have stopped processing that specific query too? >> Or is it just the coordinator node that has stopped waiting for the >> replicas to return response? >> >> On Tue, Oct 12, 2021 at 10:12 AM Jeff Jirsa wrote: >> >>> It sends an exception to the client, it doesnt sever the connection. >>> >>> >>> On Tue, Oct 12, 2021 at 10:06 AM S G wrote: >>> >>>> Do the timeout values only kill the connection with the client or send >>>> error to the client? >>>> Or do they also kill the corresponding query execution happening on the >>>> Cassandra servers (co-ordinator, replicas etc) ? >>>> >>>> On Tue, Oct 12, 2021 at 10:00 AM Jeff Jirsa wrote: >>>> >>>>> The read and write timeout values do this today. >>>>> >>>>> >>>>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L920-L943 >>>>> >>>>> >>>>> On Tue, Oct 12, 2021 at 9:53 AM S G wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> Is there a way to stop long running queries in Cassandra (versions >>>>>> 3.11.x or 4.x) ? >>>>>> The use-case is to have some kind of a circuit breaker based on >>>>>> query-time that has exceeded the client's SLAs. >>>>>> Example: If server response is useless to the client after 10 ms, >>>>>> then we could >>>>>> have a *query_killing_timeout* set to 15 ms (where additional 5ms >>>>>> allows for some buffer). >>>>>> And when that much time has elapsed, Cassandra will kill the query >>>>>> execution automatically. >>>>>> >>>>>> If this is not possible in Cassandra currently, any chance we can do >>>>>> it outside of Cassandra, like >>>>>> a shell script that monitors such long running queries (through users >>>>>> table etc) and kills the >>>>>> OS-thread responsible for that query (Looks unsafe though as that >>>>>> might leave the DB in an inconsistent state) ? >>>>>> >>>>>> We are trying this as a proactive measure to safeguard our clusters >>>>>> from any rogue queries fired accidentally or maliciously. >>>>>> >>>>>> Thanks ! >>>>>> >>>>>>
Re: Stop long running queries in Cassandra 3.11.x or Cassandra 4.x
Thanks Bowen. Any idea why is cross_node_timeout commented out by default? That seems like a good option to enable even as per the documentation: # If disabled, replicas will assume that requests # were forwarded to them instantly by the coordinator, which means that # under overload conditions we will waste that much extra time processing # already-timed-out requests. Also, taking an example from Oracle kind of RDBMS systems, is there a command like the following that can be fired from an external script to kill a long running query on each node: alter system kill session On Tue, Oct 12, 2021 at 10:49 AM Bowen Song wrote: > That will depend on whether you have cross_node_timeout enabled. However, > I have to point out that set timeout to 15ms is perhaps not a good idea, > the JVM GC can easily cause a lots of timeouts. > On 12/10/2021 18:20, S G wrote: > > ok, when a coordinator node sends timeout to the client, does it mean all > the replica nodes have stopped processing that specific query too? > Or is it just the coordinator node that has stopped waiting for the > replicas to return response? > > On Tue, Oct 12, 2021 at 10:12 AM Jeff Jirsa wrote: > >> It sends an exception to the client, it doesnt sever the connection. >> >> >> On Tue, Oct 12, 2021 at 10:06 AM S G wrote: >> >>> Do the timeout values only kill the connection with the client or send >>> error to the client? >>> Or do they also kill the corresponding query execution happening on the >>> Cassandra servers (co-ordinator, replicas etc) ? >>> >>> On Tue, Oct 12, 2021 at 10:00 AM Jeff Jirsa wrote: >>> >>>> The read and write timeout values do this today. >>>> >>>> >>>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L920-L943 >>>> >>>> >>>> On Tue, Oct 12, 2021 at 9:53 AM S G wrote: >>>> >>>>> Hello, >>>>> >>>>> Is there a way to stop long running queries in Cassandra (versions >>>>> 3.11.x or 4.x) ? >>>>> The use-case is to have some kind of a circuit breaker based on >>>>> query-time that has exceeded the client's SLAs. >>>>> Example: If server response is useless to the client after 10 ms, then >>>>> we could >>>>> have a *query_killing_timeout* set to 15 ms (where additional 5ms >>>>> allows for some buffer). >>>>> And when that much time has elapsed, Cassandra will kill the query >>>>> execution automatically. >>>>> >>>>> If this is not possible in Cassandra currently, any chance we can do >>>>> it outside of Cassandra, like >>>>> a shell script that monitors such long running queries (through users >>>>> table etc) and kills the >>>>> OS-thread responsible for that query (Looks unsafe though as that >>>>> might leave the DB in an inconsistent state) ? >>>>> >>>>> We are trying this as a proactive measure to safeguard our clusters >>>>> from any rogue queries fired accidentally or maliciously. >>>>> >>>>> Thanks ! >>>>> >>>>>
Re: Stop long running queries in Cassandra 3.11.x or Cassandra 4.x
ok, when a coordinator node sends timeout to the client, does it mean all the replica nodes have stopped processing that specific query too? Or is it just the coordinator node that has stopped waiting for the replicas to return response? On Tue, Oct 12, 2021 at 10:12 AM Jeff Jirsa wrote: > It sends an exception to the client, it doesnt sever the connection. > > > On Tue, Oct 12, 2021 at 10:06 AM S G wrote: > >> Do the timeout values only kill the connection with the client or send >> error to the client? >> Or do they also kill the corresponding query execution happening on the >> Cassandra servers (co-ordinator, replicas etc) ? >> >> On Tue, Oct 12, 2021 at 10:00 AM Jeff Jirsa wrote: >> >>> The read and write timeout values do this today. >>> >>> >>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L920-L943 >>> >>> >>> On Tue, Oct 12, 2021 at 9:53 AM S G wrote: >>> >>>> Hello, >>>> >>>> Is there a way to stop long running queries in Cassandra (versions >>>> 3.11.x or 4.x) ? >>>> The use-case is to have some kind of a circuit breaker based on >>>> query-time that has exceeded the client's SLAs. >>>> Example: If server response is useless to the client after 10 ms, then >>>> we could >>>> have a *query_killing_timeout* set to 15 ms (where additional 5ms >>>> allows for some buffer). >>>> And when that much time has elapsed, Cassandra will kill the query >>>> execution automatically. >>>> >>>> If this is not possible in Cassandra currently, any chance we can do it >>>> outside of Cassandra, like >>>> a shell script that monitors such long running queries (through users >>>> table etc) and kills the >>>> OS-thread responsible for that query (Looks unsafe though as that might >>>> leave the DB in an inconsistent state) ? >>>> >>>> We are trying this as a proactive measure to safeguard our clusters >>>> from any rogue queries fired accidentally or maliciously. >>>> >>>> Thanks ! >>>> >>>>
Re: Stop long running queries in Cassandra 3.11.x or Cassandra 4.x
Do the timeout values only kill the connection with the client or send error to the client? Or do they also kill the corresponding query execution happening on the Cassandra servers (co-ordinator, replicas etc) ? On Tue, Oct 12, 2021 at 10:00 AM Jeff Jirsa wrote: > The read and write timeout values do this today. > > > https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L920-L943 > > > On Tue, Oct 12, 2021 at 9:53 AM S G wrote: > >> Hello, >> >> Is there a way to stop long running queries in Cassandra (versions 3.11.x >> or 4.x) ? >> The use-case is to have some kind of a circuit breaker based on >> query-time that has exceeded the client's SLAs. >> Example: If server response is useless to the client after 10 ms, then we >> could >> have a *query_killing_timeout* set to 15 ms (where additional 5ms allows >> for some buffer). >> And when that much time has elapsed, Cassandra will kill the query >> execution automatically. >> >> If this is not possible in Cassandra currently, any chance we can do it >> outside of Cassandra, like >> a shell script that monitors such long running queries (through users >> table etc) and kills the >> OS-thread responsible for that query (Looks unsafe though as that might >> leave the DB in an inconsistent state) ? >> >> We are trying this as a proactive measure to safeguard our clusters from >> any rogue queries fired accidentally or maliciously. >> >> Thanks ! >> >>
Stop long running queries in Cassandra 3.11.x or Cassandra 4.x
Hello, Is there a way to stop long running queries in Cassandra (versions 3.11.x or 4.x) ? The use-case is to have some kind of a circuit breaker based on query-time that has exceeded the client's SLAs. Example: If server response is useless to the client after 10 ms, then we could have a *query_killing_timeout* set to 15 ms (where additional 5ms allows for some buffer). And when that much time has elapsed, Cassandra will kill the query execution automatically. If this is not possible in Cassandra currently, any chance we can do it outside of Cassandra, like a shell script that monitors such long running queries (through users table etc) and kills the OS-thread responsible for that query (Looks unsafe though as that might leave the DB in an inconsistent state) ? We are trying this as a proactive measure to safeguard our clusters from any rogue queries fired accidentally or maliciously. Thanks !
Re: Does Open-Source Cassandra 4 support mutual-TLS ?
Thanks Dinesh. That is ok. Having mutual TLS ensures that the clients authenticate themselves by certificates too. The other authentication of static username/password adds the next layer of authentication. That ways a hacker now needs two keys (certificate and password) to connect to the cluster. On Tue, Sep 21, 2021 at 3:16 PM Dinesh Joshi wrote: > It sort of supports it. You still need to send in the username/password > credentials along with the client certificate to authenticate. Cassandra > will not derive the identity purely from the client certificate. > > Dinesh > > > On Sep 21, 2021, at 11:59 AM, S G wrote: > > > > Hello, > > > > Does anyone know if opensource Cassandra support mutual-TLS ? > > The documentation doesn't conclusively deny or accept the support for > the same. > > > > Thanks ! > >
Does Open-Source Cassandra 4 support mutual-TLS ?
Hello, Does anyone know if opensource Cassandra support mutual-TLS ? The documentation doesn't conclusively deny or accept the support for the same. Thanks !
How many coordinator nodes in the cluster at a given time?
Hello, As mentioned in https://cassandra.apache.org/_/cassandra-basics.html, can someone clarify how many coordinator nodes are there in the cluster at any given time? 1. Is there a dedicated *single* coordinator node in the whole cluster that changes from time to time? 2. Or any node becomes a coordinator node when the client-request lands on it? I am assuming it is the latter because the former will make the coordinator node a SPOF. As an example, if we have a 100 node cluster and there are no hot partitions etc, then is it fair to say that at any given time, generally speaking, each node will be acting as a coordinator node for an equal number of requests? So for each request, there is a single coordinator node. But for all requests at a given time, the coordinator nodes are equally distributed among the cluster. Assumption: The client is intelligent enough and not sending traffic to just a single node. Thanks in advance for clarifying !
DataStax: What codec to use for converting bigint to Long?
Hi, I have a bigint field in my Cassandra that I want to convert to a long value. However, I am not sure how to specify the same in my DataStax-mapper-entity-class. Here is the code: @PartitionKey(1) @Column(name="phone", codec=TypeCodec.) private Double phoneNumber; Can someone please tell what to use in the Column annotation? Thanks for the help, SG
Re: Weird error: InvalidQueryException: unconfigured table table2
Thanks, got it working now :) Just wish that an error like: "Table x not found in keyspace y" would have been much better than: "Table x not configured". On Sat, Mar 25, 2017 at 6:13 AM, Arvydas Jonusonis < arvydas.jonuso...@gmail.com> wrote: > Make sure to prefix the table with the keyspace. > On Sat, Mar 25, 2017 at 13:28 Anuj Wadehra <anujw_2...@yahoo.co.in> wrote: > >> Ensure that all the nodes are on same schema version such that table2 >> schema is replicated properly on all the nodes. >> >> Thanks >> Anuj >> >> Sent from Yahoo Mail on Android >> <https://overview.mail.yahoo.com/mobile/?.src=Android> >> >> On Sat, Mar 25, 2017 at 3:19 AM, S G >> <sg.online.em...@gmail.com> wrote: >> Hi, >> >> I have a keyspace with two tables. >> >> I run a different query for each table: >> >> Table 1: >> Select * from table1 where id = ? >> >> Table 2: >> Select * from table2 where id1 = ? and id = ? >> >> >> My code using datastax fires above two queries one after the other. >> While it never fails for table 1, it never succeeds for table 2 >> And gives an error: >> >> >> com.datastax.driver.core.exceptions.InvalidQueryException: unconfigured >> table table2 >> at com.datastax.driver.core.Responses$Error.asException( >> Responses.java:136) >> at com.datastax.driver.core.DefaultResultSetFuture.onSet( >> DefaultResultSetFuture.java:179) >> at com.datastax.driver.core.RequestHandler.setFinalResult( >> RequestHandler.java:177) >> at com.datastax.driver.core.RequestHandler.access$2500( >> RequestHandler.java:46) >> at com.datastax.driver.core.RequestHandler$SpeculativeExecution. >> setFinalResult(RequestHandler.java:799) >> at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet( >> RequestHandler.java:633) >> at com.datastax.driver.core.Connection$Dispatcher. >> channelRead0(Connection.java:1070) >> at com.datastax.driver.core.Connection$Dispatcher. >> channelRead0(Connection.java:993) >> at io.netty.channel.SimpleChannelInboundHandler.channelRead( >> SimpleChannelInboundHandler.java:105) >> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead( >> AbstractChannelHandlerContext.java:342) >> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead( >> AbstractChannelHandlerContext.java:328) >> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( >> AbstractChannelHandlerContext.java:321) >> at io.netty.handler.timeout.IdleStateHandler.channelRead( >> IdleStateHandler.java:266) >> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead( >> AbstractChannelHandlerContext.java:342) >> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead( >> AbstractChannelHandlerContext.java:328) >> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( >> AbstractChannelHandlerContext.java:321) >> at io.netty.handler.codec.MessageToMessageDecoder.channelRead( >> MessageToMessageDecoder.java:102) >> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead( >> AbstractChannelHandlerContext.java:342) >> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead( >> AbstractChannelHandlerContext.java:328) >> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( >> AbstractChannelHandlerContext.java:321) >> at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead( >> ByteToMessageDecoder.java:293) >> at io.netty.handler.codec.ByteToMessageDecoder.channelRead( >> ByteToMessageDecoder.java:267) >> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead( >> AbstractChannelHandlerContext.java:342) >> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead( >> AbstractChannelHandlerContext.java:328) >> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( >> AbstractChannelHandlerContext.java:321) >> at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead( >> DefaultChannelPipeline.java:1280) >> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead( >> AbstractChannelHandlerContext.java:342) >> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead( >> AbstractChannelHandlerContext.java:328) >> at io.netty.channel.DefaultChannelPipeline.fireChannelRead( >> DefaultChannelPipeline.java:890) >> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read( >> AbstractNioByteChannel.java:131) >> at io.netty.channel.nio.NioEventLoop.processSelectedKey( >> NioEventLoop.java:564) >> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized( >> NioEventLoop.java:505) >> at io.netty.channel.nio.NioEventLoop.processSelectedKeys( >> NioEventLoop.java:419) >> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:391) >> >> Any idea what might be wrong? >> >> I have confirmed that all table-names and columns names are lowercase. >> Datastax java version tried : 3.1.2 and 3.1.4 >> Cassandra version: 3.10 >> >> >> Thanks >> SG >> >> >>
Using datastax driver, how can I read a non-primitive column as a JSON string?
Hi,
I have several non-primitive columns in my cassandra tables.
Some of them are user-defined-types UDTs.
While querying them through datastax driver, I want to convert such UDTs
into JSON values.
More specifically, I want to get JSON string for the value object below:
Row row = itr.next();
ColumnDefinitions cds = row.getColumnDefinitions();
cds.asList().forEach((ColumnDefinitions.Definition cd) -> {
String name = cd.getName();
Object value = row.getObject(name);
}
I have gone through
http://docs.datastax.com/en/developer/java-driver/3.1/manual/custom_codecs/
But I do not want to add a codec for every UDT I have.
Can the driver somehow return me direct JSON without explicit meddling with
codecs and all?
Thanks
SG
Re: How can I scale my read rate?
Forgot to mention that this vmstat picture is for the client-cluster reading from Cassandra. On Sat, Mar 18, 2017 at 6:47 PM, S G <sg.online.em...@gmail.com> wrote: > ok, I gave the executeAsync() a try. > Good part is that it was really easy to write the code for that. > Bad part is that it did not had a huge effect on my throughput - I gained > about 5% increase in throughput. > I suspect it is so because my queries are all get-by-primary-key queries > and were anyways completing in less than 2 milliseconds. > So there was not much wait to begin with. > > > Here is my code: > > String getByKeyQueryStr = "Select * from fooTable where key = " + key; > //ResultSet result = session.execute(getByKeyQueryStr); // Previous code > ResultSetFuture future = session.executeAsync(getByKeyQueryStr); > FutureCallback callback = new MyFutureCallback(); > executor = MoreExecutors.sameThreadExecutor(); > //executor = Executors.newFixedThreadPool(3); // Tried this too, no effect > //executor = Executors.newFixedThreadPool(10); // Tried this too, no > effect > Futures.addCallback(future, callback, executor); > > Can I improve the above code in some way? > Are there any JMX metrics that can tell me what's going on? > > From the vmstat command, I see that CPU idle time is about 70% even though > I am running about 60 threads per VM > Total 20 client-VMs with 8 cores each are querying a Cassandra cluster > with 16 VMs, 8-core each too. > > > > > > > Thanks > SG > > > On Sat, Mar 18, 2017 at 5:38 PM, S G <sg.online.em...@gmail.com> wrote: > >> Thanks. It seems that you guys have found executeAsync to yield good >> results. >> I want to share my understanding how this could benefit performance and >> some validation from the group will be awesome. >> >> I will call executeAsync() each time I want to get by primary-key. >> That ways, my client thread is not blocked anymore and I can submit a lot >> more requests per unit time. >> The async requests get piled on the underlying Netty I/O thread which >> ensures that it is always busy all the time. >> Earlier, the Netty I/O thread would have wasted some cycles when the >> sync-execute method was processing the results. >> And earlier, the client thread would also have wasted some cycles waiting >> for netty-thread to complete. >> >> With executeAsync(), none of them is waiting. >> Only thing to ensure is that the Netty thread's queue does not grow >> indefinitely. >> >> If the above theory is correct, then it sounds like a really good thing >> to try. >> If not, please do share some more details. >> >> >> >> >> On Sat, Mar 18, 2017 at 2:00 PM, <j.kes...@enercast.de> wrote: >> >>> +1 for executeAsync – had a long time to argue that it’s not bad as with >>> good old rdbms. >>> >>> >>> >>> >>> >>> >>> >>> Gesendet von meinem Windows 10 Phone >>> >>> >>> >>> *Von: *Arvydas Jonusonis <arvydas.jonuso...@gmail.com> >>> *Gesendet: *Samstag, 18. März 2017 19:08 >>> *An: *user@cassandra.apache.org >>> *Betreff: *Re: How can I scale my read rate? >>> >>> >>> >>> ..then you're not taking advantage of request pipelining. Use >>> executeAsync - this will increase your throughput for sure. >>> >>> >>> >>> http://www.datastax.com/dev/blog/java-driver-async-queries >>> >>> >>> >>> >>> >>> On Sat, Mar 18, 2017 at 08:00 S G <sg.online.em...@gmail.com> wrote: >>> >>> I have enabled JMX but not sure what metrics to look for - they are way >>> too many of them. >>> >>> I am using session.execute(...) >>> >>> >>> >>> >>> >>> On Fri, Mar 17, 2017 at 2:07 PM, Arvydas Jonusonis < >>> arvydas.jonuso...@gmail.com> wrote: >>> >>> It would be interesting to see some of the driver metrics (in your >>> stress test tool) - if you enable JMX, they should be exposed by default. >>> >>> Also, are you using session.execute(..) or session.executeAsync(..) ? >>> >>> >>> >>> >>> >> >
Re: How can I scale my read rate?
ok, I gave the executeAsync() a try. Good part is that it was really easy to write the code for that. Bad part is that it did not had a huge effect on my throughput - I gained about 5% increase in throughput. I suspect it is so because my queries are all get-by-primary-key queries and were anyways completing in less than 2 milliseconds. So there was not much wait to begin with. Here is my code: String getByKeyQueryStr = "Select * from fooTable where key = " + key; //ResultSet result = session.execute(getByKeyQueryStr); // Previous code ResultSetFuture future = session.executeAsync(getByKeyQueryStr); FutureCallback callback = new MyFutureCallback(); executor = MoreExecutors.sameThreadExecutor(); //executor = Executors.newFixedThreadPool(3); // Tried this too, no effect //executor = Executors.newFixedThreadPool(10); // Tried this too, no effect Futures.addCallback(future, callback, executor); Can I improve the above code in some way? Are there any JMX metrics that can tell me what's going on? >From the vmstat command, I see that CPU idle time is about 70% even though I am running about 60 threads per VM Total 20 client-VMs with 8 cores each are querying a Cassandra cluster with 16 VMs, 8-core each too. Thanks SG On Sat, Mar 18, 2017 at 5:38 PM, S G <sg.online.em...@gmail.com> wrote: > Thanks. It seems that you guys have found executeAsync to yield good > results. > I want to share my understanding how this could benefit performance and > some validation from the group will be awesome. > > I will call executeAsync() each time I want to get by primary-key. > That ways, my client thread is not blocked anymore and I can submit a lot > more requests per unit time. > The async requests get piled on the underlying Netty I/O thread which > ensures that it is always busy all the time. > Earlier, the Netty I/O thread would have wasted some cycles when the > sync-execute method was processing the results. > And earlier, the client thread would also have wasted some cycles waiting > for netty-thread to complete. > > With executeAsync(), none of them is waiting. > Only thing to ensure is that the Netty thread's queue does not grow > indefinitely. > > If the above theory is correct, then it sounds like a really good thing to > try. > If not, please do share some more details. > > > > > On Sat, Mar 18, 2017 at 2:00 PM, <j.kes...@enercast.de> wrote: > >> +1 for executeAsync – had a long time to argue that it’s not bad as with >> good old rdbms. >> >> >> >> >> >> >> >> Gesendet von meinem Windows 10 Phone >> >> >> >> *Von: *Arvydas Jonusonis <arvydas.jonuso...@gmail.com> >> *Gesendet: *Samstag, 18. März 2017 19:08 >> *An: *user@cassandra.apache.org >> *Betreff: *Re: How can I scale my read rate? >> >> >> >> ..then you're not taking advantage of request pipelining. Use >> executeAsync - this will increase your throughput for sure. >> >> >> >> http://www.datastax.com/dev/blog/java-driver-async-queries >> >> >> >> >> >> On Sat, Mar 18, 2017 at 08:00 S G <sg.online.em...@gmail.com> wrote: >> >> I have enabled JMX but not sure what metrics to look for - they are way >> too many of them. >> >> I am using session.execute(...) >> >> >> >> >> >> On Fri, Mar 17, 2017 at 2:07 PM, Arvydas Jonusonis < >> arvydas.jonuso...@gmail.com> wrote: >> >> It would be interesting to see some of the driver metrics (in your stress >> test tool) - if you enable JMX, they should be exposed by default. >> >> Also, are you using session.execute(..) or session.executeAsync(..) ? >> >> >> >> >> >
Re: How can I scale my read rate?
Thanks. It seems that you guys have found executeAsync to yield good results. I want to share my understanding how this could benefit performance and some validation from the group will be awesome. I will call executeAsync() each time I want to get by primary-key. That ways, my client thread is not blocked anymore and I can submit a lot more requests per unit time. The async requests get piled on the underlying Netty I/O thread which ensures that it is always busy all the time. Earlier, the Netty I/O thread would have wasted some cycles when the sync-execute method was processing the results. And earlier, the client thread would also have wasted some cycles waiting for netty-thread to complete. With executeAsync(), none of them is waiting. Only thing to ensure is that the Netty thread's queue does not grow indefinitely. If the above theory is correct, then it sounds like a really good thing to try. If not, please do share some more details. On Sat, Mar 18, 2017 at 2:00 PM, <j.kes...@enercast.de> wrote: > +1 for executeAsync – had a long time to argue that it’s not bad as with > good old rdbms. > > > > > > > > Gesendet von meinem Windows 10 Phone > > > > *Von: *Arvydas Jonusonis <arvydas.jonuso...@gmail.com> > *Gesendet: *Samstag, 18. März 2017 19:08 > *An: *user@cassandra.apache.org > *Betreff: *Re: How can I scale my read rate? > > > > ..then you're not taking advantage of request pipelining. Use executeAsync > - this will increase your throughput for sure. > > > > http://www.datastax.com/dev/blog/java-driver-async-queries > > > > > > On Sat, Mar 18, 2017 at 08:00 S G <sg.online.em...@gmail.com> wrote: > > I have enabled JMX but not sure what metrics to look for - they are way > too many of them. > > I am using session.execute(...) > > > > > > On Fri, Mar 17, 2017 at 2:07 PM, Arvydas Jonusonis < > arvydas.jonuso...@gmail.com> wrote: > > It would be interesting to see some of the driver metrics (in your stress > test tool) - if you enable JMX, they should be exposed by default. > > Also, are you using session.execute(..) or session.executeAsync(..) ? > > > > >
