Re: Lots of simultaneous connections?

2017-12-17 Thread kurt greaves
Yep. With those kind of numbers you're likely overwhelming the cluster with connections. You be better off if you can configure it to have either 1 connection per machine, or at least 1 connection per test. Creating lots of connections is definitely not a good idea, and remember that each

Re: Incremental repairs after a migration?

2017-11-11 Thread kurt greaves
you can get away with loading from only one node if you're positive all data is consistent. A repair prior to loading should be enough, but if that doesn't work just load from all nodes. On 11 Nov. 2017 23:15, "Brice Figureau" <brice+cassan...@daysofwonder.com> wrote: > On 1

Re: Incremental repairs after a migration?

2017-11-10 Thread kurt greaves
If everything goes smoothly the next incremental should cut it, but a full repair post load is probably a good idea anyway. Make sure you sstableload every sstable from every node if you want to keep consistency.

Re: Cassandra open port native_transport_port after a random time

2017-11-10 Thread kurt greaves
Can you upload some logs from startup? Obviously getting stuck starting but it'd be nice to know where it got up to. On 11 Nov. 2017 04:49, "Octavian Rinciog" wrote: > Hello! > > I'm running a cluster of 10 Cassandra 3.10 and I saw a very strange > behavior: after

Re: Warning on node decommision

2017-11-20 Thread kurt greaves
Should be fine, unless it keeps happening then it's a problem. Means exactly what it says: > > There is probably a range movement happening (move or decommission),but > this node hasn't updated its ring metadata yet It will keep attempting to replay it until the node has updated its ring

Re: Executing a check before replication / manual replication

2017-11-16 Thread kurt greaves
What's the purpose here? If they have access to cqlsh, they have access to every nodes data, not just the one they are on. An attacker modifying RF would be the least of your worries. If you manage to detect that some node is compromise you should isolate it immediately. On 16 Nov. 2017 07:33,

Re: Executing a check before replication / manual replication

2017-11-16 Thread kurt greaves
That sounds like a great way to DoS yourself. While I'm sure it could be achieved, probably in a pretty messy way, I don't think it's a good idea and seems to me like way over the top security. Especially because sure, you might be able to protect against CQL "attacks" via triggers (ugh) - but if

Re: How to know if bootstrap is still running

2017-11-12 Thread kurt greaves
bootstrap will wait for secondary indexes and MV's to build before completing. if either are still shown in compactions then it will wait for them to complete before finishing joining. If not you can try nodetool bootstrap resume if it's available on your version. On 12 Nov. 2017 19:19, "Joel

Re: Safe to run cleanup before repair?

2017-11-12 Thread kurt greaves
By default, bootstrap will stream from the primary replica of the range it is taking ownership of. So Node 3 would have to stream from Node 2 if it was taking ownership of Node 2's tokens. On 13 Nov. 2017 05:00, "Joel Samuelsson" wrote: > Yeah, sounds right. What I'm

Re: Cassandra 3.10 - Hints & Dropped messages logs Vs Cass 2.x version

2017-11-01 Thread kurt greaves
You can get dropped message statistics over JMX. for example nodetool tpstats has a counter for dropped hints from startup. that would be the preferred method for tracking this info, rather than parsing logs On 2 Nov. 2017 6:24 am, "Anumod Mullachery" wrote: Hi

Re: Reg :- Multiple Node Cluster set up in Virtual Box

2017-11-06 Thread kurt greaves
Worth keeping in mind that in 3.6 onwards nodes will not start unless they can contact a seed. Not quite SPOF but still problematic. CASSANDRA-13851

Re: Cassandra 3.10 - Hints & Dropped messages logs Vs Cass 2.x version

2017-11-02 Thread kurt greaves
in cassandra logs, post > 3.x vs 2.x. > > -Anumod > > > > On Wed, Nov 1, 2017 at 4:50 PM, kurt greaves <k...@instaclustr.com> wrote: > >> You can get dropped message statistics over JMX. for example nodetool >> tpstats has a counter for dropped hints fro

Re: decommissioned node still in gossip

2017-11-01 Thread kurt greaves
It will likely hang around in gossip for 3-15 days but then should disappear. As long as it's not showing up in the cluster it should be OK. On 1 Nov. 2017 20:25, "Peng Xiao" <2535...@qq.com> wrote: > Dear All, > > We have decommisioned a DC,but from system.log,it'still gossiping > INFO

Re: Why Cassandra need full repair after incremental repair

2017-11-02 Thread kurt greaves
Where are you seeing this? If your incremental repairs work properly, full repair is only needed in certain situations, like after node failure + replacement, or adding a datacenter.​

Re: sstablescrum fails with OOM

2017-11-03 Thread kurt greaves
Try run nodetool refresh or restarting Cassandra after removing the corrupted file On 4 Nov. 2017 03:54, "Shashi Yachavaram" wrote: > When i tried to simulate this in the lab by moving files (mv > KS-CF-ka-10143-* /tmp/files). > > Ran repair but it fails during snapshot

Re: Tombstoned data seems to remain after compaction

2017-12-10 Thread kurt greaves
The tombstone needs to compact with every SSTable that contains data for the corresponding tombstone. For example the tombstone may be in that SSTable but some data the tombstone covers may possibly be in another SSTable. Only once all SSTables that contain relevant data have been compacted with

Re: When Replacing a Node, How to Force a Consistent Bootstrap

2017-12-06 Thread kurt greaves
That's also an option but it's better to repair before and after if possible, if you don't repair beforehand you could end up missing some replicas until you repair after replacement, which could cause queries to return old/no data. Alternatively you could use ALL after replacing until the repair

Re: Tombstoned data seems to remain after compaction

2017-12-11 Thread kurt greaves
s for your reply! > > > > “”” > > The tombstone needs to compact with every SSTable that contains data for > the corresponding tombstone. > > “”” > > > > Let me explain my understanding by example: > > > > 1. A record inserted with 180 days TTL (Very long

Re: Running repair while Cassandra upgrade 2.0.X to 2.1.X

2017-12-11 Thread kurt greaves
That ticket says that streaming SSTables that are older versions is supported. Streaming is only one component of repairs, and this ticket doesn't talk about repair at all, only bootstrap. For the most part it should work but as Alain said, it's probably best avoided. Especially if you can avoid

Re: Need help with incremental repair

2017-10-30 Thread kurt greaves
Yes mark them as unrepaired first. You can get sstablerepairedset from source if you need (probably make sure you get the correct branch/tag). It's just a shell script so as long as you have C* installed in a default/canonical location it should work.

Re: compaction: huge number of random reads

2018-05-07 Thread kurt greaves
If you've got small partitions/small reads you should test lowering your compression chunk size on the table and disabling read ahead. This sounds like it might just be a case of read amplification. On Tue., 8 May 2018, 05:43 Kyrylo Lebediev, wrote: > Dear Experts, > >

Re: dtests failing with - ValueError: unsupported hash type md5

2018-05-10 Thread kurt greaves
What command did you run? Probably worth checking that cqlsh is installed in the virtual environment and that you are executing pytest from within the virtual env. On 10 May 2018 at 05:06, Rajiv Dimri wrote: > Hi All, > > > > We have setup a dtest environment to run

Re: Invalid metadata has been detected for role

2018-05-17 Thread kurt greaves
Can you post the stack trace and you're version of Cassandra? On Fri., 18 May 2018, 09:48 Abdul Patel, wrote: > Hi > > I had to decommission one dc , now while adding bacl the same nodes ( i > used nodetool decommission) they both get added fine and i also see them im >

Re: row level atomicity and isolation

2018-05-16 Thread kurt greaves
Atomicity and isolation are only guaranteed within a replica. If you have multiple concurrent requests across replicas last timestamp will win. You can get better isolation using LWT which uses paxos under the hood. On 16 May 2018 at 08:55, Rajesh Kishore wrote: > Hi, >

Re: Suggestions for migrating data from cassandra

2018-05-15 Thread kurt greaves
COPY might work but over hundreds of gigabytes you'll probably run into issues if you're overloaded. If you've got access to Spark that would be an efficient way to pull down an entire table and dump it out using the spark-cassandra-connector. On 15 May 2018 at 10:59, Jing Meng

Re: performance on reading only the specific nonPk column

2018-05-21 Thread kurt greaves
Every column will be retrieved (that's populated) from disk and the requested column will then be sliced out in memory and sent back. On 21 May 2018 at 08:34, sujeet jog wrote: > Folks, > > consider a table with 100 metrics with (id , timestamp ) as key, > if one wants to

Re: Timestamp on hints file and system.hints table data

2018-06-18 Thread kurt greaves
RAC1 > > > > On Thu, Jun 14, 2018 at 12:45 AM, kurt greaves > wrote: > >> Does the UUID on the filename correspond with a UUID in nodetool status? >> >> Sounds to me like it could be something weird with an old node that no >> longer exists, although hin

Re:

2018-06-18 Thread kurt greaves
> > 1) Am I correct to assume that the larger page size some user session has > set - the larger portion of cluster/coordinator node resources will be > hogged by the corresponding session? > 2) Do I understand correctly that page size (imagine we have no timeout > settings) is limited by RAM and

Re: Timestamp on hints file and system.hints table data

2018-06-18 Thread kurt greaves
June 2018 at 13:56, learner dba wrote: > Yes Kurt, system log is flooded with hints sent and replayed messages. > > On Monday, June 18, 2018, 7:30:34 AM EDT, kurt greaves < > k...@instaclustr.com> wrote: > > > Not sure what to make of that. Are there any log me

Re: Timestamp on hints file and system.hints table data

2018-06-13 Thread kurt greaves
e is down for months. And yes, I am surprised to look at Unix > timestamp on files. > > > > On Jun 13, 2018, at 6:41 PM, kurt greaves wrote: > > system.hints is not used in Cassandra 3. Can't explain the files though, > are you referring to the files timestamp or the

Re: Compaction strategy for update heavy workload

2018-06-13 Thread kurt greaves
that's never deleted and really small sstables sticking around > forever. If you use really large buckets, what's the point of TWCS? > > Honestly this is such a small workload you could easily use STCS or > LCS and you'd likely never, ever see a problem. > On Wed, Jun 13, 2018 at 3:34

Re: Migrating to Reaper: Switching From Incremental to Reaper's Full Subrange Repair

2018-06-13 Thread kurt greaves
Not strictly necessary but probably a good idea as you don't want two separate pools of SSTables unnecessarily. Also if you've set "only_purge_repaired_tombstones" you'll need to turn that off. On Wed., 13 Jun. 2018, 23:06 Fd Habash, wrote: > For those who are using Reaper … > > > > Currently,

Re: Compaction strategy for update heavy workload

2018-06-13 Thread kurt greaves
TWCS is probably still worth trying. If you mean updating old rows in TWCS "out of order updates" will only really mean you'll hit more SSTables on read. This might add a bit of complexity in your client if your bucketing partitions (not strictly necessary), but that's about it. As long as you're

Re: Timestamp on hints file and system.hints table data

2018-06-13 Thread kurt greaves
system.hints is not used in Cassandra 3. Can't explain the files though, are you referring to the files timestamp or the Unix timestamp in the file name? Is there a node that's been down for several months? On Wed., 13 Jun. 2018, 23:41 Nitan Kainth, wrote: > Hi, > > I observed a strange

Re: Cassandra 3.0.X migarte to VPC

2018-06-07 Thread kurt greaves
> > I meant migrating to gosspsnitch during adding new dc. New dc will be > empty so all the data will be streamed based on snitch property chosen Should work fine on the new DC, as long as the original DC is using a snitch that supports datacenters - then just don't mix and match snitches

Re: 3.11.2 memory leak

2018-06-19 Thread kurt greaves
At this point I'd wait for 3.11.3. If you can't, you can get away with backporting a few repair fixes or just doing sub range repairs on 3.11.2 On Wed., 20 Jun. 2018, 01:10 Abdul Patel, wrote: > Hi All, > > Do we kmow whats the stable version for now if u wish to upgrade ? > > On Tuesday, June

Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage

2018-05-29 Thread kurt greaves
(43 on our large machine) and running with Xmx128M or XmX31G > (derived from $MAX_HEAP_SIZE). For both Xmx, we saw the high CPU caused by > nodetool. > > > > Regards, > > Thomas > > > > *From:* kurt greaves [mailto:k...@instaclustr.com] > *Sent:* Dienstag, 29.

Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage

2018-05-28 Thread kurt greaves
> > 1) nodetool is reusing the $MAX_HEAP_SIZE environment variable, thus if we > are running Cassandra with e.g. Xmx31G, nodetool is started with Xmx31G as > well This was fixed in 3.0.11/3.10 in CASSANDRA-12739 . Not sure why it didn't make

Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage

2018-05-29 Thread kurt greaves
.apache.org/ > jira/browse/CASSANDRA-14475 > > > > Thanks, > > Thomas > > > > *From:* kurt greaves [mailto:k...@instaclustr.com] > *Sent:* Dienstag, 29. Mai 2018 05:54 > *To:* User > *Subject:* Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usag

Re: Is it ok to add more than one node to a exist cluster

2018-06-27 Thread kurt greaves
>> >> Dayu >> >> >> >> At 2018-06-27 17:50:34, "kurt greaves" wrote: >> >> Don't bootstrap nodes simultaneously unless you really know what you're >> doing, and you're using single tokens. It's not straightforward and will >> li

Re: Is it ok to add more than one node to a exist cluster

2018-06-27 Thread kurt greaves
Don't bootstrap nodes simultaneously unless you really know what you're doing, and you're using single tokens. It's not straightforward and will likely lead to data loss/inconsistencies. This applies for all current versions. On 27 June 2018 at 10:21, dayu wrote: > Hi, > I have read a

Re: Determining active sstables and table- dir

2018-05-01 Thread kurt greaves
In 2.2 it's cf_id from system.schema_columnfamilies. If it's not then that's a bug. From 2.2 we stopped including table name in the SSTable name, so whatever directory contains the SSTables is the active one. Conversely, if you've dropped a table and re-added it, the directory without any SSTables

Re: Shifting data to DCOS

2018-05-02 Thread kurt greaves
orkflow? > Can anyone please suggest the best way to move data from one cluster to > another? > > Any help will be greatly appreciated. > > On Tue, Apr 17, 2018 at 6:52 AM, Faraz Mateen <fmat...@an10.io> wrote: > >> Thanks for the response guys. >> >> L

Re: Regular NullPointerExceptions from `nodetool compactionstats` on 3.7 node

2018-04-25 Thread kurt greaves
Typically have seen that in the past when the node is overloaded. Is that a possibility for you? If it works consistently after restarting C* it's likely the issue. On 20 April 2018 at 19:27, Paul Pollack wrote: > Hi all, > > We have a cluster running on Cassandra 3.7

Re: Version Upgrade

2018-05-03 Thread kurt greaves
> > In other words, if I am running Cassandra 1.2.x and upgrading to 2.0.x, > 2.0.x will continue to read all the old Cassandra 1.2.x table. However, if > I then want to upgrade to Cassandra 2.1.x, I’d better make sure all tables > have been upgraded to 2.0.x before making the next upgrade.

Re: Inconsistent Quorum Read after Quorum Write

2018-07-03 Thread kurt greaves
Shouldn't happen. Any chance you could trace the queries, or have you been able to reproduce it? Also, what version of Cassandra? On Wed., 4 Jul. 2018, 06:41 Visa, wrote: > Hi all, > > We recently experienced an unexpected behavior with C* consistency. > > For example, a table t consists of 4

Re: Rebuild to a new DC fails every time

2018-01-07 Thread kurt greaves
If you're on 3.9 it's likely unrelated as streaming_socket_timeout_in_ms is 48 hours. Appears rebuild is trying to stream the same file twice. Are there other exceptions in the logs related to the file, or can you find out if it's previously been sent by the same session? Search the logs for the

Re: Question upon gracefully restarting c* node(s)

2018-01-07 Thread kurt greaves
None are essential. Cassandra will gracefully shutdown in any scenario as long as it's not killed with a SIGKILL. However, drain does have a few benefits over just a normal shutdown. It will stop a few extra services (batchlog, compactions) and importantly it will also force recycling of dirty

Re: secondary index creation causes C* oom

2018-01-10 Thread kurt greaves
> 1.not sure if secondary index creation is the same as index rebuild > Fairly sure they are the same. > 2.we noticed that the memory table flush looks still working,not the same > as CASSANDRA-12796 mentioned,but the compactionExecutor pending is > increasing. > Do you by chance have

Re: Slender Cassandra Cluster Project

2018-01-16 Thread kurt greaves
Sounds like a great idea. Probably would be valuable to add to the official docs as an example set up if you're willing. Only thing I'd add is that you should have keyspaces that replicate over only a subset of DC's, plus one/some replicated to all DC's On 17 Jan. 2018 03:26, "Kenneth Brotman"

Re: vnodes: high availability

2018-01-15 Thread kurt greaves
Yeah it's very unlikely that you will have 2 nodes in the cluster with NO intersecting token ranges (vnodes) for an RF of 3 (probably even 2). If node A goes down all 256 ranges will go down, and considering there are only 49 other nodes all with 256 vnodes each, it's very likely that every node

Re: Cassandra 3.11 - nodetool cleanup - Compaction interrupted ...

2018-01-22 Thread kurt greaves
It's fine and intended behaviour. Upgradesstables also has the same effect. Basically cleanup operates on all SSTables on a node (for each table) and will cancel any in-progress compactions and instead run cleanup across them, as you can't have two different compactions including the same file.

Re: New token allocation and adding a new DC

2018-01-16 Thread kurt greaves
I believe you are able to get away with just altering the keyspace to include both DC's even before the DC exists, and then adding your nodes to that new DC using the algorithm. Note you'll probably want to take the opportunity to reduce the number of vnodes to something reasonable. Based off

Re: vnodes: high availability

2018-01-16 Thread kurt greaves
odes in a cluster means higher probability of simultaneous node > failures. > And from high-availability standpoint, looks like situation is made even > worse by recommended setting vnodes=256. > > Need to do some math to get numbers/formulas, but now situation doesn't > seem

Re: New token allocation and adding a new DC

2018-01-17 Thread kurt greaves
do.de > wrote: > On Wed, Jan 17, 2018 at 4:21 AM, kurt greaves <k...@instaclustr.com> > wrote: > >> I believe you are able to get away with just altering the keyspace to >> include both DC's even before the DC exists, and then adding your nodes to >> that new DC using

Re: Cleanup blocking snapshots - Options?

2018-01-14 Thread kurt greaves
Disabling the snapshots is the best and only real option other than upgrading at the moment. Although apparently it was thought that there was only a small race condition in 2.1 that triggered this and it wasn't worth fixing. If you are triggering it easily maybe it is worth fixing in 2.1 as well.

Re: Repair fails for unknown reason

2018-01-09 Thread kurt greaves
The parent repair session will be on the node that you kicked off the repair on. Are the logs above from that node? Can you make it a bit clearer how many nodes are involved and the corresponding logs from each node? On 9 January 2018 at 09:49, Hannu Kröger wrote: > We have

Re: Too many tombstones using TTL

2018-01-11 Thread kurt greaves
You should be able to avoid querying the tombstones if it's time series data. Using TWCS just make sure you don't query data that you know is expired (assuming you have the time component in your clustering key)​.

Re: default_time_to_live setting in time series data

2018-01-11 Thread kurt greaves
No. Nothing measurable. In fact it should be beneficial if paired with TWCS On 12 Jan. 2018 1:17 pm, "Eunsu Kim" wrote: Hi everyone We are collecting monitoring data in excess of 100K TPS in Cassandra. All data is time series data and must have a TTL. Currently we

Re: Dropped Mutations

2018-01-11 Thread kurt greaves
Dropped mutations aren't data loss. Data loss implies the data was already there and is now gone, whereas for a dropped mutation the data was never there in the first place. A dropped mutation just results in a inconsistency, or potentially no data if all mutations are dropped, and C* will tell

Re: C* keyspace layout

2018-01-11 Thread kurt greaves
Can you elaborate? What interrupt are you referring to? That's a perfectly legitimate topology, it's usefulness depends on your application. On 11 January 2018 at 13:04, Peng Xiao <2535...@qq.com> wrote: > Hi there, > > We plan to set keyspace1 in DC1 and DC2,keyspace2 in DC3 and DC4,all still >

Re: Not what I‘ve expected Performance

2018-02-01 Thread kurt greaves
art more Workers in parallel which boosts in my example, but is still way to slow and far away from requiring to throttle it. And that is what I actually expected when 100 Processes start beating with the Database Cluster. Definitelly I'll give your Code a try. 2018-02-01 6:36 GMT+01:00

Re: Nodes show different number of tokens than initially

2018-02-01 Thread kurt greaves
So one time I tried to understand why only a single node could have a token, and it appeared that it came over the fence from facebook and has been kept ever since. Personally I don't think it's necessary, and agree that it is kind of problematic (but there's probably lot's of stuff that relies on

Re: group by select queries

2018-01-31 Thread kurt greaves
y_id ; > > > > account_id | security_id | counter | avg_exec_price | quantity | > update_time > > +-+-++-- > +- > > user_1 |AMZN | 2 | 1239.2 | 1011 | >

Re: Upgrading sstables not using all available compaction slots on version 2.2

2018-01-31 Thread kurt greaves
Would you be able to create a JIRA ticket for this? Not sure if this is still a problem in 3.0+ but worth creating a ticket to investigate. It'd be really helpful if you could try and reproduce on 3.0.15 or 3.11.1 to see if it's an issue there as well.​

Re: Security Updates

2018-01-31 Thread kurt greaves
Regarding security releases, nothing currently exists to notify users when security related patches are released. At the moment I imagine announcements would only be made in NEWS.txt or on the user mailing list... but only if you're lucky. On 31 January 2018 at 19:18, Michael Shuler

Re: Nodes show different number of tokens than initially

2018-01-31 Thread kurt greaves
> > I don’t know why this is a surprise (maybe because people like to talk > about multiple rings, but the fact that replication strategy is set per > keyspace and that you could use SimpleStrategy in a multiple dc cluster > demonstrates this), but we can chat about that another time This is

Re: TWCS not deleting expired sstables

2018-01-31 Thread kurt greaves
> > > > > > > *From: *Kenneth Brotman <kenbrot...@yahoo.com.INVALID> > *Date: *Tuesday, January 30, 2018 at 7:37 AM > *To: *<user@cassandra.apache.org> > *Subject: *RE: TWCS not deleting expired sstables > > > > Wow! It’s in the DataStax docum

Re: Not what I‘ve expected Performance

2018-01-31 Thread kurt greaves
How are you copying? With CQLSH COPY or your own script? If you've got spark already it's quite simple to copy between tables and it should be pretty much as fast as you can get it. (you may even need to throttle). There's some sample code here (albeit it's copying between clusters but easily

Re: Cleanup blocking snapshots - Options?

2018-01-31 Thread kurt greaves
erspective, a bit annoying right now  > > > > Have asked on https://issues.apache.org/jira/browse/CASSANDRA-13873 > regarding a backport to 2.1, but possibly won’t get attention, cause the > ticket has been resolved for 2.2+ already. > > > > Regards, > > Thomas > &g

Re: Nodes show different number of tokens than initially

2018-01-31 Thread kurt greaves
So the only reason that the new node would "steal" the token is if it started up earlier - which is based off how many heartbeats have occurred since entering NORMAL status on each node. I can't see any reason the new nodes would have higher generation numbers, so sounds likely there's a bug

Re: group by select queries

2018-01-29 Thread kurt greaves
What consistency were you querying at? Can you retry with CONSISTENCY ALL? ​

Re: Nodes show different number of tokens than initially

2018-01-29 Thread kurt greaves
Shouldn't happen. Can you send through nodetool ring output from one of those nodes? Also, did the logs have anything to say about tokens when you started the 3 seed nodes?​

Re: TWCS not deleting expired sstables

2018-01-29 Thread kurt greaves
Likely a read repair caused old data to be brought into a newer SSTable. Try running sstableexpiredblockers to find out if there's a newer SSTable blocking that one from being dropped.​

Re: Heavy one-off writes best practices

2018-02-04 Thread kurt greaves
> > Would you know if there is evidence that inserting skinny rows in sorted > order (no batching) helps C*? This won't have any effect as each insert will be handled separately by the coordinator (or a different coordinator, even). Sorting is also very unlikely to help even if you did batch.

Re: node restart causes application latency

2018-02-12 Thread kurt greaves
Drain will take care of stopping gossip, and does a few tasks before stopping gossip (stops batchlog, hints, auth, cache saver and a few other things). I'm not sure why this causes a side effect when you restart the node, but there should be no need to issue a disablegossip anyway, just leave that

Re: node restart causes application latency

2018-02-12 Thread kurt greaves
​Actually, it's not really clear to me why disablebinary and thrift are necessary prior to drain, because they happen in the same order during drain anyway. It also really doesn't make sense that disabling gossip after drain would make a difference here, because it should be already stopped. This

Roadmap for 4.0

2018-02-11 Thread kurt greaves
Hi friends, *TL;DR: Making a plan for 4.0, ideally everyone interested should provide up to two lists, one for tickets they can contribute resources to getting finished, and one for features they think would be desirable for 4.0, but not necessarily have the resources to commit to helping with.*

Re: Roadmap for 4.0

2018-02-15 Thread kurt greaves
means an extended testing cycle. If all of those patches > landed tomorrow, I'd still expect us to be months away from a release, > because we need to bake the next major - there's too many changes to throw > out an alpha/beta/rc and hope someone actually runs it. > > I don't belie

Re: Memtable flush -> SSTable: customizable or same for all compaction strategies?

2018-02-20 Thread kurt greaves
Probably a lot of work but it would be incredibly useful for vnodes if flushing was range aware (to be used with RangeAwareCompactionStrategy). The writers are already range aware for JBOD, but that's not terribly valuable ATM. On 20 February 2018 at 21:57, Jeff Jirsa wrote: >

Re: vnode random token assignment and replicated data antipatterns

2018-02-20 Thread kurt greaves
> > Outside of rack awareness, would the next primary ranges take the replica > ranges? Yes. ​

Re: Cassandra Needs to Grow Up by Version Five!

2018-02-21 Thread kurt greaves
> > Instead of saying "Make X better" you can quantify "Here's how we can make > X better" in a jira and the conversation will continue with interested > parties (opening jiras are free!). Being combative and insulting project on > mailing list may help vent some frustrations but it is counter

Re: Memtable flush -> SSTable: customizable or same for all compaction strategies?

2018-02-21 Thread kurt greaves
> > Also, I was wondering if the key cache maintains a count of how many local > accesses a key undergoes. Such information might be very useful for > compactions of sstables by splitting data by frequency of use so that those > can be preferentially compacted. No we don't currently have metrics

Re: Rapid scaleup of cassandra nodes with snapshots and initial_token in the yaml

2018-02-15 Thread kurt greaves
Ben did a talk that might have some useful information. It's much more complicated with vnodes though and I doubt you'll be able to get it to be as rapid as you'd want. sets up schema to match This shouldn't be

Re: Hinted Handoff

2018-08-06 Thread kurt greaves
> > Does Cassandra TTL out the hints after max_hint_window_in_ms? From my > understanding, Cassandra only stops collecting hints after > max_hint_window_in_ms but can still keep replaying the hints if the node > comes back again. Is this correct? Is there a way to TTL out hints? No, but it won't

Re: JBOD disk failure

2018-08-15 Thread kurt greaves
gt; Thank you for the answers. We are using the current version 3.11.3 So this > one includes CASSANDRA-6696. > > So if I get this right, losing system tables will need a full node > rebuild. Otherwise repair will get the node consistent again. > > > > Regards, > > Ch

Re: Data Corruption due to multiple Cassandra 2.1 processes?

2018-08-13 Thread kurt greaves
ckport referencing 11540 or re-open 11540? > > > > Thanks for your help. > > > > Thomas > > > > *From:* kurt greaves > *Sent:* Montag, 13. August 2018 13:24 > *To:* User > *Subject:* Re: Data Corruption due to multiple Cassandra 2.1 processes? > &g

Re: 90million reads

2018-08-14 Thread kurt greaves
Not a great idea to make config changes without testing. For a lot of changes you can make the change on one node and measure of three is an improvement however. You'd probably be best to add nodes (double should be sufficient), do tuning and testing afterwards, and then decommission a few nodes

Re: JBOD disk failure

2018-08-14 Thread kurt greaves
If that disk had important data in the system tables however you might have some trouble and need to replace the entire instance anyway. On 15 August 2018 at 12:20, Jeff Jirsa wrote: > Depends on version > > For versions without the fix from Cassandra-6696, the only safe option on > single disk

Re: Data Corruption due to multiple Cassandra 2.1 processes?

2018-08-13 Thread kurt greaves
Yeah that's not ideal and could lead to problems. I think corruption is only likely if compactions occur, but seems like data loss is a potential not to mention all sorts of other possible nasties that could occur running two C*'s at once. Seems to me that 11540 should have gone to 2.1 in the

Re: Configuration parameter to reject incremental repair?

2018-08-13 Thread kurt greaves
No flag currently exists. Probably a good idea considering the serious issues with incremental repairs since forever, and the change of defaults since 3.0. On 7 August 2018 at 16:44, Steinmaurer, Thomas < thomas.steinmau...@dynatrace.com> wrote: > Hello, > > > > we are running Cassandra in AWS

Re: JBOD disk failure

2018-08-17 Thread kurt greaves
As far as I'm aware, yes. I recall hearing someone mention tying system tables to a particular disk but at the moment that doesn't exist. On Fri., 17 Aug. 2018, 01:04 Eric Evans, wrote: > On Wed, Aug 15, 2018 at 3:23 AM kurt greaves wrote: > > Yep. It might require a full nod

Re: Configuration parameter to reject incremental repair?

2018-08-20 Thread kurt greaves
Yeah I meant 2.2. Keep telling myself it was 3.0 for some reason. On 20 August 2018 at 19:29, Oleksandr Shulgin wrote: > On Mon, Aug 13, 2018 at 1:31 PM kurt greaves wrote: > >> No flag currently exists. Probably a good idea considering the serious >> issues with increme

Re: bigger data density with Cassandra 4.0?

2018-08-29 Thread kurt greaves
ood. On 28 August 2018 at 01:37, Dinesh Joshi wrote: > Although the extent of benefits depend on the specific use case, the > cluster size is definitely not a limiting factor. > > Dinesh > > On Aug 27, 2018, at 5:05 AM, kurt greaves wrote: > > I believe there are cavea

Re: bigger data density with Cassandra 4.0?

2018-08-27 Thread kurt greaves
I believe there are caveats that it will only really help if you're not using vnodes, or you have a very small cluster, and also internode encryption is not enabled. Alternatively if you're using JBOD vnodes will be marginally better, but JBOD is not a great idea (and doesn't guarantee a massive

Re: 2.2 eats memory

2018-08-27 Thread kurt greaves
I'm thinking it's unlikely that top is lying to you. Are you sure that you're measuring free memory versus available memory? Cassandra will utilise the OS page cache heavily, which will cache files in memory but leave the memory able to be reclaimed if needed. Have you checked the output of free?

Re: Nodetool refresh v/s sstableloader

2018-08-29 Thread kurt greaves
Removing dev... Nodetool refresh only picks up new SSTables that have been placed in the tables directory. It doesn't account for actual ownership of the data like SSTableloader does. Refresh will only work properly if the SSTables you are copying in are completely covered by that nodes tokens. It

Re: URGENT: disable reads from node

2018-08-29 Thread kurt greaves
Note that you'll miss incoming writes if you do that, so you'll be inconsistent even after the repair. I'd say best to just query at QUORUM until you can finish repairs. On 29 August 2018 at 21:22, Alexander Dejanovski wrote: > Hi Vlad, you must restart the node but first disable joining the

Re: Re: bigger data density with Cassandra 4.0?

2018-08-29 Thread kurt greaves
do)? > > Sent using Zoho Mail <https://www.zoho.com/mail/> > > > ==== Forwarded message > From : kurt greaves > To : "User" > Date : Wed, 29 Aug 2018 12:03:47 +0430 > Subject : Re: bigger data density with Cassandra 4.0? >

Re: batchstatement

2018-07-16 Thread kurt greaves
What is the primary key for the user_by_ext table? I'd assume it's ext_id, which would imply your update doesn't make sense as you can't change the primary key for a row - which would be the problem you're seeing. On Sat., 14 Jul. 2018, 06:14 Randy Lynn, wrote: > TL/DR: > - only 1 out of 14

<    1   2   3   4   >