Grmpf! 1000+ consecutive must be wrong. I guess I mixed sth up. But it repaired over and over again for 1 or 2 days.
2016-12-07 9:01 GMT+01:00 Benjamin Roth <benjamin.r...@jaumo.com>: > Hi Paolo, > > First of all thanks for your review! > > I had the same concerns as you but I thought it is beeing handled > correctly (which does in some situations) but I found one that creates the > inconsistencies you mentioned. That is kind of split brain syndrom, when > multiple nodes fail between repairs. See here: https://cl.ly/3t0X1c0q1L1h. > > I am not happy about it but I support your decision. We should then add > another dtest to test this scenario as existing dtests don't. > > Some issues unfortunately remain: > - 12888 is not resolved > - MV repairs may be still f**** slow. Imagine an inconsistency of a single > cell (also may be due to a validation race condition, see CASSANDRA-12991) > on a big partition. I had issues with reaper and a 30min timeout leading to > 1000+ (yes!) consecutive repairs of a single subrange because it always > timed out and I recognized very late. When I deployed 12888 on my system, > this remaining subrange was repaired in a snap > - I guess rebuild works the same as repair and has to go through the write > path, right? > > => The MV repair may induce so much overhead that it is maybe cheaper to > kill and replace a inconsistent node than to repair it. But that may > introduce inconsistencies again. All in all it is not perfect. All this > does not really un-frustrate me a 100%. > > Do you have any more thoughts? > > Unfortunately I have very little time these days as my second child was > born on monday. So thanks for your support so far. Maybe I have some ideas > on this issues during the next days and I will work on that ticket probably > next week to come to a solution that is at least deployable. I'd also > appreciate your opinion on CASSANDRA-12991. > > 2016-12-07 2:53 GMT+01:00 Paulo Motta <pauloricard...@gmail.com>: > >> Hello Benjamin, >> >> Thanks for your effort on this investigation! For bootstraps and range >> transfers, I think we can indeed simplify and stream base tables and MVs >> as >> ordinary tables, unless there is some caveat I'm missing (I didn't find >> any >> special case for bootstrap/range transfers on CASSANDRA-6477 or in the MV >> design doc, please correct me if I'm wrong). >> >> Regarding repair of base tables, applying mutations via the write path is >> a >> matter of correctness, given that the base table updates needs to >> potentially remove previously referenced keys in the views, so repairing >> only the base table may leave unreferenced keys in the views, breaking the >> MV contract. Furthermore, these unreferenced keys may be propagated to >> other replicas and never removed if you repair only the view. If you don't >> do overwrites in the base table, this is probably not a problem but the DB >> cannot ensure this (at least not before CASSANDRA-9779). Furthermore, as >> you already noticed repairing only the base table is probably faster so I >> don't see a reason to repair the base and MVs separately since this is >> potentially more costly. I believe your frustration is mostly due to the >> bug described on CASSANDRA-12905, but after that and CASSANDRA-12888 are >> fixed repair on base table should work just fine. >> >> Based on this, I propose: >> - Fix CASSANDRA-12905 with your original patch that retries acquiring the >> MV lock instead of throwing WriteTimeoutException during streaming, since >> this is blocking 3.10. >> - Fix CASSANDRA-12888 by doing sstable-based streaming for base tables >> while still applying MV updates in the paired replicas. >> - Create new ticket to use ordinary streaming for non-repair MV stream >> sessions and keep current behavior for MV streaming originating from >> repair. >> - Create new ticket to include only the base tables and not MVs in >> keyspace-level repair, since repairing the base already repairs the views >> to avoid people shooting themselves in the foot. >> >> Please let me know what do you think. Any suggestions or feedback is >> appreciated. >> >> Cheers, >> >> Paulo >> >> 2016-12-02 8:27 GMT-02:00 Benjamin Roth <benjamin.r...@jaumo.com>: >> >> > As I haven't received a single reply on that, I went over to implement >> and >> > test it on my own with our production cluster. I had a real pain with >> > bringing up a new node, so I had to move on. >> > >> > Result: >> > Works like a charm. I ran many dtests that relate in any way with >> storage, >> > stream, bootstrap, ... with good results. >> > The bootstrap finished in under 5:30h, not a single error log during >> > bootstrap. Also afterwards, repairs run smooth, cluster seems to operate >> > quite well. >> > >> > I still need: >> > >> > - Reviews (see 12888, 12905, 12984) >> > - Some opinion if I did the CDC case right. IMHO CDC is not required >> on >> > bootstrap and we don't need to send the mutations through the write >> path >> > just to write the commit log. This will also break incremental >> repairs. >> > Instead for CDC the sstables are streamed like normal but mutations >> are >> > written to commitlog additionally. The worst I see is that the node >> > crashes >> > and the commitlogs for that repair streams are replayed leading to >> > duplicate writes, which is not really crucial and not a regular case. >> > Any >> > better ideas? >> > - Docs have to be updated (12985) if patch is accepted >> > >> > I really appreciate ANY feedback. IMHO the impact of that fixes is >> immense >> > and maybe will be a huge step to get MVs production ready. >> > >> > Thank you very much, >> > Benjamin >> > >> > >> > ---------- Forwarded message ---------- >> > From: Benjamin Roth <benjamin.r...@jaumo.com> >> > Date: 2016-11-29 17:04 GMT+01:00 >> > Subject: Streaming and MVs >> > To: dev@cassandra.apache.org >> > >> > >> > I don't know where else to discuss this issue, so I post it here. >> > >> > I am trying to get CS to run stable with MVs since the beginning of >> july. >> > Normal reads + write do work as expected but when it comes to repairs or >> > bootstrapping it still feels far far away from what I would call fast >> and >> > stable. The other day I just wanted to bootstrap a new node. I tried it >> 2 >> > times. >> > First time the bootstrap failed due to WTEs. I fixed this issue by not >> > timing out in streams but then it turned out that the bootstrap (load >> > roughly 250-300 GB) didn't even finish in 24h. What if I really had a >> > problem and had to get up some nodes fast? No way! >> > >> > I think the root cause of it all is the way streams are handled on >> tables >> > with MVs. >> > Sending them to the regular write path implies so many bottlenecks and >> > sometimes also redundant writes. Let me explain: >> > >> > 1. Bootstrap >> > During a bootstrap, all ranges from all KS and all CFs that will belong >> to >> > the new node will be streamed. MVs are treated like all other CFs and >> all >> > ranges that will move to the new node will also be streamed during >> > bootstrap. >> > Sending streams of the base tables through the write path will have the >> > following negative impacts: >> > >> > - Writes are sent to the commit log. Not necessary. When node is >> stopped >> > during bootstrap, bootstrap will simply start over. No need to >> recover >> > from >> > commit logs. Non-MV tables won't have a CL anyway >> > - MV mutations will not be applied instantly but send to the batch >> log. >> > This is of course necessary during the range movement (if PK of MV >> > differs >> > from base table) but what happens: The batchlog will be completely >> > flooded. >> > This leads to ridiculously large batchlogs (I observerd BLs with 60GB >> > size), zillions of compactions and quadrillions of tombstones. This >> is a >> > pure resource killer, especially because BL uses a CF as a queue. >> > - Applying every mutation separately causes read-before-writes >> during MV >> > mutation. This is of course an order of magnitude slower than simply >> > streaming down an SSTable. This effect becomes even worse while >> > bootstrap >> > progresses and creates more and more (uncompacted) SSTables. Many of >> > them >> > wont ever be compacted because the batchlog eats all the resources >> > available for compaction >> > - Streaming down the MV tables AND applying the mutations of the >> > basetables leads to redundant writes. Redundant writes are local if >> PK >> > of >> > the MV == PK of the base table and - even worse - remote if not. >> Remote >> > MV >> > updates will impact nodes that aren't even part of the bootstrap. >> > - CDC should also not be necessary during bootstrap, should it? TBD >> > >> > 2. Repair >> > Negative impact is similar to bootstrap but, ... >> > >> > - Sending repairs through write path will not mark the streamed >> tables >> > as repaired. See CASSANDRA-12888. Doing NOT so will instantly solve >> that >> > issue. Much simpler with any other solution >> > - It will change the "repair design" a bit. Repairing a base table >> will >> > not automatically repair the MV. But is this bad at all? To be honest >> > as a >> > newbie it was very hard for me to understand what I had to do to be >> sure >> > that everything is repaired correctly. Recently I was told NOT to >> > repair MV >> > CFs but only to repair the base tables. This means one cannot just >> call >> > "nodetool repair $keyspace" - this is complicated, not transparent >> and >> > it >> > sucks. I changed the behaviour in my own branch and let run the >> dtests >> > for >> > MVs. 2 tests failed: >> > - base_replica_repair_test of course failes due to the design >> change >> > - really_complex_repair_test fails because it intentionally times >> out >> > the batch log. IMHO this is a bearable situation. It is >> comparable to >> > resurrected tombstones when running a repair after GCGS expired. >> You >> > also >> > would not expect this to be magically fixed. gcgs default is 10 >> > days and I >> > can expect that anybody also repairs its MVs during that period, >> not >> > only >> > the base table >> > >> > 3. Rebuild >> > Same like bootstrap, isn't it? >> > >> > Did I forget any cases? >> > What do you think? >> > >> > -- >> > Benjamin Roth >> > Prokurist >> > >> > Jaumo GmbH · www.jaumo.com >> > Wehrstraße 46 · 73035 Göppingen · Germany >> > Phone +49 7161 304880-6 <07161%203048806> · Fax +49 7161 304880-1 >> > <07161%203048801> >> > AG Ulm · HRB 731058 · Managing Director: Jens Kammerer >> > >> > >> > >> > -- >> > Benjamin Roth >> > Prokurist >> > >> > Jaumo GmbH · www.jaumo.com >> > Wehrstraße 46 · 73035 Göppingen · Germany >> > Phone +49 7161 304880-6 · Fax +49 7161 304880-1 >> > AG Ulm · HRB 731058 · Managing Director: Jens Kammerer >> > >> > > > > -- > Benjamin Roth > Prokurist > > Jaumo GmbH · www.jaumo.com > Wehrstraße 46 · 73035 Göppingen · Germany > Phone +49 7161 304880-6 <07161%203048806> · Fax +49 7161 304880-1 > <07161%203048801> > AG Ulm · HRB 731058 · Managing Director: Jens Kammerer > -- Benjamin Roth Prokurist Jaumo GmbH · www.jaumo.com Wehrstraße 46 · 73035 Göppingen · Germany Phone +49 7161 304880-6 · Fax +49 7161 304880-1 AG Ulm · HRB 731058 · Managing Director: Jens Kammerer
