Schedule scheme looks good. I believe in process / sidecar can both coexist. As an admin would love to be able to run one or the other or none.
Thank you for taking a lead and producing a plan that can actually be executed. -- Rahul Singh rahul.si...@anant.us Anant Corporation On Apr 12, 2018, 6:35 PM -0400, Joseph Lynch <joe.e.ly...@gmail.com>, wrote: > Given the feedback here and on the ticket, I've written up a proposal > for a repair > sidecar tool > <https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit#heading=h.5f10ng8gzle8 > in the ticket's design document. If there are no major concerns we're going > to start working on porting the Priam implementation into this new tool > soon. > > -Joey > > On Tue, Apr 10, 2018 at 4:17 PM, Elliott Sims <elli...@backblaze.com> wrote: > > > My two cents as a (relatively small) user. I'm coming at this from the > > ops/user side, so my apologies if some of these don't make sense based on a > > more detailed understanding of the codebase: > > > > Repair is definitely a major missing piece of Cassandra. Integrated would > > be easier, but a sidecar might be more flexible. As an intermediate step > > that works towards both options, does it make sense to start with > > finer-grained tracking and reporting for subrange repairs? That is, expose > > a set of interfaces (both internally and via JMX) that give a scheduler > > enough information to run subrange repairs across multiple keyspaces or > > even non-overlapping ranges at the same time. That lets people experiment > > with and quickly/safely/easily iterate on different scheduling strategies > > in the short term, and long-term those strategies can be integrated into a > > built-in scheduler > > > > On the subject of scheduling, I think adjusting parallelism/aggression with > > a possible whitelist or blacklist would be a lot more useful than a "time > > between repairs". That is, if repairs run for a few hours then don't run > > for a few (somewhat hard-to-predict) hours, I still have to size the > > cluster for the load when the repairs are running. The only reason I can > > think of for an interval between repairs is to allow re-compaction from > > repair anticompactions, and subrange repairs seem to eliminate this. Even > > if they didn't, a more direct method along the lines of "don't repair when > > the compaction queue is too long" might make more sense. Blacklisted > > timeslots might be useful for avoiding peak time or batch jobs, but only if > > they can be specified for consistent time-of-day intervals instead of > > unpredictable lulls between repairs. > > > > I really like the idea of automatically adjusting gc_grace_seconds based on > > repair state. The only_purge_repaired_tombstones option fixes this > > elegantly for sequential/incremental repairs on STCS, but not for subrange > > repairs or LCS (unless a scheduler gains the ability somehow to determine > > that every subrange in an sstable has been repaired and mark it > > accordingly?) > > > > > > On 2018/04/03 17:48:14, Blake Eggleston <b...@apple.com> wrote: > > > Hi dev@, > > > > > > > > > > > > > The question of the best way to schedule repairs came up on > > CASSANDRA-14346, and I thought it would be good to bring up the idea of an > > external tool on the dev list. > > > > > > > > > > > > > Cassandra lacks any sort of tools for automating routine tasks that are > > required for running clusters, specifically repair. Regular repair is a > > must for most clusters, like compaction. This means that, especially as far > > as eventual consistency is concerned, Cassandra isn’t totally functional > > out of the box. Operators either need to find a 3rd party solution or > > implement one themselves. Adding this to Cassandra would make it easier to > > use. > > > > > > > > > > > > > Is this something we should be doing? If so, what should it look like? > > > > > > > > > > > > > Personally, I feel like this is a pretty big gap in the project and would > > like to see an out of process tool offered. Ideally, Cassandra would just > > take care of itself, but writing a distributed repair scheduler that you > > trust to run in production is a lot harder than writing a single process > > management application that can failover. > > > > > > > > > > > > > Any thoughts on this? > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > Blake > > > > > > > >
