My understanding of the H2-backed stores is that at least part of the original rationale behind them was that they were meant to be an interim point on the way to external SQL-backed stores which should theoretically provide significant benefits w.r.t. to GC (obviously unproven, especially at scale).
I don't disagree that the H2 stores themselves are problematic (to say the least); do we have evidence that returning to memory based stores will be an improvement on that? On Thu, Mar 30, 2017 at 12:16 PM, David McLaughlin <dmclaugh...@apache.org> wrote: > Hi all, > > I'd like to start a discussion around storage in Aurora. > > I think one of the biggest mistakes we made in migrating our storage to H2 > was deleting the memory stores as we moved. We made a pretty big bet that > we could eventually make H2/relational databases work. I don't think that > bet has paid off and that we need to revisit the direction we're taking. > > My belief is that the current H2/MyBatis approach is untenable for large > production clusters, at least without changing our current single-master > architecture. At Twitter we are already having to fight to keep GC > manageable even without DbTaskStore enabled, so I don't see a path forward > where we could eventually enable that. So far experiments with H2 off-heap > storage have provided marginal (if any) gains. > > Would anyone object to restoring the in-memory stores and creating new > implementations for the missing ones (UpdateStore)? I'd even go further and > propose that we consider in-memory H2 and MyBatis a failed experiment and > we drop that storage layer completely. > > Cheers, > David >
