Thanks Robert, (comments inline, below) On Thu, Jan 31, 2019 at 9:10 AM Robert Newson <rnew...@apache.org> wrote: > I think it would significantly complicate this work to make it a per-database > decision.
Thanks, and that makes perfect sense. That question was really just for my curiosity. > I called out the problems with reduce functionality in the first post of this > thread specifically to shake out people's concerns there, so thank you for > voicing yours. The current approach to reduce only works because we control > the writing of the b+tree nodes, including when they're split, etc, so we're > able to maintain intermediate values on the inner nodes as the data changes > over time. This is not something we can do with FoundationDB directly (or, > indeed, anything else). We're looking for a solution here. Yes, I don't want to dive too deep into the nitty gritty here (my experience with FoundationDB is only a quick skim of the docs, anyway). I was thinking of something along the lines of making a pseudo-btree (just for reductions, distinct from the map emits) where each btree node is a FoundationDB value. It might not be useful or efficient for anything *other* than ranged reduce queries, so perhaps it could be opt-in per ddoc or per view (and v4.x, where x > 0). It could be updated within the same transactions as the map emits, or maybe it could be updated as a cache layer separately from the map emits. > The best we have so far preserves the group level reduces (including group > level of 0, i.e, the reduce value of everything in the view). Those group > level reduces will be at least as efficient as today, I think. For arbitrary > start/endkey reduce we might decide to not support them, or to support them > the expensive way (without the benefit of precomputed intermediate values). This is very good. The majority of our reduce queries use group_level. And the ones that don't define a group_level are either less common or use a very simple builtin reduce fn (usually _count or _sum). If group level reduces remain at least as efficient as today, that may actually be good enough for us. Thanks for the clarifications! Nick Evans
