neoremind commented on PR #16418:
URL: https://github.com/apache/lucene/pull/16418#issuecomment-5162872651
One observation: if accumulated delta coverage measured by cost() goes about
the "size" of base dense core column, then triggers a fold-to-base compaction
(analogy of major compaction vs. only-delta minor compaction), and from then
on, every subsequent compaction becomes a fold-to-dense rewrite because of
`baseGen != -1`, no matter how few docs actually changed. So the write cost
silently changes from O(changed docs) to O(column)/16.
With that, the lifecycle in practice looks like:
- every flush writes a delta with only the changed docs
- too many deltas -> fold them into one sparse generation
- deltas end up covering the whole column -> fold back to a single dense
column
- **too many deltas over a dense generation -> fold everything into the
dense generation whatsoever**
- a merge flattens everything back to a normal column
So wondering is it possible to do a small-delta-only fold despite the
presence of a dense generation to produce something like `{denseGen,
foldedDelta}` and then decide when to kick off another round of fold-to-dense
based on what scale of deltas have actually accumulated? I understand this may
go beyond the scope of the PR, and there are TODOs there relating to this, but
wanted to call out the fact here since it may help future enhancement. Couple
of thoughts of delta merge policy, it could provide extension points of 1) when
to do minor-delta-only-fold, like exceed `getMaxDocValuesOverlays`, 2) what
contiguous generation deltas (avoid lost update) to pick and what survives, 3)
when to do fold-to-dense compaction.
This reminds me that Google's Mesa and Napa landed on essentially the same
design.
[Napa](https://research.google/pubs/napa-powering-scalable-data-warehousing-with-robust-query-performance-at-google/)
stats "Deltas are constantly consolidated to form larger deltas". And
[Mesa](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42851.pdf)
describes "a two level delta compaction policy" where "For recent versions,
the cumulative deltas compacta small number of singletons". I think my point
here echos these.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
