On 4/13/24 21:02, Tomas Vondra wrote:
On 4/13/24 01:23, David Steele wrote:
Even for the summarizer, though, I do worry about the complexity of
maintaining it over time. It seems like it would be very easy to
introduce a bug and have it go unnoticed until it causes problems in the
field. A lot of testing was done outside of the test suite for this
feature and I'm not sure if we can rely on that focus with every release.
I'm not sure there's a simpler way to implement this. I haven't really
worked on that part (not until the CoW changes a couple weeks ago), but
I think Robert was very conscious of the complexity.
I don't think expect this code to change very often, but I agree it's
not great to rely on testing outside the regular regression test suite.
But I'm not sure how much more we can do, really - for example my
testing was very much "randomized stress testing" with a lot of data and
long runs, looking for unexpected stuff. That's not something we could
do in the usual regression tests, I think.
But if you have suggestions how to extend the testing ...
Doing stress testing in the regular test suite is obviously a problem
due to runtime, but it would still be great to see tests for issues that
were found during external stress testing.
For example, the issue you and Jakub found was fixed in 55a5ee30 but
there is no accompanying test and no existing test was broken by the change.
For me an incremental approach would be to introduce the WAL summarizer
first. There are already plenty of projects that do page-level
incremental (WAL-G, pg_probackup, pgBackRest) and could help shake out
the bugs. Then introduce the client tools later when they are more
robust. Or, release the client tools now but mark them as experimental
or something so people know that changes are coming and they don't get
blindsided by that in the next release. Or, at the very least, make the
caveats very clear so users can make an informed choice.
I don't think introducing just the summarizer, without any client tools,
would really work. How would we even test the summarizer, for example?
If the only users of that code are external tools, we'd do only some
very rudimentary tests. But the more complex tests would happen in the
external tools, which means it wouldn't be covered by cfbot, buildfarm
and so on. Considering the external tools are likely a bit behind, It's
not clear to me how I would do the stress testing, for example.
IMHO we should aim to have in-tree clients when possible, even if some
external tools can do more advanced stuff etc.
This however reminds me my question is the summarizer provides the right
interface(s) for the external tools. One option is to do pg_basebackup
and then parse the incremental files, but is that suitable for the
external tools, or should there be a more convenient way?
Running a pg_basebackup to get the incremental changes would not be at
all satisfactory. Luckily there are the
pg_wal_summary_contents()/pg_available_wal_summaries() functions, which
seem to provide the required information. I have not played with them
much but I think they will do the trick.
They are pretty awkward to work with since they are essentially
time-series data but what you'd really want, I think, is the ability to
get page changes for a particular relfileid/segment.
Regards,
-David
