So to implement recursive queries I think what we need is a memoizing node
like Materialize which allows multiple simultaneous readers.
Looking into how to implement this I find that the read position of a
Materialize node is actually implemented directly in tuplestore.c.
That means that tuplestore.c would have to grow a bit to allow for an array of
structs that contain current, readpos_file, readpos_offset, markpos_current,
markpos_file, and markpos_offset.
An alternative I've been thinking about is creating a kind of
slave_tuplestore. It would keep its own readpos etc, but pass all actual
operations through to another tuplestore. I don't think it would pay for
itself though. The complexity involved especially around when arrays are grown
and dumped to disk would be greater than the complexity saved in the regular
Secondly, it would be valuable for recursive queries to be able to throw away
old tuples when all the marks and read positions have passed them (if they're
not random-access of course). This is the same feature that Simon suggested
for merge joins.
I'm thinking the way to do that is to keep an array of arrays instead of a
single array. Whenever (all) the mark and read position(s) advances over the
end of an array we can free it.
Alternatively whenever it comes time to grow the array we could check whether
older tuples aren't needed any more and memmove the later tuples back to the
start of the array instead of growing the array. That would be simpler but it
would be more expensive to do the memmove. Often there would be very few
tuples moving though, and repalloc often does a memcpy anyways, so perhaps it
wouldn't really be a problem.
---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?