* Kouhei Kaigai (kai...@ak.jp.nec.com) wrote: > I (plan to) use custom-scan of course. Once a relation is referenced > and optimizer decided GPU acceleration is cheaper, associated custom- > scan node read the data from underlying relation (or in-memory cache > if exists) then move to the shared memory buffer to deliver GPU > management background worker that launches asynchronous DMA one by one. > After that, custom-scan node receives filtered records via shared- > memory buffer, so it can construct tuples to be returned to the upper > node.
Alright- but have you discussed this with Robert? We're going to be
whacking things around for parallel support with new nodes and more
built-in helper functionality for doing this work and I'm not anxious to
have CustomScan end up being a legacy interface that we're required to
pull forward because we accepted it before things had settled.
> > I'm worried about your ideas to try and cache things on the GPU though,
> > if you're not prepared to deal with locks happening in shared memory on
> > the rows you've got cached out on the GPU, or hint bits, or the visibility
> > map being updated, etc...
> >
> It does not remain any state/information on the GPU side. Things related
> to PG internal stuff is job of CPU.
Right, good, I'm glad to hear that this approach is for doing things at
only a individual statement level and it's good to know that it can be
performant at that level now.
> > Well, we're going to need to expand that a bit for aggregates, I'm afraid,
> > but we should be able to define the API for those aggregates very tightly
> > based on what PG does today and require that any FDW purporting to provides
> > those aggregates do it the way PG does. Note that this doesn't solve all
> > the problems- we've got other issues with regard to pushing aggregates down
> > into FDWs that need to be solved.
> >
> I see. It probably needs more detailed investigation.
These issues will hopefully not be a problem (or at least, one that can
be worked around) for non-FDW implementations which are part of core and
implemented in a similar way to the existing aggregates.. Where the
scan node could continue to be a simple SeqScan as it is today.
> The custom-scan API is thin abstraction towards the plan node interface,
> not tightly convinced with a particular use case, like GPU, remote-join
> and so on. So, I'm quite optimistic for the future maintainability.
I don't see how you can be when there hasn't been any discussion that
I've seen about how parallel query execution is going to change things
for us.
> Also, please remind the discussion at the last developer meeting.
> The purpose of custom-scan (we didn't name it at that time) is to avoid
> unnecessary project branch for people who want to implement their own
> special feature but no facilities to enhance optimizer/executor are
> supported.
> Even though we have in-core parallel execution feature by CPU, it also
> makes sense to provide some unique implementation that may be suitable
> for a specific region.
The issue here is that we're going to be expected to maintain an
interface once we provide it and so that isn't something we should be
doing lightly. Particularly when it's as involved as this kind of
change is with what's going on in the backend where we are nearly 100%
sure to be changing things in the next release or two.
Thanks,
Stephen
signature.asc
Description: Digital signature
