Re: [HACKERS] FDW: should GetFdwRoutine be called when drop table?

[Tom Lane]

Robert Haas  writes:
> On Sat, Feb 20, 2016 at 1:43 AM, Tom Lane  wrote:
>> Yes, that's exactly the problem: you'd need some sort of atomic commit
>> mechanism to make this work safely.
>> 
>> It's possible we could give FDWs a bunch of hooks that would let them
>> manage post-commit cleanup the same way smgr does, but it's a far larger
>> project than it might have seemed.

> I've been thinking about the idea of letting foreign data wrappers
> have either (a) a relfilenode that is not zero, representing local
> storage; or perhaps even (b) an array of relfilenodes.  The
> relfilenode, or relfilenodes, would be automatically dropped.  It
> seems like this would be handy for things like cstore_fdw or the
> problem mentioned here, where you do want to manage local storage.

Hmm, mumble.  This assumes that the "FDW" is willing to keep its data
in something that looks externally just like a Postgres heap file (as
opposed to, say, keeping it somewhere else in the filesystem).  That
pretty much gives up the notion that this is "foreign" data access and
instead means that you're trying to force-fit an alternate storage
manager into our FDW-shaped slot.  I doubt it will fit very well.
For one thing, we have never supposed that FDWs were 100% responsible
for managing the data they access, which is why they are not hooked up
to either DROP or a boatload of other maintenance activities like VACUUM,
CLUSTER, REINDEX, etc.  Not to mention that an alternate storage manager
might have its own maintenance activities that don't really fit any of
those concepts.

> If you then also had the generic XLOG patch, maybe you could make it
> WAL-logged, too, if you wanted...

While I've not paid close attention, I had the idea that the "generic
XLOG" patches that have been discussed would still be restricted to
dealing with data that fits into Postgres-style pages (because, for
example, it would have to pass bufmgr's page sanity checks).  That's a
restriction that an alternate storage manager would likely not want.

My point remains that building anything actually useful in this space
is not a small task.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] FDW: should GetFdwRoutine be called when drop table?

[Robert Haas]

On Sat, Feb 20, 2016 at 1:43 AM, Tom Lane  wrote:
> Andres Freund  writes:
>> On 2016-02-19 14:18:19 -0500, Peter Eisentraut wrote:
>>> On 2/19/16 12:21 PM, Feng Tian wrote:
 I have an fdw that each foreign table will acquire some persisted resource.
>
>>> But foreign data wrappers are meant to be wrappers around data managed
>>> elsewhere, not their own storage managers (although that is clearly
>>> tempting), so there might well be other places where this breaks down.
>
>> Sounds like even a BEGIN;DROP TABLE foo;ROLLBACK; will break this
>> approach.
>
> Yes, that's exactly the problem: you'd need some sort of atomic commit
> mechanism to make this work safely.
>
> It's possible we could give FDWs a bunch of hooks that would let them
> manage post-commit cleanup the same way smgr does, but it's a far larger
> project than it might have seemed.

I've been thinking about the idea of letting foreign data wrappers
have either (a) a relfilenode that is not zero, representing local
storage; or perhaps even (b) an array of relfilenodes.  The
relfilenode, or relfilenodes, would be automatically dropped.  It
seems like this would be handy for things like cstore_fdw or the
problem mentioned here, where you do want to manage local storage.  If
you then also had the generic XLOG patch, maybe you could make it
WAL-logged, too, if you wanted...

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] FDW: should GetFdwRoutine be called when drop table?

[Tom Lane]

Andres Freund  writes:
> On 2016-02-19 14:18:19 -0500, Peter Eisentraut wrote:
>> On 2/19/16 12:21 PM, Feng Tian wrote:
>>> I have an fdw that each foreign table will acquire some persisted resource.

>> But foreign data wrappers are meant to be wrappers around data managed
>> elsewhere, not their own storage managers (although that is clearly
>> tempting), so there might well be other places where this breaks down.

> Sounds like even a BEGIN;DROP TABLE foo;ROLLBACK; will break this
> approach.

Yes, that's exactly the problem: you'd need some sort of atomic commit
mechanism to make this work safely.

It's possible we could give FDWs a bunch of hooks that would let them
manage post-commit cleanup the same way smgr does, but it's a far larger
project than it might have seemed.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] FDW: should GetFdwRoutine be called when drop table?

[Andres Freund]

On 2016-02-19 14:18:19 -0500, Peter Eisentraut wrote:
> On 2/19/16 12:21 PM, Feng Tian wrote:
> > I have an fdw that each foreign table will acquire some persisted resource.
> > In my case, some files in file system.   To drop the table cleanly, I
> > have written
> > an object_access_hook that remove those files.  The hook is installed in
> > _PG_init.  
> > 
> > It all worked well except one case. Suppose a user login, the very first
> > command is 
> > drop foreign table.  Drop foreign table will not load the module, so
> > that the hook 
> > is not installed and the files are not properly cleaned up.
> 
> You could load your library with one of the *_library_preload settings
> to make sure the hook is always present.
> 
> But foreign data wrappers are meant to be wrappers around data managed
> elsewhere, not their own storage managers (although that is clearly
> tempting), so there might well be other places where this breaks down.

Sounds like even a BEGIN;DROP TABLE foo;ROLLBACK; will break this
approach.

Andres


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] FDW: should GetFdwRoutine be called when drop table?

[Peter Eisentraut]

On 2/19/16 12:21 PM, Feng Tian wrote:
> I have an fdw that each foreign table will acquire some persisted resource.
> In my case, some files in file system.   To drop the table cleanly, I
> have written
> an object_access_hook that remove those files.  The hook is installed in
> _PG_init.  
> 
> It all worked well except one case. Suppose a user login, the very first
> command is 
> drop foreign table.  Drop foreign table will not load the module, so
> that the hook 
> is not installed and the files are not properly cleaned up.

You could load your library with one of the *_library_preload settings
to make sure the hook is always present.

But foreign data wrappers are meant to be wrappers around data managed
elsewhere, not their own storage managers (although that is clearly
tempting), so there might well be other places where this breaks down.



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] FDW: should GetFdwRoutine be called when drop table?

[Feng Tian]

Hi, Hackers,

I have an fdw that each foreign table will acquire some persisted resource.
In my case, some files in file system.   To drop the table cleanly, I have
written
an object_access_hook that remove those files.  The hook is installed in
_PG_init.

It all worked well except one case. Suppose a user login, the very first
command is
drop foreign table.  Drop foreign table will not load the module, so that
the hook
is not installed and the files are not properly cleaned up.

Should drop foreign table call GetFdwRoutine?   _PG_init is the only
entrance point
that I know for registering hooks, I feel we need to trigger a load for all
DML/DDL on
FDW including drop.   Does this make sense?

Thanks,
Feng