Re: [HACKERS] Metadata about relation creation & full scans.

[Andres Freund]

On March 22, 2017 3:54:07 PM GMT+01:00, Tom Lane  wrote:
>Andres Freund  writes:
>> On 2017-03-22 10:14:14 -0400, Tom Lane wrote:
>>> (2) How you gonna update this in vacuum?  It cannot do a
>transactional
>>> update.
>
>> I think we can just do that in a separate transaction, at the tail
>end
>> of vacuum_rel() - if we crash just before that, not that much is
>lost.
>
>Blech.  What if someone's queued an exclusive lock request on the
>table?

Point - could do that with a session lock if needed.


>Actually though, maybe you could get away with updating the last-vacuum
>field nontransactionally.  I think we're already updating some of the
>other pg_class fields nontransactionally.

We do, reltuples/pages, relfrozenxid and such. That should work too, the whole 
inplace business ain't pretty, but one more column won't make it much worse.

Andres

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [HACKERS] Metadata about relation creation & full scans.

[Tom Lane]

Andres Freund  writes:
> On 2017-03-22 10:14:14 -0400, Tom Lane wrote:
>> (2) How you gonna update this in vacuum?  It cannot do a transactional
>> update.

> I think we can just do that in a separate transaction, at the tail end
> of vacuum_rel() - if we crash just before that, not that much is lost.

Blech.  What if someone's queued an exclusive lock request on the table?

Actually though, maybe you could get away with updating the last-vacuum
field nontransactionally.  I think we're already updating some of the
other pg_class fields nontransactionally.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Metadata about relation creation & full scans.

[Andres Freund]

On 2017-03-22 10:14:14 -0400, Tom Lane wrote:
> Andres Freund  writes:
> > I propose that for each pg_class entry we start to keep the following
> > additional metadata:
> > - CATALOG_VERSION_NO at relation creation
> > - PG_VERSION_NUM at relation creation
> > - CATALOG_VERSION_NO at last full scan by vacuum
> > - PG_VERSION_NUM at last full scan by vacuum
> 
> (1) It's very very hard for me to believe that we need *two* versions of
> the version number.  Pick one.

PG_VERSION_NUM is probably enough, alright.


> (2) How you gonna update this in vacuum?  It cannot do a transactional
> update.

I think we can just do that in a separate transaction, at the tail end
of vacuum_rel() - if we crash just before that, not that much is lost.
That requires to hand up whether the whole table is scanned, which'd not
be entirely pretty.  Alternatively we could "just" PROC_IN_VACUUM* in
lazy_vacuum_rel(), but that seems like a cure worse than the disease.

Greetings,

Andres Freund


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Metadata about relation creation & full scans.

[Tom Lane]

Andres Freund  writes:
> I propose that for each pg_class entry we start to keep the following
> additional metadata:
> - CATALOG_VERSION_NO at relation creation
> - PG_VERSION_NUM at relation creation
> - CATALOG_VERSION_NO at last full scan by vacuum
> - PG_VERSION_NUM at last full scan by vacuum

(1) It's very very hard for me to believe that we need *two* versions of
the version number.  Pick one.

(2) How you gonna update this in vacuum?  It cannot do a transactional
update.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Metadata about relation creation & full scans.

[Andres Freund]

Hi,

as repeatedly previously discussed, our lack of knowledge in which
release a relation has been created / fully scanned prevents us from
reclaiming infomask bits (and similar), and makes debugging harder
because it's unclear how for one has to go back to look for bugs.

I propose that for each pg_class entry we start to keep the following
additional metadata:
- CATALOG_VERSION_NO at relation creation
- PG_VERSION_NUM at relation creation
- CATALOG_VERSION_NO at last full scan by vacuum
- PG_VERSION_NUM at last full scan by vacuum

pg_upgrade would preserve those fields across upgrades.

The 'last full scan' information is useful, because it'd allow us to
refuse pg_upgrade to run until all tables have been fully vacuumed at
least once since some infomask bit has been removed.  That'd e.g. allow
us to reclaim MOVED_OFF/HEAP_MOVED_IN in $version_introduced + 1 -
having to vacuum each table at least once when moving from a version
older than $version_introduced wouldn't be too bad.

I'm suggesting to also keep PG_VERSION_NUM, because that'd allow us to
backport preparation for features into the stable branches, that we
otherwise couldn't introduce, due to on-disk compatibility reasons.  The
lack of this IIRC came up in a number of discussions. Besides that, it'd
have been useful for debugging more than once.


The 'relation creation' information would e.g. have been quite useful
when we were working through the multixact bugs - not knowing in which
release corrupted tuples might have originated has made that harder.


I'd suggest adding such fields into the CATALOG_VARLEN bit of pg_class,
there's no point in having the information in relcache and such.

Greetings,

Andres Freund


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers