Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Fujii Masao]

On Fri, Mar 17, 2017 at 11:03 PM, Michael Paquier
 wrote:
> On Fri, Mar 17, 2017 at 10:58 PM, Robert Haas  wrote:
>> Fine!  I've committed the pg_clog renaming, but I'd really like to
>> draw the line here.  I'm not going to commit the pg_subtrans ->
>> pg_subxact naming and am -1 on anyone else doing so.  I think that
>> having the names of things in the code match what shows up in the file
>> system is an awfully desirable property which we should preserve
>> insofar as we can do so without compromising other goals.  Not
>> invalidating what users and DBAs think they know about how PostgreSQL
>> works is a good thing, too; there are a lot more people who know
>> something about the directory layout than will read this thread.
>
> I'm cool with that, thanks for the commit.

monitoring.sgml still has some references to "clog". We should update them?

Regards,

-- 
Fujii Masao


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Robert Haas]

On Thu, Mar 16, 2017 at 10:21 PM, Michael Paquier
 wrote:
> On Fri, Mar 17, 2017 at 11:17 AM, Robert Haas  wrote:
>> I understand that the point of renaming pg_clog to pg_xact is that
>> pg_clog contains the dreaded letters l-o-g, which we hypothesize
>> causes DBAs to remove it.  (Alternate hypothesis: "So, that's what's
>> clogging my database!")
>>
>> Renaming pg_subtrans to pg_subxact has no such redeeming properties.
>>
>> More, with each of these renamings, we're further separating what
>> things are called in the code (xlog, clog, subtrans) with what they're
>> called in the filesystem (wal, xact, subxact).
>>
>> So if we must rename pg_clog, OK, but can't we leave pg_subtrans
>> alone?  It's not hurting anybody.
>
> The only argument behind the renaming of pg_subtrans is really
> consistency with pg_xact, because both deal with transactions. I don't
> personally mind if this portion of the renaming is left off, as you
> say anything labelled with "log" is at the origin of this thread.

Fine!  I've committed the pg_clog renaming, but I'd really like to
draw the line here.  I'm not going to commit the pg_subtrans ->
pg_subxact naming and am -1 on anyone else doing so.  I think that
having the names of things in the code match what shows up in the file
system is an awfully desirable property which we should preserve
insofar as we can do so without compromising other goals.  Not
invalidating what users and DBAs think they know about how PostgreSQL
works is a good thing, too; there are a lot more people who know
something about the directory layout than will read this thread.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Michael Paquier]

On Fri, Mar 17, 2017 at 10:58 PM, Robert Haas  wrote:
> Fine!  I've committed the pg_clog renaming, but I'd really like to
> draw the line here.  I'm not going to commit the pg_subtrans ->
> pg_subxact naming and am -1 on anyone else doing so.  I think that
> having the names of things in the code match what shows up in the file
> system is an awfully desirable property which we should preserve
> insofar as we can do so without compromising other goals.  Not
> invalidating what users and DBAs think they know about how PostgreSQL
> works is a good thing, too; there are a lot more people who know
> something about the directory layout than will read this thread.

I'm cool with that, thanks for the commit.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Michael Paquier]

On Fri, Mar 17, 2017 at 11:17 AM, Robert Haas  wrote:
> I understand that the point of renaming pg_clog to pg_xact is that
> pg_clog contains the dreaded letters l-o-g, which we hypothesize
> causes DBAs to remove it.  (Alternate hypothesis: "So, that's what's
> clogging my database!")
>
> Renaming pg_subtrans to pg_subxact has no such redeeming properties.
>
> More, with each of these renamings, we're further separating what
> things are called in the code (xlog, clog, subtrans) with what they're
> called in the filesystem (wal, xact, subxact).
>
> So if we must rename pg_clog, OK, but can't we leave pg_subtrans
> alone?  It's not hurting anybody.

The only argument behind the renaming of pg_subtrans is really
consistency with pg_xact, because both deal with transactions. I don't
personally mind if this portion of the renaming is left off, as you
say anything labelled with "log" is at the origin of this thread.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Robert Haas]

On Thu, Mar 16, 2017 at 9:31 PM, Michael Paquier
 wrote:
> On Fri, Mar 17, 2017 at 12:19 AM, David Steele  wrote:
>> This patch does not apply cleanly at cccbdde:
>>
>> $ git apply ../other/0001-Rename-pg_clog-to-pg_xact.patch
>> error: doc/src/sgml/ref/pg_resetxlog.sgml: No such file or directory
>> error: patch failed: src/backend/postmaster/autovacuum.c:2468
>> error: src/backend/postmaster/autovacuum.c: patch does not apply
>
> This has rotten again...
>
>> Marked "Waiting on Author".
>>
>> I'd really like to see the rest of the renames happen for v10.  It seems
>> like the process got stalled after the pg_wal rename.
>
> Let's see what happens, attached are refreshed versions. Uncertainty
> is the fun part of a CF.

I understand that the point of renaming pg_clog to pg_xact is that
pg_clog contains the dreaded letters l-o-g, which we hypothesize
causes DBAs to remove it.  (Alternate hypothesis: "So, that's what's
clogging my database!")

Renaming pg_subtrans to pg_subxact has no such redeeming properties.

More, with each of these renamings, we're further separating what
things are called in the code (xlog, clog, subtrans) with what they're
called in the filesystem (wal, xact, subxact).

So if we must rename pg_clog, OK, but can't we leave pg_subtrans
alone?  It's not hurting anybody.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Michael Paquier]

On Fri, Mar 17, 2017 at 12:19 AM, David Steele  wrote:
> This patch does not apply cleanly at cccbdde:
>
> $ git apply ../other/0001-Rename-pg_clog-to-pg_xact.patch
> error: doc/src/sgml/ref/pg_resetxlog.sgml: No such file or directory
> error: patch failed: src/backend/postmaster/autovacuum.c:2468
> error: src/backend/postmaster/autovacuum.c: patch does not apply

This has rotten again...

> Marked "Waiting on Author".
>
> I'd really like to see the rest of the renames happen for v10.  It seems
> like the process got stalled after the pg_wal rename.

Let's see what happens, attached are refreshed versions. Uncertainty
is the fun part of a CF.
-- 
Michael


0001-Rename-pg_clog-to-pg_xact.patch
Description: Binary data


0002-Rename-pg_subtrans-to-pg_subxact.patch
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[David Steele]

On 1/17/17 2:31 AM, Michael Paquier wrote:
> On Tue, Nov 29, 2016 at 1:35 PM, Michael Paquier
>  wrote:
>> On Tue, Nov 22, 2016 at 8:35 PM, Haribabu Kommi
>>  wrote:
>>> Hi Craig,
>>>
>>> This is a gentle reminder.
>>>
>>> you assigned as reviewer to the current patch in the 11-2016 commitfest.
>>> But you haven't shared your review yet. Please share your review about
>>> the patch. This will help us in smoother operation of commitfest.
>>>
>>> Please Ignore if you already shared your review.
>>
>> I have moved this CF entry to 2017-01, the remaining, still unreviewed
>> patch are for renaming pg_subxact and pg_clog.
> 
> The second patch has rotten a little because of the backup
> documentation. By the way, is something going to happen in the CF?
> Craig, you are a reviewer of this patch.

This patch does not apply cleanly at cccbdde:

$ git apply ../other/0001-Rename-pg_clog-to-pg_xact.patch
error: doc/src/sgml/ref/pg_resetxlog.sgml: No such file or directory
error: patch failed: src/backend/postmaster/autovacuum.c:2468
error: src/backend/postmaster/autovacuum.c: patch does not apply

Marked "Waiting on Author".

I'd really like to see the rest of the renames happen for v10.  It seems
like the process got stalled after the pg_wal rename.

-- 
-David
da...@pgmasters.net


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Michael Paquier]

On Tue, Jan 17, 2017 at 4:31 PM, Michael Paquier
 wrote:
> On Tue, Nov 29, 2016 at 1:35 PM, Michael Paquier
>  wrote:
>> On Tue, Nov 22, 2016 at 8:35 PM, Haribabu Kommi
>>  wrote:
>>> Hi Craig,
>>>
>>> This is a gentle reminder.
>>>
>>> you assigned as reviewer to the current patch in the 11-2016 commitfest.
>>> But you haven't shared your review yet. Please share your review about
>>> the patch. This will help us in smoother operation of commitfest.
>>>
>>> Please Ignore if you already shared your review.
>>
>> I have moved this CF entry to 2017-01, the remaining, still unreviewed
>> patch are for renaming pg_subxact and pg_clog.
>
> The second patch has rotten a little because of the backup
> documentation. By the way, is something going to happen in the CF?
> Craig, you are a reviewer of this patch.

Moved to CF 2017-03.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Michael Paquier]

On Tue, Nov 29, 2016 at 1:35 PM, Michael Paquier
 wrote:
> On Tue, Nov 22, 2016 at 8:35 PM, Haribabu Kommi
>  wrote:
>> Hi Craig,
>>
>> This is a gentle reminder.
>>
>> you assigned as reviewer to the current patch in the 11-2016 commitfest.
>> But you haven't shared your review yet. Please share your review about
>> the patch. This will help us in smoother operation of commitfest.
>>
>> Please Ignore if you already shared your review.
>
> I have moved this CF entry to 2017-01, the remaining, still unreviewed
> patch are for renaming pg_subxact and pg_clog.

The second patch has rotten a little because of the backup
documentation. By the way, is something going to happen in the CF?
Craig, you are a reviewer of this patch.
-- 
Michael


0001-Rename-pg_clog-to-pg_xact.patch
Description: invalid/octet-stream


0002-Rename-pg_subtrans-to-pg_subxact.patch
Description: invalid/octet-stream

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Michael Paquier]

On Tue, Nov 22, 2016 at 8:35 PM, Haribabu Kommi
 wrote:
> Hi Craig,
>
> This is a gentle reminder.
>
> you assigned as reviewer to the current patch in the 11-2016 commitfest.
> But you haven't shared your review yet. Please share your review about
> the patch. This will help us in smoother operation of commitfest.
>
> Please Ignore if you already shared your review.

I have moved this CF entry to 2017-01, the remaining, still unreviewed
patch are for renaming pg_subxact and pg_clog.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Haribabu Kommi]

Hi Craig,

This is a gentle reminder.

you assigned as reviewer to the current patch in the 11-2016 commitfest.
But you haven't shared your review yet. Please share your review about
the patch. This will help us in smoother operation of commitfest.

Please Ignore if you already shared your review.


Regards,
Hari Babu
Fujitsu Australia

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Alvaro Herrera]

Bruce Momjian wrote:
> On Mon, Oct 24, 2016 at 11:59:42AM -0300, Alvaro Herrera wrote:

> > There is one difference though, which is that the really destructive
> > use of pg_resetxlog is the one that removes pg_xlog files.  The other
> > uses that simply set flags in the control file are not as bad -- you can
> > simply go back to what the original value was.  I think we would only
> > need the obnoxious string requirement in the most dangerous mode.

> OK, but we are going need to document that this special flag is only
> needed for certain option uses.

Perhaps, but not in too much detail; just that "users need to follow
on-screen indications" or some such would be enough.

-- 
Álvaro Herrerahttps://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Bruce Momjian]

On Mon, Oct 24, 2016 at 11:59:42AM -0300, Alvaro Herrera wrote:
> Bruce Momjian wrote:
> > On Sat, Oct 22, 2016 at 11:18:28PM +0300, Greg Stark wrote:
> 
> > > I think the apt-get behaviour was specifically designed to ensure it
> > > couldn't easily be put into a script which I would have said was
> > > desirable -- except I suspect there are situations where Postgres
> > > database scripts need to do a resetxlog. I'm not sure I can think of
> > > any examples offhand but I wouldn't be too surprised.
> > 
> > Yes, pg_upgrade has eight calls to pg_resetxlog to set various value.
> 
> There is one difference though, which is that the really destructive
> use of pg_resetxlog is the one that removes pg_xlog files.  The other
> uses that simply set flags in the control file are not as bad -- you can
> simply go back to what the original value was.  I think we would only
> need the obnoxious string requirement in the most dangerous mode.
> 
> Alternatively, we could separate the tool in two different executables.

OK, but we are going need to document that this special flag is only
needed for certain option uses.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+  Ancient Roman grave inscription +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Alvaro Herrera]

Bruce Momjian wrote:
> On Sat, Oct 22, 2016 at 11:18:28PM +0300, Greg Stark wrote:

> > I think the apt-get behaviour was specifically designed to ensure it
> > couldn't easily be put into a script which I would have said was
> > desirable -- except I suspect there are situations where Postgres
> > database scripts need to do a resetxlog. I'm not sure I can think of
> > any examples offhand but I wouldn't be too surprised.
> 
> Yes, pg_upgrade has eight calls to pg_resetxlog to set various value.

There is one difference though, which is that the really destructive
use of pg_resetxlog is the one that removes pg_xlog files.  The other
uses that simply set flags in the control file are not as bad -- you can
simply go back to what the original value was.  I think we would only
need the obnoxious string requirement in the most dangerous mode.

Alternatively, we could separate the tool in two different executables.

-- 
Álvaro Herrerahttps://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Bruce Momjian]

On Sat, Oct 22, 2016 at 11:18:28PM +0300, Greg Stark wrote:
> On Fri, Oct 21, 2016 at 9:03 PM, Stephen Frost  wrote:
> > WARNING: The following essential packages will be removed.
> > This should NOT be done unless you know exactly what you are doing!
> >   login
> > 0 upgraded, 0 newly installed, 1 to remove and 71 not upgraded.
> > After this operation, 1,212 kB disk space will be freed.
> > You are about to do something potentially harmful.
> > To continue type in the phrase 'Yes, do as I say!'
> 
> Another case was:
>glxgears -iacknowledgethatthistoolisnotabenchmark
> 
> Which was required to get it to print the FPS for a while. The problem
> -- and also the advantage -- of this is that it's scriptable. That
> means people can still put it in recipe books and scripts and others
> can copy it without being aware what it does or even that they're
> doing it.
> 
> I think the apt-get behaviour was specifically designed to ensure it
> couldn't easily be put into a script which I would have said was
> desirable -- except I suspect there are situations where Postgres
> database scripts need to do a resetxlog. I'm not sure I can think of
> any examples offhand but I wouldn't be too surprised.

Yes, pg_upgrade has eight calls to pg_resetxlog to set various value.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+  Ancient Roman grave inscription +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Michael Paquier]

On Sun, Oct 23, 2016 at 5:18 PM, Vik Fearing  wrote:
> On 10/22/2016 06:00 PM, David Steele wrote:
>> On 10/22/16 6:58 PM, Bruce Momjian wrote:
>>> On Sat, Oct 22, 2016 at 07:33:56AM +0900, Michael Paquier wrote:
 On Sat, Oct 22, 2016 at 4:29 AM, Alvaro Herrera

> Also +1 to renaming pg_subtrans to pg_subxact.

 Nice suggestion, good naming for consistency with the rest.
>>>
>>> Agreed.
>>
>> +1
>
> +1 from me, too.

OK, glad to see that we are reaching a conclusion to this thread.
Attached are two patches. 0001 renames pg_clog to pg_xact. I found a
comment in xact.c that was not yet updated. And 0002 renames
pg_subtrans to pg_subxact.
-- 
Michael
From 1711e73fe59b37c0626c1382c6f044b28c01f3d9 Mon Sep 17 00:00:00 2001
From: Michael Paquier 
Date: Sun, 23 Oct 2016 20:52:22 +0900
Subject: [PATCH 1/2] Rename pg_clog to pg_xact

pg_upgrade is updated to handle the transfer correctly to post-10
clusters.
---
 doc/src/sgml/backup.sgml   |  4 ++--
 doc/src/sgml/catalogs.sgml |  4 ++--
 doc/src/sgml/config.sgml   |  2 +-
 doc/src/sgml/func.sgml |  2 +-
 doc/src/sgml/maintenance.sgml  |  8 
 doc/src/sgml/ref/pg_resetxlog.sgml |  4 ++--
 doc/src/sgml/ref/pg_rewind.sgml|  2 +-
 doc/src/sgml/storage.sgml  | 10 +-
 doc/src/sgml/wal.sgml  |  2 +-
 src/backend/access/heap/heapam.c   |  4 ++--
 src/backend/access/transam/README  | 10 +-
 src/backend/access/transam/clog.c  |  2 +-
 src/backend/access/transam/commit_ts.c |  2 +-
 src/backend/access/transam/multixact.c |  2 +-
 src/backend/access/transam/subtrans.c  |  5 ++---
 src/backend/access/transam/transam.c   |  2 +-
 src/backend/access/transam/twophase.c  |  4 ++--
 src/backend/access/transam/xact.c  | 18 +-
 src/backend/access/transam/xlog.c  |  2 +-
 src/backend/commands/vacuum.c  | 10 +-
 src/backend/postmaster/autovacuum.c|  2 +-
 src/backend/storage/buffer/README  |  2 +-
 src/backend/storage/ipc/procarray.c|  4 ++--
 src/backend/utils/time/tqual.c |  6 +++---
 src/bin/initdb/initdb.c|  2 +-
 src/bin/pg_upgrade/exec.c  |  6 ++
 src/bin/pg_upgrade/pg_upgrade.c| 30 ++
 src/include/access/slru.h  |  4 ++--
 28 files changed, 83 insertions(+), 72 deletions(-)

diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml
index 6eaed1e..d22af0d 100644
--- a/doc/src/sgml/backup.sgml
+++ b/doc/src/sgml/backup.sgml
@@ -382,10 +382,10 @@ tar -cf backup.tar /usr/local/pgsql/data
   directories. This will not work because the
   information contained in these files is not usable without
   the commit log files,
-  pg_clog/*, which contain the commit status of
+  pg_xact/*, which contain the commit status of
   all transactions. A table file is only usable with this
   information. Of course it is also impossible to restore only a
-  table and the associated pg_clog data
+  table and the associated pg_xact data
   because that would render all other tables in the database
   cluster useless.  So file system backups only work for complete
   backup and restoration of an entire database cluster.
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 29738b0..839b52a 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -1847,7 +1847,7 @@
All transaction IDs before this one have been replaced with a permanent
(frozen) transaction ID in this table.  This is used to track
whether the table needs to be vacuumed in order to prevent transaction
-   ID wraparound or to allow pg_clog to be shrunk.  Zero
+   ID wraparound or to allow pg_xact to be shrunk.  Zero
(InvalidTransactionId) if the relation is not a table.
   
  
@@ -2514,7 +2514,7 @@
All transaction IDs before this one have been replaced with a permanent
(frozen) transaction ID in this database.  This is used to
track whether the database needs to be vacuumed in order to prevent
-   transaction ID wraparound or to allow pg_clog to be shrunk.
+   transaction ID wraparound or to allow pg_xact to be shrunk.
It is the minimum of the per-table
pg_class.relfrozenxid values.
   
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index adab2f8..8a5d202 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -5834,7 +5834,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH 
csv;
 

 Vacuum also allows removal of old files from the
-pg_clog subdirectory, which is why the default
+pg_xact subdirectory, which is why the default
 is a relatively low 200 million transactions.
 This parameter can only be set at server start, but the 

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Vik Fearing]

On 10/22/2016 06:00 PM, David Steele wrote:
> On 10/22/16 6:58 PM, Bruce Momjian wrote:
>> On Sat, Oct 22, 2016 at 07:33:56AM +0900, Michael Paquier wrote:
>>> On Sat, Oct 22, 2016 at 4:29 AM, Alvaro Herrera
>>>
 Also +1 to renaming pg_subtrans to pg_subxact.
>>>
>>> Nice suggestion, good naming for consistency with the rest.
>>
>> Agreed.
> 
> +1

+1 from me, too.
-- 
Vik Fearing  +33 6 46 75 15 36
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Greg Stark]

On Fri, Oct 21, 2016 at 9:03 PM, Stephen Frost  wrote:
> WARNING: The following essential packages will be removed.
> This should NOT be done unless you know exactly what you are doing!
>   login
> 0 upgraded, 0 newly installed, 1 to remove and 71 not upgraded.
> After this operation, 1,212 kB disk space will be freed.
> You are about to do something potentially harmful.
> To continue type in the phrase 'Yes, do as I say!'

Another case was:
   glxgears -iacknowledgethatthistoolisnotabenchmark

Which was required to get it to print the FPS for a while. The problem
-- and also the advantage -- of this is that it's scriptable. That
means people can still put it in recipe books and scripts and others
can copy it without being aware what it does or even that they're
doing it.

I think the apt-get behaviour was specifically designed to ensure it
couldn't easily be put into a script which I would have said was
desirable -- except I suspect there are situations where Postgres
database scripts need to do a resetxlog. I'm not sure I can think of
any examples offhand but I wouldn't be too surprised.



-- 
greg


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[David Steele]

On 10/22/16 6:58 PM, Bruce Momjian wrote:

On Sat, Oct 22, 2016 at 07:33:56AM +0900, Michael Paquier wrote:

On Sat, Oct 22, 2016 at 4:29 AM, Alvaro Herrera


Also +1 to renaming pg_subtrans to pg_subxact.


Nice suggestion, good naming for consistency with the rest.


Agreed.


+1

--
-David
da...@pgmasters.net


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Bruce Momjian]

On Sat, Oct 22, 2016 at 07:33:56AM +0900, Michael Paquier wrote:
> On Sat, Oct 22, 2016 at 4:29 AM, Alvaro Herrera
>  wrote:
> > David Steele wrote:
> >> On 10/21/16 3:12 AM, David G. Johnston wrote:
> >>
> >> > I have no problem continuing keeping with historical precedent and
> >> > allowing mnemonic abbreviations in our directory and file names at this
> >> > point.
> >>
> >> I'm still in favor of pg_xact.  A search of the 9.6 docs brings up a number
> >> of hits for "xact": pg_last_xact_replay_timestamp(),
> >> pg_advisory_xact_lock(), pg_advisory_xact_lock_shared(),
> >> pg_last_committed_xact(),  pg_prepared_xacts(), etc. There are also 
> >> numerous
> >> column names that have "xact" in them.
> >
> > I'm +1 on pg_clog -> pg_xact.
> 
> So let's say that's the winner then.
> 
> > Also +1 to renaming pg_subtrans to pg_subxact.
> 
> Nice suggestion, good naming for consistency with the rest.

Agreed.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+  Ancient Roman grave inscription +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Michael Paquier]

On Sat, Oct 22, 2016 at 4:29 AM, Alvaro Herrera
 wrote:
> David Steele wrote:
>> On 10/21/16 3:12 AM, David G. Johnston wrote:
>>
>> > I have no problem continuing keeping with historical precedent and
>> > allowing mnemonic abbreviations in our directory and file names at this
>> > point.
>>
>> I'm still in favor of pg_xact.  A search of the 9.6 docs brings up a number
>> of hits for "xact": pg_last_xact_replay_timestamp(),
>> pg_advisory_xact_lock(), pg_advisory_xact_lock_shared(),
>> pg_last_committed_xact(),  pg_prepared_xacts(), etc. There are also numerous
>> column names that have "xact" in them.
>
> I'm +1 on pg_clog -> pg_xact.

So let's say that's the winner then.

> Also +1 to renaming pg_subtrans to pg_subxact.

Nice suggestion, good naming for consistency with the rest.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Alvaro Herrera]

David Steele wrote:
> On 10/21/16 3:12 AM, David G. Johnston wrote:
> 
> > I have no problem continuing keeping with historical precedent ​and
> > allowing mnemonic abbreviations in our directory and file names at this
> > point.
> 
> I'm still in favor of pg_xact.  A search of the 9.6 docs brings up a number
> of hits for "xact": pg_last_xact_replay_timestamp(),
> pg_advisory_xact_lock(), pg_advisory_xact_lock_shared(),
> pg_last_committed_xact(),  pg_prepared_xacts(), etc. There are also numerous
> column names that have "xact" in them.

I'm +1 on pg_clog -> pg_xact.

Also +1 to renaming pg_subtrans to pg_subxact.

In general, +1 to short mnemonic meaningful names.  -1 to overly long
names, -1 to meaningless names.

Regarding pg_receivexlog I think I'd propose pg_recvwal rather than
pg_receivewal, because of pg_recvlogical.

-- 
Álvaro Herrerahttps://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Stephen Frost]

* Robert Haas (robertmh...@gmail.com) wrote:
> I don't think that the problem is that people are accidentally typing
> "pg_resetxlog $PGDATA" and pressing return.  They're typing that on
> purpose, and if you change the sequence of characters required to get
> that effect, they'll just type the new thing instead. The problem is
> that they don't understand when it's appropriate to use that command
> and what the consequences are.

I agree that they don't understand, and that's why I believe that making
the command name, or a required option, a bit more ominous would make
them pause and realize that maybe they want to consider other options
first.

This is not exactly unheard of- apt-get requires an entire phrase be to
be entered when you're asking it to do something extremely dangerous
(remove an essential package):

---
root@ionith:~# apt-get remove login
Reading package lists... Done
Building dependency tree   
Reading state information... Done
The following packages will be REMOVED:
  login
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  login
0 upgraded, 0 newly installed, 1 to remove and 71 not upgraded.
After this operation, 1,212 kB disk space will be freed.
You are about to do something potentially harmful.
To continue type in the phrase 'Yes, do as I say!'
 ?] 
---

My thought would be to make pg_resetxlog do something more along the
lines of what pg_control does and have it, by default, just investigate
the state of things and complain about problems and then have it include
an option to actually reset things with an appropriately ominous name.
The goal there being, primairly, to give a way for users to get
information about why PG isn't starting or what it is complaining about,
with some additional information about what happens if they forcibly
reset the xlog, noting that it could very likely cause corruption, etc.

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Robert Haas]

On Fri, Oct 21, 2016 at 9:47 AM, Stephen Frost  wrote:
> * Robert Haas (robertmh...@gmail.com) wrote:
>> On Thu, Oct 20, 2016 at 12:12 PM, Stephen Frost  wrote:
>> > That said, I'd also like to see a --force or similar option or mechanism
>> > put in place to reduce the risk of users trashing their system because
>> > they think pg_resetwal is "safe." ("It's just gonna reset things to make
>> > the database start again, should be fine.").
>>
>> You know we already have that, right?
>
> Yes, but I was meaning an option which would be required to make
> pg_resetxlog actually *do* anything.  In other words, I'd rather have it
> report some info back to the user, if it's run without the
> '--really-force' or what-have-you option, and only proceed with
> clearing the WAL or rewriting pg_control when '--really-force' is used.

I don't think that the problem is that people are accidentally typing
"pg_resetxlog $PGDATA" and pressing return.  They're typing that on
purpose, and if you change the sequence of characters required to get
that effect, they'll just type the new thing instead. The problem is
that they don't understand when it's appropriate to use that command
and what the consequences are.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Stephen Frost]

* Robert Haas (robertmh...@gmail.com) wrote:
> On Thu, Oct 20, 2016 at 12:12 PM, Stephen Frost  wrote:
> > That said, I'd also like to see a --force or similar option or mechanism
> > put in place to reduce the risk of users trashing their system because
> > they think pg_resetwal is "safe." ("It's just gonna reset things to make
> > the database start again, should be fine.").
> 
> You know we already have that, right?

Yes, but I was meaning an option which would be required to make
pg_resetxlog actually *do* anything.  In other words, I'd rather have it
report some info back to the user, if it's run without the
'--really-force' or what-have-you option, and only proceed with
clearing the WAL or rewriting pg_control when '--really-force' is used.

> > pg_destroydb almost seems like a better choice, though I suppose
> > 'pg_clearwal' would be more acceptable.  Doesn't have quite the same
> > impact though.
> >
> > Not sure on the best answer here, but it's definitely foot-gun that some
> > users have ended up using on themselves with depressing regularity.
> 
> Just to provide some perspective from the other side of this, I
[...]

I wasn't suggesting that we remove the capability.  There are certainly
use-cases for it, but, unfortunately, I've seen a number of cases where
users simply google'd an error that they got back when trying to start
PG and found someone saying "well, I got this error, but then I ran
pg_resetxlog, and now the database starts up again."

It likely doesn't help that the top links tend to be to mailing list
archives where pg_resetxlog was brought up.

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[David Steele]

On 10/21/16 3:12 AM, David G. Johnston wrote:


I have no problem continuing keeping with historical precedent ​and
allowing mnemonic abbreviations in our directory and file names at this
point.


I'm still in favor of pg_xact.  A search of the 9.6 docs brings up a 
number of hits for "xact": pg_last_xact_replay_timestamp(), 
pg_advisory_xact_lock(), pg_advisory_xact_lock_shared(), 
pg_last_committed_xact(),  pg_prepared_xacts(), etc. There are also 
numerous column names that have "xact" in them.


It's not just an arcane developer term when it shows up in a number of 
our user-facing functions/columns.


--
-David
da...@pgmasters.net


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[David G. Johnston]

On Thu, Oct 20, 2016 at 6:03 PM, Robert Haas  wrote:

> On Thu, Oct 20, 2016 at 3:46 PM, Tom Lane  wrote:
> > I'm mostly with Stephen on this.  As the names stand, they encourage
> > people to go look at the documentation,
> > https://www.postgresql.org/docs/devel/static/storage-file-layout.html
> > which will provide more information than you'd ever get out of any
> > reasonable directory name.
>
> Well, we could change them all to pg_a, pg_b, pg_c, pg_d, ... which
> would encourage that even more strongly.  But I don't think that
> proposal can be taken seriously.  Giving things meaningful names is a
> good practice in almost every case.
>

Those don't have the virtue of being at least somewhat ​
m
nemonic
​ like pg_xact.

I'll toss my lot in with Steven's and Tom's on this.

​I have no problem continuing keeping with historical precedent ​and
allowing mnemonic abbreviations in our directory and file names at this
point.

David J.

​

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Michael Paquier]

On Fri, Oct 21, 2016 at 10:03 AM, Robert Haas  wrote:
> On Thu, Oct 20, 2016 at 3:46 PM, Tom Lane  wrote:
>> I'm mostly with Stephen on this.  As the names stand, they encourage
>> people to go look at the documentation,
>> https://www.postgresql.org/docs/devel/static/storage-file-layout.html
>> which will provide more information than you'd ever get out of any
>> reasonable directory name.
>
> Well, we could change them all to pg_a, pg_b, pg_c, pg_d, ... which
> would encourage that even more strongly.  But I don't think that
> proposal can be taken seriously.  Giving things meaningful names is a
> good practice in almost every case.

Moving on with the topic of this thread... I would think that
"pg_xact" is what we are moving to rename pg_clog.

On top of that, after reading the thread, here are the options and
binaries that could be renamed for consistency with the renaming of
pg_xlog:
- pg_xlogdump -> pg_waldump
- pg_resetxlog -> pg_resetwal
- pg_receivexlog -> pg_receivewal
- initdb --xlogdir -> --waldir
- pg_basebackup --xlogmethod --xlogdir -> --walmethod --waldir
That's quite a number, and each change is trivial. Previous options
would be better kept for backward-compatibility. Personally, I see no
urge in changing all that and I am fine with just renaming the *log
directories of PGDATA for this thread. But as the point has been
raised, here are all things that could be done.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Robert Haas]

On Thu, Oct 20, 2016 at 3:46 PM, Tom Lane  wrote:
> I'm mostly with Stephen on this.  As the names stand, they encourage
> people to go look at the documentation,
> https://www.postgresql.org/docs/devel/static/storage-file-layout.html
> which will provide more information than you'd ever get out of any
> reasonable directory name.

Well, we could change them all to pg_a, pg_b, pg_c, pg_d, ... which
would encourage that even more strongly.  But I don't think that
proposal can be taken seriously.  Giving things meaningful names is a
good practice in almost every case.

> Having said that, I still don't like "pg_logical", but I suppose
> renaming it would have more downsides than upsides.

Remind me what your beef is?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Michael Paquier]

On Fri, Oct 21, 2016 at 12:35 AM, Robert Haas  wrote:
> On Wed, Oct 12, 2016 at 10:22 PM, Michael Paquier
>  wrote:
>> OK. I can live with that as well. Attached are three patches. The
>> pg_xlog -> pg_wal move, the pg_clog -> pg_transaction move, and the
>> pg_clog -> pg_xact move. Only one survivor to be chosen among the last
>> two ones.
>
> Committed 0001.

Thanks!
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Tom Lane]

Stephen Frost  writes:
> * David Fetter (da...@fetter.org) wrote:
>> On Thu, Oct 20, 2016 at 02:23:32PM -0400, Tom Lane wrote:
>>> I don't see one single one of those subdirectory names that I'd call
>>> self-documenting.

>> That's a problem we should do something about, even if we can't do it
>> by renaming these all in one go.  At the very least, we can do this
>> for any new names.

> I disagree that excessivly long names for files or directories are
> useful and should be fully self-documenting.

I'm mostly with Stephen on this.  As the names stand, they encourage
people to go look at the documentation,
https://www.postgresql.org/docs/devel/static/storage-file-layout.html
which will provide more information than you'd ever get out of any
reasonable directory name.

Having said that, I still don't like "pg_logical", but I suppose
renaming it would have more downsides than upsides.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Stephen Frost]

* David Fetter (da...@fetter.org) wrote:
> On Thu, Oct 20, 2016 at 02:23:32PM -0400, Tom Lane wrote:
> > Robert Haas  writes:
> > > On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane  wrote:
> > >> We have the two precedents "pg_subtrans" and "pg_multixact", so
> > >> unless we want to get into renaming those too, I think "pg_trans"
> > >> and "pg_xact" are really the only options worth considering.
> > >> 
> > >> Personally I'd go for "pg_trans", but it's only a weak preference.
> > 
> > > Heaven forfend we actually use enough characters to make it 
> > > self-documenting.
> > 
> > $ ls $PGDATA
> > PG_VERSION pg_dynshmem/   pg_notify/ pg_stat_tmp/  
> > postgresql.auto.conf
> > base/  pg_hba.confpg_replslot/   pg_subtrans/  postgresql.conf
> > global/pg_ident.conf  pg_serial/ pg_tblspc/postmaster.opts
> > pg_clog/   pg_logical/pg_snapshots/  pg_twophase/  postmaster.pid
> > pg_commit_ts/  pg_multixact/  pg_stat/   pg_wal/
> > 
> > I don't see one single one of those subdirectory names that I'd call
> > self-documenting.
> 
> That's a problem we should do something about, even if we can't do it
> by renaming these all in one go.  At the very least, we can do this
> for any new names.

I disagree that excessivly long names for files or directories are
useful and should be fully self-documenting.  We should name our
directories with useful hints at what they are used for that remind
those who are knowledgable where things live.  Secondary to that is
using an approach to naming which avoid implying anything about the
directories to those who are *not* knowledgable, which leads us to the
current discussion regarding the removal of directories with 'log' in
the name.

Individuals who are not knowledgable in this area are not going to get
any more benefit from a directory named 'pg_trans' or
'pg_transaction_status' than one named 'pg_clog' or 'pg_xact' and
therefore this whole line of reasoning is a red herring.

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[David Fetter]

On Thu, Oct 20, 2016 at 02:23:32PM -0400, Tom Lane wrote:
> Robert Haas  writes:
> > On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane  wrote:
> >> We have the two precedents "pg_subtrans" and "pg_multixact", so
> >> unless we want to get into renaming those too, I think "pg_trans"
> >> and "pg_xact" are really the only options worth considering.
> >> 
> >> Personally I'd go for "pg_trans", but it's only a weak preference.
> 
> > Heaven forfend we actually use enough characters to make it 
> > self-documenting.
> 
> $ ls $PGDATA
> PG_VERSION pg_dynshmem/   pg_notify/ pg_stat_tmp/  
> postgresql.auto.conf
> base/  pg_hba.confpg_replslot/   pg_subtrans/  postgresql.conf
> global/pg_ident.conf  pg_serial/ pg_tblspc/postmaster.opts
> pg_clog/   pg_logical/pg_snapshots/  pg_twophase/  postmaster.pid
> pg_commit_ts/  pg_multixact/  pg_stat/   pg_wal/
> 
> I don't see one single one of those subdirectory names that I'd call
> self-documenting.

That's a problem we should do something about, even if we can't do it
by renaming these all in one go.  At the very least, we can do this
for any new names.

> Are you proposing we rename them all with carpal-
> tunnel-syndrome-promoting names?

Are you saying that people are getting carpal tunnel syndrome from
hitting the tab key, which has been standard for completion in shells
for decades?  I'm pretty sure that doesn't actually happen.

> There's certainly some case to be made for renaming at least one of
> "pg_subtrans" and "pg_multixact" so that these three similarly-purposed
> subdirectories can all have similar names.  But I think on the whole
> that's (a) fixing what ain't broken, and (b) making it even more unlikely
> that we'll ever get to consensus on changing anything.  We've managed to
> agree that we need to change the names ending in "log"; let's do that
> and be happy that we've removed one foot-gun from the system.

Removing foot guns, un-sexy as it may be from a developer's
perspective, is very useful work.

Best,
David.
-- 
David Fetter  http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter  XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Robert Haas]

On Thu, Oct 20, 2016 at 2:23 PM, Tom Lane  wrote:
> Robert Haas  writes:
>> On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane  wrote:
>>> We have the two precedents "pg_subtrans" and "pg_multixact", so
>>> unless we want to get into renaming those too, I think "pg_trans"
>>> and "pg_xact" are really the only options worth considering.
>>>
>>> Personally I'd go for "pg_trans", but it's only a weak preference.
>
>> Heaven forfend we actually use enough characters to make it self-documenting.
>
> $ ls $PGDATA
> PG_VERSION pg_dynshmem/   pg_notify/ pg_stat_tmp/  
> postgresql.auto.conf
> base/  pg_hba.confpg_replslot/   pg_subtrans/  postgresql.conf
> global/pg_ident.conf  pg_serial/ pg_tblspc/postmaster.opts
> pg_clog/   pg_logical/pg_snapshots/  pg_twophase/  postmaster.pid
> pg_commit_ts/  pg_multixact/  pg_stat/   pg_wal/
>
> I don't see one single one of those subdirectory names that I'd call
> self-documenting.  Are you proposing we rename them all with carpal-
> tunnel-syndrome-promoting names?

No.  Are you proposing that self-documenting names are a bad thing
rather than a good thing?

> There's certainly some case to be made for renaming at least one of
> "pg_subtrans" and "pg_multixact" so that these three similarly-purposed
> subdirectories can all have similar names.  But I think on the whole
> that's (a) fixing what ain't broken, and (b) making it even more unlikely
> that we'll ever get to consensus on changing anything.  We've managed to
> agree that we need to change the names ending in "log"; let's do that
> and be happy that we've removed one foot-gun from the system.

I agree that there is an overwhelming consensus in favor of getting
"log" out of the names, but I do not agree that the only two possible
alternative names are "pg_trans" and "pg_xact", which strikes me as
being akin to saying that the only two options for dinner tonight are
overripe peaches and lunch meats a week past the sell-by date.  If I
had to pick only between those two, I suppose I'd go with "pg_xact"
which is clear enough to someone familiar with PostgreSQL internals,
as opposed to "pg_trans" which I don't think will be clear even to
those people.  But I don't like either one very much.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Bruce Momjian]

On Thu, Oct 20, 2016 at 02:02:27PM -0400, Robert Haas wrote:
> On Thu, Oct 20, 2016 at 1:39 PM, Bruce Momjian  wrote:
> > On Thu, Oct 20, 2016 at 12:29:47PM -0400, Robert Haas wrote:
> >> > When it comes to the name, I tend to think of 'pg_xact' as saying "this
> >> > is where we persist info we need to keep about transactions."  Today
> >> > that's just the commit status info, but I could imagine that there
> >> > might, someday, be other things that go in there.  "pg_multixact" is
> >> > an example of something quite similar but does have more than just one
> >> > "thing."  Also, using "pg_xact" and then "pg_subxact" and "pg_multixact"
> >> > bring them all under one consistent naming scheme.
> >>
> >> I don't dispute the fact that you tend to think of it that way, but I
> >> think it's a real stretch to say that "pg_xact" is a clear name from
> >> the point of view of the uninitiated.  Now, maybe the point is to be a
> >> little bit deliberately unclear, but "xact" for "transaction" is not a
> >> lot better than "xlog" for "write-ahead log".  It's just arbitrary
> >> abbreviations we made up and you either know what they mean or you
> >> don't.  We could call it "pg_xkcd" and we wouldn't be removing much in
> >> the way of clarity.
> >
> > What is your suggestion for a name?  If you have none, I suggest we use
> > "pg_xact".
> 
> I'm not sure.  pg_transaction_status would be clear, but it's long.
> Is pg_xact actually better than pg_clog?

Uh, yeah, no "log".  Wasn't that the point?

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+  Ancient Roman grave inscription +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Tom Lane]

Robert Haas  writes:
> On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane  wrote:
>> We have the two precedents "pg_subtrans" and "pg_multixact", so
>> unless we want to get into renaming those too, I think "pg_trans"
>> and "pg_xact" are really the only options worth considering.
>> 
>> Personally I'd go for "pg_trans", but it's only a weak preference.

> Heaven forfend we actually use enough characters to make it self-documenting.

$ ls $PGDATA
PG_VERSION pg_dynshmem/   pg_notify/ pg_stat_tmp/  postgresql.auto.conf
base/  pg_hba.confpg_replslot/   pg_subtrans/  postgresql.conf
global/pg_ident.conf  pg_serial/ pg_tblspc/postmaster.opts
pg_clog/   pg_logical/pg_snapshots/  pg_twophase/  postmaster.pid
pg_commit_ts/  pg_multixact/  pg_stat/   pg_wal/

I don't see one single one of those subdirectory names that I'd call
self-documenting.  Are you proposing we rename them all with carpal-
tunnel-syndrome-promoting names?

There's certainly some case to be made for renaming at least one of
"pg_subtrans" and "pg_multixact" so that these three similarly-purposed
subdirectories can all have similar names.  But I think on the whole
that's (a) fixing what ain't broken, and (b) making it even more unlikely
that we'll ever get to consensus on changing anything.  We've managed to
agree that we need to change the names ending in "log"; let's do that
and be happy that we've removed one foot-gun from the system.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Robert Haas]

On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane  wrote:
> Robert Haas  writes:
>> Is pg_xact actually better than pg_clog?
>
> Yes, because it doesn't contain the three letters "log".

I figured somebody was going to say that.

> We have the two precedents "pg_subtrans" and "pg_multixact", so
> unless we want to get into renaming those too, I think "pg_trans"
> and "pg_xact" are really the only options worth considering.
>
> Personally I'd go for "pg_trans", but it's only a weak preference.

Heaven forfend we actually use enough characters to make it self-documenting.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Tom Lane]

Robert Haas  writes:
> Is pg_xact actually better than pg_clog?

Yes, because it doesn't contain the three letters "log".

We have the two precedents "pg_subtrans" and "pg_multixact", so
unless we want to get into renaming those too, I think "pg_trans"
and "pg_xact" are really the only options worth considering.

Personally I'd go for "pg_trans", but it's only a weak preference.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Robert Haas]

On Thu, Oct 20, 2016 at 1:39 PM, Bruce Momjian  wrote:
> On Thu, Oct 20, 2016 at 12:29:47PM -0400, Robert Haas wrote:
>> > When it comes to the name, I tend to think of 'pg_xact' as saying "this
>> > is where we persist info we need to keep about transactions."  Today
>> > that's just the commit status info, but I could imagine that there
>> > might, someday, be other things that go in there.  "pg_multixact" is
>> > an example of something quite similar but does have more than just one
>> > "thing."  Also, using "pg_xact" and then "pg_subxact" and "pg_multixact"
>> > bring them all under one consistent naming scheme.
>>
>> I don't dispute the fact that you tend to think of it that way, but I
>> think it's a real stretch to say that "pg_xact" is a clear name from
>> the point of view of the uninitiated.  Now, maybe the point is to be a
>> little bit deliberately unclear, but "xact" for "transaction" is not a
>> lot better than "xlog" for "write-ahead log".  It's just arbitrary
>> abbreviations we made up and you either know what they mean or you
>> don't.  We could call it "pg_xkcd" and we wouldn't be removing much in
>> the way of clarity.
>
> What is your suggestion for a name?  If you have none, I suggest we use
> "pg_xact".

I'm not sure.  pg_transaction_status would be clear, but it's long.
Is pg_xact actually better than pg_clog?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Bruce Momjian]

On Thu, Oct 20, 2016 at 12:29:47PM -0400, Robert Haas wrote:
> > When it comes to the name, I tend to think of 'pg_xact' as saying "this
> > is where we persist info we need to keep about transactions."  Today
> > that's just the commit status info, but I could imagine that there
> > might, someday, be other things that go in there.  "pg_multixact" is
> > an example of something quite similar but does have more than just one
> > "thing."  Also, using "pg_xact" and then "pg_subxact" and "pg_multixact"
> > bring them all under one consistent naming scheme.
> 
> I don't dispute the fact that you tend to think of it that way, but I
> think it's a real stretch to say that "pg_xact" is a clear name from
> the point of view of the uninitiated.  Now, maybe the point is to be a
> little bit deliberately unclear, but "xact" for "transaction" is not a
> lot better than "xlog" for "write-ahead log".  It's just arbitrary
> abbreviations we made up and you either know what they mean or you
> don't.  We could call it "pg_xkcd" and we wouldn't be removing much in
> the way of clarity.

What is your suggestion for a name?  If you have none, I suggest we use
"pg_xact".

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+  Ancient Roman grave inscription +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Robert Haas]

On Thu, Oct 20, 2016 at 12:12 PM, Stephen Frost  wrote:
> That said, I'd also like to see a --force or similar option or mechanism
> put in place to reduce the risk of users trashing their system because
> they think pg_resetwal is "safe." ("It's just gonna reset things to make
> the database start again, should be fine.").

You know we already have that, right?

> pg_destroydb almost seems like a better choice, though I suppose
> 'pg_clearwal' would be more acceptable.  Doesn't have quite the same
> impact though.
>
> Not sure on the best answer here, but it's definitely foot-gun that some
> users have ended up using on themselves with depressing regularity.

Just to provide some perspective from the other side of this, I
handled a severity-1 escalation a few weeks ago where a user basically
called up and explained their situation and I told them based on all
of the facts and circumstances presented that running pg_resetxlog was
going to be the best way forward and they said that was pretty much
what they were expecting to hear, and did so.  I think there's far too
much anti-pg_resetxlog bias on this list.  The situation where you
need to use it is not common in general, but it's pretty common in
support cases that reach me.  I don't think I'd be exaggerating if I
said that 25% of the customer escalations I handle personally require
the use pg_resetxlog.  Of course, that's because by the time the
ticket gets to me, it's typically the case that all of the easy, neat
solutions have already been ruled out.

My experience is that users who are faced with this situation
typically understand that they are in a bad situation and when I
explain to them --- in a clear and direct way but without the sort of
hysterical exaggeration sometimes seen on this mailing list --- what
the consequences are, they are not surprised by those consequences and
they are willing to accept them because they know that the
alternatives are worse.  For example, consider a customer with a
mostly read-only 20TB database.  If that user runs pg_resetxlog, they
can be back up in 5 minutes and it is likely (though of course not
certain) that corruption will be minimal.  If they restore from
backup, they will be down for many more hours.  After pg_resetxlog,
they can restore from backup on another system or do the
dump-and-restore which I invariably recommend while they are up.  For
many customers, this is a good trade-off.  The decision, because of
the risks associated with it, is usually passed up from the DBA I'm
working with to some business leader.  If that business leader feels
that benefits of being up immediately rather than many hours later are
sufficient to justify the risk of a possibly-corrupted database,
running pg_resetxlog is a perfectly reasonable decision.

I have not yet had one DBA or business leader call back and say "hey,
that thing where we ran pg_resetxlog?  you should have discouraged us
more, because that was a disaster".  Now maybe those disasters have
happened and I am just not aware of them, but I think we should all
take a deep breath and remind ourselves that the database exists in
service to the business and not the other way around.  My job -- when
doing support -- is to tell you what your options are and what
consequences they may have, good and bad -- not to tell you how to run
your company.  Telling users that pg_resetxlog will "destroy your
database" is over-the-top hyperbole.  It's a sharp tool with risks and
benefits and it deserves to be presented exactly that way.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Robert Haas]

On Thu, Oct 20, 2016 at 12:05 PM, Stephen Frost  wrote:
>> To be honest, I don't really like either pg_transaction or pg_xact.
>
>> Neither name captures the fact that what we're really tracking here is
>> the transaction *status*.  pg_xact is slightly worse because it's a
>> poor abbreviation for transaction, but I think the argument against
>> even pg_transaction is similar to the one Tom previously levied
>> against pg_logical - viz. "logical what?".  The transaction themselves
>> are not stored in the directory, just the commit status.  The fact
>> that commit status is saved is the source of the "c" in "clog".
>
> This really needs to move forward also.
>
> When it comes to the name, I tend to think of 'pg_xact' as saying "this
> is where we persist info we need to keep about transactions."  Today
> that's just the commit status info, but I could imagine that there
> might, someday, be other things that go in there.  "pg_multixact" is
> an example of something quite similar but does have more than just one
> "thing."  Also, using "pg_xact" and then "pg_subxact" and "pg_multixact"
> bring them all under one consistent naming scheme.

I don't dispute the fact that you tend to think of it that way, but I
think it's a real stretch to say that "pg_xact" is a clear name from
the point of view of the uninitiated.  Now, maybe the point is to be a
little bit deliberately unclear, but "xact" for "transaction" is not a
lot better than "xlog" for "write-ahead log".  It's just arbitrary
abbreviations we made up and you either know what they mean or you
don't.  We could call it "pg_xkcd" and we wouldn't be removing much in
the way of clarity.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Joshua D. Drake]

On 10/20/2016 09:12 AM, Stephen Frost wrote:

* Tom Lane (t...@sss.pgh.pa.us) wrote:

Robert Haas  writes:



That said, I'd also like to see a --force or similar option or mechanism
put in place to reduce the risk of users trashing their system because
they think pg_resetwal is "safe." ("It's just gonna reset things to make
the database start again, should be fine.").

pg_destroydb almost seems like a better choice, though I suppose
'pg_clearwal' would be more acceptable.  Doesn't have quite the same
impact though.


pg_dropwal

Users won't *drop* things they shouldn't on purpose (usually) but they 
will reset and will clear them. Destroydb isn't anymore accurate because 
it doesn't destroy it. Instead it makes it so I can log in again and see 
my data.


(Yes we all know the real implications with it but from a DUH user 
perspective...)


Sincerely,

JD


--
Command Prompt, Inc.  http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Stephen Frost]

* Tom Lane (t...@sss.pgh.pa.us) wrote:
> Robert Haas  writes:
> > One idea would be to rename pg_resetxlog to pg_resetwal.  I think
> > that's actually an improvement.
> 
> This would fit in as part of a general plan to s/xlog/wal/g throughout
> our user-visible names and documentation.  Which seems like a good idea
> to me; there's no need to expose two different terms for the same thing.
> 
> (I don't feel a great need to unify the terminology in the code, though.
> Just in stuff users see.)

+1 on the general change of xlog -> wal.

That said, I'd also like to see a --force or similar option or mechanism
put in place to reduce the risk of users trashing their system because
they think pg_resetwal is "safe." ("It's just gonna reset things to make
the database start again, should be fine.").

pg_destroydb almost seems like a better choice, though I suppose
'pg_clearwal' would be more acceptable.  Doesn't have quite the same
impact though.

Not sure on the best answer here, but it's definitely foot-gun that some
users have ended up using on themselves with depressing regularity.

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Stephen Frost]

* Robert Haas (robertmh...@gmail.com) wrote:
> On Wed, Oct 12, 2016 at 10:22 PM, Michael Paquier
>  wrote:
> > OK. I can live with that as well. Attached are three patches. The
> > pg_xlog -> pg_wal move, the pg_clog -> pg_transaction move, and the
> > pg_clog -> pg_xact move. Only one survivor to be chosen among the last
> > two ones.
> 
> Committed 0001.

Glad to see that happen, finally.

> To be honest, I don't really like either pg_transaction or pg_xact.

> Neither name captures the fact that what we're really tracking here is
> the transaction *status*.  pg_xact is slightly worse because it's a
> poor abbreviation for transaction, but I think the argument against
> even pg_transaction is similar to the one Tom previously levied
> against pg_logical - viz. "logical what?".  The transaction themselves
> are not stored in the directory, just the commit status.  The fact
> that commit status is saved is the source of the "c" in "clog".

This really needs to move forward also.

When it comes to the name, I tend to think of 'pg_xact' as saying "this
is where we persist info we need to keep about transactions."  Today
that's just the commit status info, but I could imagine that there
might, someday, be other things that go in there.  "pg_multixact" is
an example of something quite similar but does have more than just one
"thing."  Also, using "pg_xact" and then "pg_subxact" and "pg_multixact"
bring them all under one consistent naming scheme.

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Tom Lane]

Robert Haas  writes:
> One idea would be to rename pg_resetxlog to pg_resetwal.  I think
> that's actually an improvement.

This would fit in as part of a general plan to s/xlog/wal/g throughout
our user-visible names and documentation.  Which seems like a good idea
to me; there's no need to expose two different terms for the same thing.

(I don't feel a great need to unify the terminology in the code, though.
Just in stuff users see.)

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Robert Haas]

On Wed, Oct 12, 2016 at 10:22 PM, Michael Paquier
 wrote:
> OK. I can live with that as well. Attached are three patches. The
> pg_xlog -> pg_wal move, the pg_clog -> pg_transaction move, and the
> pg_clog -> pg_xact move. Only one survivor to be chosen among the last
> two ones.

Committed 0001.

To be honest, I don't really like either pg_transaction or pg_xact.
Neither name captures the fact that what we're really tracking here is
the transaction *status*.  pg_xact is slightly worse because it's a
poor abbreviation for transaction, but I think the argument against
even pg_transaction is similar to the one Tom previously levied
against pg_logical - viz. "logical what?".  The transaction themselves
are not stored in the directory, just the commit status.  The fact
that commit status is saved is the source of the "c" in "clog".

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Robert Haas]

On Wed, Oct 19, 2016 at 7:05 AM, Christoph Berg  wrote:
> (tl;dr: rename pg_xlog yes, rename pg_resetxlog only if we have a good
> alternative.)

I'm amused by the idea of a TL;DR in parentheses at the very bottom of
the email, but maybe I'm just easily amused.

One idea would be to rename pg_resetxlog to pg_resetwal.  I think
that's actually an improvement.  The "x" in "xlog" is not a clear
abbreviation for "write-ahead", especially when we also use "x" in
other contexts to mean "transaction" - think "xid".  Given our current
practices, we could justifiably rename pg_clog to pg_xlog -- after
all, that's where we log the status of the xacts.  Ugh.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Christoph Berg]

Re: Bruce Momjian 2016-10-19 <20161018220909.ga11...@momjian.us>
> > There's actually another instance of "rename so people shoot their
> > feet less often" here: pg_resetxlog, which is a user-facing tool.
> > Folks on #postgresql have repeatedly been joking that it should rather
> > be named pg_eatmydata, given the number of "howtos" on the web that
> > make use of it. Maybe this is the chance to find a less
> > innocent-sounding name. (Or add a mandatory --rewind-my-data switch.)
> 
> I wonder how many of the uses of pg_resetxlog were caused by mistakenly
> removing the pg_xlog directory.   My point is renaming pg_xlog might
> reduce mistake use of pg_resetxlog.

I don't think there's much of a connection. There are people who
clean up disk space by removing everything that sounds like log files.
For those people renaming the directories makes sense so they don't
have that idea.

There are other people who have random database startup problems, ask
google, and end up with applying some pg_resetxlog advice (that
doesn't necessarily fit their problem). For those people renaming
pg_resetxlog to something that sounds more like "it will break your
data, use as last resort only" might make sense. (Though I don't have
a good suggestion, and the cost of renaming utilities is higher than
renaming some internal directory names.)

The question would now be if there's people who used pg_resetxlog
because they thought it freed up disk space, and if renaming either
would have prevented that. I'm less sure about that.

(tl;dr: rename pg_xlog yes, rename pg_resetxlog only if we have a good
alternative.)

Christoph


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Bruce Momjian]

On Fri, Oct 14, 2016 at 07:19:02PM +0200, Christoph Berg wrote:
> Re: Stephen Frost 2016-10-14 <20161014113523.gz13...@tamriel.snowman.net>
> > > We have a tool called pg_xlogdump in the standard installation.  initdb
> > > has an option --xlogdir, pg_basebackup has --xlog and others.  Renaming
> > > the xlog directory would make this all a bit confusing, unless we're
> > > prepared to rename the programs and options as well.
> > 
> > pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
> > should be terribly concerned about either leaving it named as-is or
> > renaming it.  I agree that we should consider adding alternative names
> > to the options for initdb and pg_basebackup.
> 
> There's actually another instance of "rename so people shoot their
> feet less often" here: pg_resetxlog, which is a user-facing tool.
> Folks on #postgresql have repeatedly been joking that it should rather
> be named pg_eatmydata, given the number of "howtos" on the web that
> make use of it. Maybe this is the chance to find a less
> innocent-sounding name. (Or add a mandatory --rewind-my-data switch.)

I wonder how many of the uses of pg_resetxlog were caused by mistakenly
removing the pg_xlog directory.   My point is renaming pg_xlog might
reduce mistake use of pg_resetxlog.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+  Ancient Roman grave inscription +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Magnus Hagander]

On Fri, Oct 14, 2016 at 11:48 AM, Stephen Frost  wrote:

> * Magnus Hagander (mag...@hagander.net) wrote:
> > On Fri, Oct 14, 2016 at 4:35 AM, Stephen Frost 
> wrote:
> > > * Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote:
> > > > On 10/12/16 11:22 AM, Tom Lane wrote:
> > > > > The main problem we're trying to fix here is people thinking that
> > > > > something with "log" in the name contains discardable data.  Just
> > > > > relocating the directory without renaming it won't improve that.
> > > >
> > > > I think it would help if we moved it to something like
> > > > "internal/pg_xlog" and "internal/pg_clog".  Keep the name but move it
> > > > out of sight.
> > >
> > > I disagree that this will materially help with the issue.
> > >
> > > > We have a tool called pg_xlogdump in the standard installation.
> initdb
> > > > has an option --xlogdir, pg_basebackup has --xlog and others.
> Renaming
> > > > the xlog directory would make this all a bit confusing, unless we're
> > > > prepared to rename the programs and options as well.
> > >
> > > pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
> > > should be terribly concerned about either leaving it named as-is or
> > > renaming it.  I agree that we should consider adding alternative names
> > > to the options for initdb and pg_basebackup.
> >
> > Ugh. Changing the names of those options are probably going to break a
> > *lot* of things, a lot more than changing the names of the directories.
> Is
> > it really worth doing that? Especially if we are even more clear that
> > people should not be touching those directories in the first place.
>
> I would point out that I said "adding alternative names", not
> "change/remove the existing options."
>

Oh, I missed that. Thanks for clarifying :)



> That said, while I like the general idea of "make everything referring
> to transaction log/write-ahead xlog/xlog/wal/whatever refer in the same
> way", I'm not sure that it's a really pressing issue or that changing
> the name of the directory should drive those other changes.
>

Agreed.



> > Those are just the tip of the iceberg. We do use the term xlog in a lot
> of
> > places (almost as many as wal, but that's a different problem)
>
> True, and yes, we should consider the places that already talk about
> "wal", but, overall, I believe we can take this one step at a time and
> don't have to do everything all at once simply because we're changing
> the directory name.  We're not changing what it *is*, after all.
>

Yeah, I think if we aim to fix everything at once, we'll get nothing done.
That old saying of not letting perfection getting in the way of progress
applies.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Stephen Frost]

* Christoph Berg (m...@debian.org) wrote:
> Re: Stephen Frost 2016-10-14 <20161014113523.gz13...@tamriel.snowman.net>
> > > We have a tool called pg_xlogdump in the standard installation.  initdb
> > > has an option --xlogdir, pg_basebackup has --xlog and others.  Renaming
> > > the xlog directory would make this all a bit confusing, unless we're
> > > prepared to rename the programs and options as well.
> > 
> > pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
> > should be terribly concerned about either leaving it named as-is or
> > renaming it.  I agree that we should consider adding alternative names
> > to the options for initdb and pg_basebackup.
> 
> There's actually another instance of "rename so people shoot their
> feet less often" here: pg_resetxlog, which is a user-facing tool.

That is *not* a user-facing tool.

> Folks on #postgresql have repeatedly been joking that it should rather
> be named pg_eatmydata, given the number of "howtos" on the web that
> make use of it. Maybe this is the chance to find a less
> innocent-sounding name. (Or add a mandatory --rewind-my-data switch.)

Not sure on the best approach here, but I agree that it's a problem that
people think pg_resetxlog is some kind of recovery tool (hint: it's
not).

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Stephen Frost]

* Magnus Hagander (mag...@hagander.net) wrote:
> On Fri, Oct 14, 2016 at 4:35 AM, Stephen Frost  wrote:
> > * Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote:
> > > On 10/12/16 11:22 AM, Tom Lane wrote:
> > > > The main problem we're trying to fix here is people thinking that
> > > > something with "log" in the name contains discardable data.  Just
> > > > relocating the directory without renaming it won't improve that.
> > >
> > > I think it would help if we moved it to something like
> > > "internal/pg_xlog" and "internal/pg_clog".  Keep the name but move it
> > > out of sight.
> >
> > I disagree that this will materially help with the issue.
> >
> > > We have a tool called pg_xlogdump in the standard installation.  initdb
> > > has an option --xlogdir, pg_basebackup has --xlog and others.  Renaming
> > > the xlog directory would make this all a bit confusing, unless we're
> > > prepared to rename the programs and options as well.
> >
> > pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
> > should be terribly concerned about either leaving it named as-is or
> > renaming it.  I agree that we should consider adding alternative names
> > to the options for initdb and pg_basebackup.
> 
> Ugh. Changing the names of those options are probably going to break a
> *lot* of things, a lot more than changing the names of the directories. Is
> it really worth doing that? Especially if we are even more clear that
> people should not be touching those directories in the first place.

I would point out that I said "adding alternative names", not
"change/remove the existing options."

That said, while I like the general idea of "make everything referring
to transaction log/write-ahead xlog/xlog/wal/whatever refer in the same
way", I'm not sure that it's a really pressing issue or that changing
the name of the directory should drive those other changes.

> Those are just the tip of the iceberg. We do use the term xlog in a lot of
> places (almost as many as wal, but that's a different problem)

True, and yes, we should consider the places that already talk about
"wal", but, overall, I believe we can take this one step at a time and
don't have to do everything all at once simply because we're changing
the directory name.  We're not changing what it *is*, after all.

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Christoph Berg]

Re: Stephen Frost 2016-10-14 <20161014113523.gz13...@tamriel.snowman.net>
> > We have a tool called pg_xlogdump in the standard installation.  initdb
> > has an option --xlogdir, pg_basebackup has --xlog and others.  Renaming
> > the xlog directory would make this all a bit confusing, unless we're
> > prepared to rename the programs and options as well.
> 
> pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
> should be terribly concerned about either leaving it named as-is or
> renaming it.  I agree that we should consider adding alternative names
> to the options for initdb and pg_basebackup.

There's actually another instance of "rename so people shoot their
feet less often" here: pg_resetxlog, which is a user-facing tool.
Folks on #postgresql have repeatedly been joking that it should rather
be named pg_eatmydata, given the number of "howtos" on the web that
make use of it. Maybe this is the chance to find a less
innocent-sounding name. (Or add a mandatory --rewind-my-data switch.)

Christoph


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Oskari Saarenmaa]

14.10.2016, 07:38, Peter Eisentraut kirjoitti:

On 10/12/16 11:22 AM, Tom Lane wrote:

The main problem we're trying to fix here is people thinking that
something with "log" in the name contains discardable data.  Just
relocating the directory without renaming it won't improve that.


I think it would help if we moved it to something like
"internal/pg_xlog" and "internal/pg_clog".  Keep the name but move it
out of sight.

We have a tool called pg_xlogdump in the standard installation.  initdb
has an option --xlogdir, pg_basebackup has --xlog and others.  Renaming
the xlog directory would make this all a bit confusing, unless we're
prepared to rename the programs and options as well.


pg_receivexlog should probably be renamed, seeing how we have 
pg_recvlogical perhaps pg_recvwal would work?


The --xlog, -x, --xlog-method and -X flags for pg_basebackup are a bit 
confusing as it is.  Perhaps they can be kept around as deprecated 
aliases for a new --wal stream/fetch switch: I don't think we need new 
--wal and --wal-method switches.


pg_resetxlog should probably be called something different than just 
plain pg_resetwal to make it obvious that running it will cause data loss.


pg_xlogdump is a developer tool, users shouldn't care; it's hard enough 
to use as it is as it doesn't do anything useful when you try to point 
it to a recycled wal file.


All in all, I think this is a good opportunity to clarify what the tools 
are actually supposed to do and who should be running them.  As an 
author of a backup tool utilizing some of these tools & options I don't 
think renaming commands and/or arguments is a big deal -- we have to 
deal with a bunch of changes for every new major version anyway.


/ Oskari


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Magnus Hagander]

On Fri, Oct 14, 2016 at 4:35 AM, Stephen Frost  wrote:

> * Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote:
> > On 10/12/16 11:22 AM, Tom Lane wrote:
> > > The main problem we're trying to fix here is people thinking that
> > > something with "log" in the name contains discardable data.  Just
> > > relocating the directory without renaming it won't improve that.
> >
> > I think it would help if we moved it to something like
> > "internal/pg_xlog" and "internal/pg_clog".  Keep the name but move it
> > out of sight.
>
> I disagree that this will materially help with the issue.
>
> > We have a tool called pg_xlogdump in the standard installation.  initdb
> > has an option --xlogdir, pg_basebackup has --xlog and others.  Renaming
> > the xlog directory would make this all a bit confusing, unless we're
> > prepared to rename the programs and options as well.
>
> pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
> should be terribly concerned about either leaving it named as-is or
> renaming it.  I agree that we should consider adding alternative names
> to the options for initdb and pg_basebackup.
>

Ugh. Changing the names of those options are probably going to break a
*lot* of things, a lot more than changing the names of the directories. Is
it really worth doing that? Especially if we are even more clear that
people should not be touching those directories in the first place.

Those are just the tip of the iceberg. We do use the term xlog in a lot of
places (almost as many as wal, but that's a different problem)

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Magnus Hagander]

On Fri, Oct 14, 2016 at 9:56 AM, Tom Lane  wrote:

> Jim Nasby  writes:
> > On 10/14/16 9:06 AM, Stephen Frost wrote:
> >> It'd probably be easier to move the things that are *not* PG internal
> >> (eg: config files, et al) *out* of the data directory and into somewhere
> >> sensible, like /etc ...
>
> > I do think it would be an improvement to segregate things users are
> > expected to touch (*.conf and pg_log are what come to mind) from
> > everything else, which could easily be done by moving everything else to
> > an internal/ directory. I agree that's not much of an improvement for
> > pg_[cx]log, but we could create internal/ as well as rename some things.
>
> I can't get excited about that at all.  It will just get in the way of
> the actually useful end goal we had for directory restructuring, which
> was to separate data to be copied by pg_basebackup from data not to be
> copied by it.  And in reality, people who don't understand that the
> contents of PGDATA should be treated as "hands off unless documented
> otherwise" are not going to be any less likely to shoot themselves in
> the foot just because there's a directory named "internal" or
> "here_be_dragons" or "keep_out_this_means_you" in the way.


+1. I think we should move the don't-include-in-backup files to a separate
directory for *technical* reasons.

I don't think moving files that are already supposed to be internal into a
directory called internal is going to help much, an din particular not for
xlog. If we don't rename it, the problem will remain.

And for the "don't touch" part, if anything we should move the config files
into a subdirectory of PGDATA and not the other way around.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Tom Lane]

Jim Nasby  writes:
> On 10/14/16 9:06 AM, Stephen Frost wrote:
>> It'd probably be easier to move the things that are *not* PG internal
>> (eg: config files, et al) *out* of the data directory and into somewhere
>> sensible, like /etc ...

> I do think it would be an improvement to segregate things users are 
> expected to touch (*.conf and pg_log are what come to mind) from 
> everything else, which could easily be done by moving everything else to 
> an internal/ directory. I agree that's not much of an improvement for 
> pg_[cx]log, but we could create internal/ as well as rename some things.

I can't get excited about that at all.  It will just get in the way of
the actually useful end goal we had for directory restructuring, which
was to separate data to be copied by pg_basebackup from data not to be
copied by it.  And in reality, people who don't understand that the
contents of PGDATA should be treated as "hands off unless documented
otherwise" are not going to be any less likely to shoot themselves in
the foot just because there's a directory named "internal" or
"here_be_dragons" or "keep_out_this_means_you" in the way.

I'd also suggest that an extra level of directory search to get at
database data files is not free.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Stephen Frost]

* Jim Nasby (jim.na...@bluetreble.com) wrote:
> On 10/14/16 9:06 AM, Stephen Frost wrote:
> >>> And internal/base and internal/global and internal/pg_... because
> >>> these shouldn't be touched by the users either.
> >>>
> >>> I don't think this would lead anywhere.
> >It'd probably be easier to move the things that are *not* PG internal
> >(eg: config files, et al) *out* of the data directory and into somewhere
> >sensible, like /etc ...
> 
> I do think it would be an improvement to segregate things users are
> expected to touch (*.conf and pg_log are what come to mind) from
> everything else, which could easily be done by moving everything
> else to an internal/ directory. I agree that's not much of an
> improvement for pg_[cx]log, but we could create internal/ as well as
> rename some things.

Apologis, I should have made it clear what I was getting at.

The Debian/Ubuntu packaging already does segregate out the *.conf and
pg_log into other locations based on the FHS.  On a Debian-based system,
there really shouldn't be anything that the user is expected to touch
under /var/lib (where the PG data directory lives).

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Jim Nasby]

On 10/14/16 9:06 AM, Stephen Frost wrote:

> And internal/base and internal/global and internal/pg_... because
> these shouldn't be touched by the users either.
>
> I don't think this would lead anywhere.

It'd probably be easier to move the things that are *not* PG internal
(eg: config files, et al) *out* of the data directory and into somewhere
sensible, like /etc ...


I do think it would be an improvement to segregate things users are 
expected to touch (*.conf and pg_log are what come to mind) from 
everything else, which could easily be done by moving everything else to 
an internal/ directory. I agree that's not much of an improvement for 
pg_[cx]log, but we could create internal/ as well as rename some things.

--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)   mobile: 512-569-9461


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Stephen Frost]

* Christoph Berg (m...@debian.org) wrote:
> Re: Stephen Frost 2016-10-14 <20161014113523.gz13...@tamriel.snowman.net>
> > > I think it would help if we moved it to something like
> > > "internal/pg_xlog" and "internal/pg_clog".  Keep the name but move it
> > > out of sight.
> > 
> > I disagree that this will materially help with the issue.
> 
> And internal/base and internal/global and internal/pg_... because
> these shouldn't be touched by the users either.
> 
> I don't think this would lead anywhere.

It'd probably be easier to move the things that are *not* PG internal
(eg: config files, et al) *out* of the data directory and into somewhere
sensible, like /etc ...

Oh, wait ...

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Christoph Berg]

Re: Stephen Frost 2016-10-14 <20161014113523.gz13...@tamriel.snowman.net>
> > I think it would help if we moved it to something like
> > "internal/pg_xlog" and "internal/pg_clog".  Keep the name but move it
> > out of sight.
> 
> I disagree that this will materially help with the issue.

And internal/base and internal/global and internal/pg_... because
these shouldn't be touched by the users either.

I don't think this would lead anywhere.

Christoph


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Stephen Frost]

* Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote:
> On 10/12/16 11:22 AM, Tom Lane wrote:
> > The main problem we're trying to fix here is people thinking that
> > something with "log" in the name contains discardable data.  Just
> > relocating the directory without renaming it won't improve that.
> 
> I think it would help if we moved it to something like
> "internal/pg_xlog" and "internal/pg_clog".  Keep the name but move it
> out of sight.

I disagree that this will materially help with the issue.

> We have a tool called pg_xlogdump in the standard installation.  initdb
> has an option --xlogdir, pg_basebackup has --xlog and others.  Renaming
> the xlog directory would make this all a bit confusing, unless we're
> prepared to rename the programs and options as well.

pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
should be terribly concerned about either leaving it named as-is or
renaming it.  I agree that we should consider adding alternative names
to the options for initdb and pg_basebackup.

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Peter Eisentraut]

On 10/12/16 11:22 AM, Tom Lane wrote:
> The main problem we're trying to fix here is people thinking that
> something with "log" in the name contains discardable data.  Just
> relocating the directory without renaming it won't improve that.

I think it would help if we moved it to something like
"internal/pg_xlog" and "internal/pg_clog".  Keep the name but move it
out of sight.

We have a tool called pg_xlogdump in the standard installation.  initdb
has an option --xlogdir, pg_basebackup has --xlog and others.  Renaming
the xlog directory would make this all a bit confusing, unless we're
prepared to rename the programs and options as well.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Stephen Frost]

* David Steele (da...@pgmasters.net) wrote:
> On 10/4/16 1:48 AM, Michael Paquier wrote:
> 
> > So this is still open for votes. Here are the candidates and who
> > voiced for what:
> > - pg_transaction: Michael P, Thomas M. => Current 0002 is doing that.
> > - pg_xact: David S, Robert
> > - pg_trans: Tom
> > - pg_transaction_status: Peter E.
> 
> Christoph also prefers pg_xact [1].

Since we're asking for votes, I also prefer pg_xact over the other
options listed here.

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Tom Lane]

Peter Eisentraut  writes:
> On 10/4/16 1:48 AM, Michael Paquier wrote:
>> So this is still open for votes. Here are the candidates and who
>> voiced for what:
>> - pg_transaction: Michael P, Thomas M. => Current 0002 is doing that.
>> - pg_xact: David S, Robert
>> - pg_trans: Tom
>> - pg_transaction_status: Peter E.

> I think this would all be a lot easier if we just moved all the pg_*
> directories one directory down.

The main problem we're trying to fix here is people thinking that
something with "log" in the name contains discardable data.  Just
relocating the directory without renaming it won't improve that.

The relocation aspect was meant to address another problem, which
is making it easier to distinguish which parts of the tree should
be copied by tools like pg_basebackup.  But that's not what Michael
is asking for votes on.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Peter Eisentraut]

On 10/4/16 1:48 AM, Michael Paquier wrote:
> So this is still open for votes. Here are the candidates and who
> voiced for what:
> - pg_transaction: Michael P, Thomas M. => Current 0002 is doing that.
> - pg_xact: David S, Robert
> - pg_trans: Tom
> - pg_transaction_status: Peter E.

I think this would all be a lot easier if we just moved all the pg_*
directories one directory down.  We did this with "global" many years
ago and it worked out well.  We didn't actually have to change the file
names, the top-level directory became cleaner, and no one was messing
around with the files anymore.  Adjusting external tools also becomes
easier.  There is so much lore out there about clog and xlog; it would
be annoying to break with that.

I can see a reason for not doing that with pg_xlog, because that is a
bit of an external interface, with initdb -X and such, and pg_wal is a
pretty good alternative name.  But no one deals with these other
directories directly, so just move them out of the way.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Robert Haas]

On Tue, Oct 4, 2016 at 1:48 AM, Michael Paquier
 wrote:
> Yes, pg_wal is fine as new name in replacement of pg_xlog. Now for the
> pg_clog...

Of course the irony here is that "WAL" stands for "Write Ahead Log".
So we're renaming a directly that has "log" in the name (pg_xlog) to a
directory that has an "l" in the name which stands for "log" (pg_wal).
Whee!

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[David Steele]

On 10/4/16 1:48 AM, Michael Paquier wrote:

> So this is still open for votes. Here are the candidates and who
> voiced for what:
> - pg_transaction: Michael P, Thomas M. => Current 0002 is doing that.
> - pg_xact: David S, Robert
> - pg_trans: Tom
> - pg_transaction_status: Peter E.

Christoph also prefers pg_xact [1].

-- 
-David
da...@pgmasters.net

[1]
https://www.postgresql.org/message-id/20161003141959.qhvd6roesj4kp...@msg.df7cb.de


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Tom Lane]

Robert Haas  writes:
> I think the tests for PQserverVersion(conn) / 100 >= 1000 are strange.
> I submit that either PQserverVersion(conn) >= 10 or
> PQserverVersion(conn) / 1 >= 10 is an easier-to-understand test.
> I vote for the first style.

+1, that's the way most existing tests of this sort are written.

(Right at the moment, there's kind of a lot of zeroes there, but
once we get to version 11 it'll be less bad.)

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Christoph Berg]

Re: Michael Paquier 2016-09-30 

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Robert Haas]

On Fri, Sep 30, 2016 at 1:45 AM, Michael Paquier
 wrote:
> As there have been no reviews at code level, I am moving that to the next CF.

Code review of 0001:

I think the tests for PQserverVersion(conn) / 100 >= 1000 are strange.
I submit that either PQserverVersion(conn) >= 10 or
PQserverVersion(conn) / 1 >= 10 is an easier-to-understand test.
I vote for the first style.  So in pg_basebackup.c I'd do:

#define MINIMUM_VERSION_FOR_PG_WAL 10
...
int version = PQserverVersion(conn);
...
if (version >= MINIMUM_VERSION_FOR_PG_WAL)
   /* do whatever */

Also, for this sort of thing:

+if (!conn)
+/* Error message already written in GetConnection() */
+exit(1);

...because of the comment, I'd add braces around this.

Tom changed the error-reporting framework for pg_upgrade in
f002ed2b8e45286fe73a36e119cba2016ea0de19, so you'll need to do some
rebasing over that.  I haven't checked what the new conventions are,
but obviously you'll want to try to be consistent with them.

Other than those minor nitpicks, I think this looks OK.  I'm not sure
we have consensus on 0002, but 0001 seems to be popular with far more
people than not.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Thomas Munro]

On Thu, Sep 29, 2016 at 9:17 PM, Michael Paquier
 wrote:
> On Tue, Aug 30, 2016 at 11:04 AM, Michael Paquier
>  wrote:
>> On Mon, Aug 29, 2016 at 9:39 PM, Craig Ringer
>>  wrote:
>>> Cool. I'll mark as waiting on author pending that.
>>>
>>> It'll be good to get this footgun put away.
>>
>> OK, so done. I have put the renaming of pg_xlog to pg_wal on top patch
>> stack as that's the one making no discussion it seems: a lot of people
>> like pg_wal. I have added as well handling for the renaming in
>> pg_basebackup by using PQserverVersion. One thing to note is that a
>> connection needs to be made to the target server *before* creating the
>> soft link of pg_xlog/pg_wal because we need to know the version of the
>> target server. pg_upgrade is handled as well. And that's all in 0001.
>>
>> Patch 0002 does pg_clog -> pg_trans, "trans" standing for
>> "transaction". Switching to pg_trans_status or pg_xact_status is not
>> that complicated to change anyway...
>
> Any input to offer for those patches? If there is nothing happening, I
> guess that the best move is just to move it to next CF. At least I can
> see that the flow would be to get those renamings done.

+1 for pg_xlog -> pg_wal.

Of the existing suggestions, would like to add my vote for the
following renames, matching pg_multixact:

pg_clog -> pg_xact
pg_subtrans -> pg_subxact

If longer names are on the table, I would consider expanding all three of those:

pg_clog -> pg_transaction
pg_subtrans -> pg_subtransaction
pg_multixact -> pg_multitransaction

They sound eminently non-deletable.

-- 
Thomas Munro
http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Michael Paquier]

On Tue, Aug 30, 2016 at 11:04 AM, Michael Paquier
 wrote:
> On Mon, Aug 29, 2016 at 9:39 PM, Craig Ringer
>  wrote:
>> Cool. I'll mark as waiting on author pending that.
>>
>> It'll be good to get this footgun put away.
>
> OK, so done. I have put the renaming of pg_xlog to pg_wal on top patch
> stack as that's the one making no discussion it seems: a lot of people
> like pg_wal. I have added as well handling for the renaming in
> pg_basebackup by using PQserverVersion. One thing to note is that a
> connection needs to be made to the target server *before* creating the
> soft link of pg_xlog/pg_wal because we need to know the version of the
> target server. pg_upgrade is handled as well. And that's all in 0001.
>
> Patch 0002 does pg_clog -> pg_trans, "trans" standing for
> "transaction". Switching to pg_trans_status or pg_xact_status is not
> that complicated to change anyway...

Any input to offer for those patches? If there is nothing happening, I
guess that the best move is just to move it to next CF. At least I can
see that the flow would be to get those renamings done.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Jim Nasby]

On 8/29/16 7:34 PM, Andres Freund wrote:

What if we left symlinks for the config files? Or perhaps even better,
> provide a tool that will create them for people that actually need
> them.

See the thread around
http://archives.postgresql.org/message-id/20160826104446.n3cif4m7modslkrs%40msg.df7cb.de


Right, but I was referring only to the config files. AFAIK it should be 
safe for those to be symlinks.

--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)   mobile: 512-569-9461


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Michael Paquier]

On Mon, Aug 29, 2016 at 9:39 PM, Craig Ringer
 wrote:
> Cool. I'll mark as waiting on author pending that.
>
> It'll be good to get this footgun put away.

OK, so done. I have put the renaming of pg_xlog to pg_wal on top patch
stack as that's the one making no discussion it seems: a lot of people
like pg_wal. I have added as well handling for the renaming in
pg_basebackup by using PQserverVersion. One thing to note is that a
connection needs to be made to the target server *before* creating the
soft link of pg_xlog/pg_wal because we need to know the version of the
target server. pg_upgrade is handled as well. And that's all in 0001.

Patch 0002 does pg_clog -> pg_trans, "trans" standing for
"transaction". Switching to pg_trans_status or pg_xact_status is not
that complicated to change anyway...
-- 
Michael
From 9806c07cbb5bd31946b48274dde8b5db65aa6169 Mon Sep 17 00:00:00 2001
From: Michael Paquier 
Date: Mon, 29 Aug 2016 15:05:46 +0900
Subject: [PATCH 2/2] Rename pg_clog to pg_trans

pg_upgrade is updated to handle the transfer correctly to post-10
clusters.
---
 doc/src/sgml/backup.sgml   |  4 ++--
 doc/src/sgml/catalogs.sgml |  4 ++--
 doc/src/sgml/config.sgml   |  2 +-
 doc/src/sgml/func.sgml |  2 +-
 doc/src/sgml/maintenance.sgml  |  8 
 doc/src/sgml/ref/pg_resetxlog.sgml |  4 ++--
 doc/src/sgml/ref/pg_rewind.sgml|  2 +-
 doc/src/sgml/storage.sgml  | 10 +-
 doc/src/sgml/wal.sgml  |  2 +-
 src/backend/access/heap/heapam.c   |  4 ++--
 src/backend/access/transam/README  | 12 ++--
 src/backend/access/transam/clog.c  |  2 +-
 src/backend/access/transam/commit_ts.c |  2 +-
 src/backend/access/transam/multixact.c |  2 +-
 src/backend/access/transam/subtrans.c  |  4 ++--
 src/backend/access/transam/transam.c   |  2 +-
 src/backend/access/transam/twophase.c  |  4 ++--
 src/backend/access/transam/xact.c  | 18 +-
 src/backend/access/transam/xlog.c  |  2 +-
 src/backend/commands/vacuum.c  | 10 +-
 src/backend/postmaster/autovacuum.c|  2 +-
 src/backend/storage/buffer/README  |  2 +-
 src/backend/storage/ipc/procarray.c|  4 ++--
 src/backend/utils/time/tqual.c |  6 +++---
 src/bin/initdb/initdb.c|  2 +-
 src/bin/pg_upgrade/exec.c  |  6 ++
 src/bin/pg_upgrade/pg_upgrade.c| 30 ++
 src/include/access/slru.h  |  4 ++--
 28 files changed, 84 insertions(+), 72 deletions(-)

diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml
index 074a6b0..1fa4d0e 100644
--- a/doc/src/sgml/backup.sgml
+++ b/doc/src/sgml/backup.sgml
@@ -382,10 +382,10 @@ tar -cf backup.tar /usr/local/pgsql/data
   directories. This will not work because the
   information contained in these files is not usable without
   the commit log files,
-  pg_clog/*, which contain the commit status of
+  pg_trans/*, which contain the commit status of
   all transactions. A table file is only usable with this
   information. Of course it is also impossible to restore only a
-  table and the associated pg_clog data
+  table and the associated pg_trans data
   because that would render all other tables in the database
   cluster useless.  So file system backups only work for complete
   backup and restoration of an entire database cluster.
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 4e09e06..783d49c 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -1847,7 +1847,7 @@
All transaction IDs before this one have been replaced with a permanent
(frozen) transaction ID in this table.  This is used to track
whether the table needs to be vacuumed in order to prevent transaction
-   ID wraparound or to allow pg_clog to be shrunk.  Zero
+   ID wraparound or to allow pg_trans to be shrunk.  Zero
(InvalidTransactionId) if the relation is not a table.
   
  
@@ -2514,7 +2514,7 @@
All transaction IDs before this one have been replaced with a permanent
(frozen) transaction ID in this database.  This is used to
track whether the database needs to be vacuumed in order to prevent
-   transaction ID wraparound or to allow pg_clog to be shrunk.
+   transaction ID wraparound or to allow pg_trans to be shrunk.
It is the minimum of the per-table
pg_class.relfrozenxid values.
   
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 8e0bccc..dff2945 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -5816,7 +5816,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
 

 Vacuum also allows removal of old files from the
-pg_clog subdirectory, which is why the default
+pg_trans 

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Peter Eisentraut]

On 8/28/16 8:45 PM, Jim Nasby wrote:
> People accidentally blowing away pg_clog or pg_xlog is a pretty common 
> occurrence, and I don't think there's all that many tools that reference 
> them. I think it's well worth renaming them.

I think a related problem is that the default log directory is called
"pg_log", which is very similar to those other two.  There is no quick
way to tell their meaning apart.

I would consider changing the default from "pg_log" to "log".  Then we'd
also be at the point where we can say, everything starting with "pg_" is
internal, don't touch it.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Peter Eisentraut]

On 8/29/16 12:54 PM, Robert Haas wrote:
> As for the new names, how about pg_wal and
> pg_xact?  I don't think "pg_trans" is as good; is it transactional or
> transient or transport or transitive or what?

pg_transaction_status? (or pg_xact_status if you want to keep it short)

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Andres Freund]

On 2016-08-29 19:27:29 -0500, Jim Nasby wrote:
> On 8/27/16 1:11 PM, Tom Lane wrote:
> > Alvaro Herrera  writes:
> > > I'm for renaming too, but I'd go with Peter E's suggestion: move pg_xlog
> > > to something like $PGDATA/var/wal or $PGDATA/srv/wal or something like 
> > > that.
> > 
> > I think that would make sense if we were to relocate *everything* under
> > PGDATA into some FHS-like subdirectory structure.  But I'm against moving
> > the config files for previously-stated reasons, and I doubt it makes sense
> > to adopt an FHS-like structure only in part.
> 
> What if we left symlinks for the config files? Or perhaps even better,
> provide a tool that will create them for people that actually need
> them.

See the thread around
http://archives.postgresql.org/message-id/20160826104446.n3cif4m7modslkrs%40msg.df7cb.de


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Jim Nasby]

On 8/27/16 1:11 PM, Tom Lane wrote:

Alvaro Herrera  writes:

I'm for renaming too, but I'd go with Peter E's suggestion: move pg_xlog
to something like $PGDATA/var/wal or $PGDATA/srv/wal or something like that.


I think that would make sense if we were to relocate *everything* under
PGDATA into some FHS-like subdirectory structure.  But I'm against moving
the config files for previously-stated reasons, and I doubt it makes sense
to adopt an FHS-like structure only in part.


What if we left symlinks for the config files? Or perhaps even better, 
provide a tool that will create them for people that actually need them.

--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)   mobile: 512-569-9461


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Bruce Momjian]

On Fri, Aug 26, 2016 at 05:14:36PM +0200, Magnus Hagander wrote:
> On Aug 26, 2016 5:13 PM, "Joshua D. Drake"  wrote:
> >
> > On 08/25/2016 07:39 PM, Michael Paquier wrote:
> >>
> >> Hi all,
> >>
> >> I am relaunching $subject as 10 development will begin soon. As far as
> >> I know, there is agreement that we can do something here. Among the
> >> different proposals I have found:
> >> - pg_clog renamed to pg_commit_status, pg_xact or pg_commit
> >> - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal
> >
> >
> > I think the use of the pg_ prefix is redundant. Just a directory called: wal
> will do (for example).
> >
> > In reference to pg_xlog specifically, it should be wal. It is the Write 
> > Ahead
> Log, not the Journal (although I recognize they can be synonymous). All the
> documentation talks about Write Ahead Log.
> >
> 
> Yes, let's avoid inventing a *third* name for it, please. It's already bad
> enough that we have both wal and xlog.

As far as removing 'pg_', consider that many users place pg_xlog on a
different device, so using "wal" would not be clear it was related to
Postgres. Perhaps we can have initdb --xlogdir use pg_wal, and maybe
document this suggestion.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Joshua D. Drake]

On 08/29/2016 10:00 AM, Daniel Verite wrote:


Let's imagine that pg_xlog is named wal instead.
How does that help our user with the disk space problem?
Does that point to a path of resolution? I don't see it.
What do we think that user's next move will be?
After all, WAL means Write Ahead *Log*.


If, they look it up (which they would likely have to) they will see it 
isn't removable. :D That is the point I am making. If it is called xlog 
the brain says "logs". If it says wal the brain says, "What is wal?"


At that point they have to look it up and then if they still delete it; 
well that is what emergency rates are for :P


Sincerely,

JD



--
Command Prompt, Inc.  http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
Unless otherwise stated, opinions are my own.


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Daniel Verite]

Joshua D. Drake wrote:

> You log in, see that all the space and you find that you are using a
> ton of disk space. You look around for anything you can delete. You
> find a directory called pg_xlog, it says log, junior ignorant, don't
> want to be a sysadmin 101 says, "delete logs".

Yes, it happens. I don't deny the problem, but I'm wondering
about the wishful thinking we're possibly falling into here
concerning the solution.

Let's imagine that pg_xlog is named wal instead.
How does that help our user with the disk space problem?
Does that point to a path of resolution? I don't see it.
What do we think that user's next move will be?
After all, WAL means Write Ahead *Log*.

On the other hand, by decommissioning pg_xlog as a name,
that makes it obsolete in all presentations, tutorials, docs
that refer to that directory, and there are many of them.
There are years of confusion ahead with questions like
"where is that pg_xlog directory that I'm supposed to
monitor, or move into a different partition, etc...",
for a benefit that remains to be seen.

Best regards,
-- 
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Robert Haas]

On Sat, Aug 27, 2016 at 2:03 PM, Michael Paquier
 wrote:
> - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
> David's input),  Magnus

I'm in favor of this.  But I don't like Peter's proposal to use a more
complicated structure.  As for the new names, how about pg_wal and
pg_xact?  I don't think "pg_trans" is as good; is it transactional or
transient or transport or transitive or what?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Fujii Masao]

On Sat, Aug 27, 2016 at 5:33 PM, Michael Paquier
 wrote:
> On Sat, Aug 27, 2016 at 6:34 AM, Andres Freund  wrote:
>> On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote:
>>> I agree with all that.  But the subject line is specifically about
>>> moving pg_xlog.  So if your opinion is that we shouldn't move pg_xlog,
>>> then that is noted.  But if we were to move it, we can think about a
>>> good place to move it to.
>>
>> I think it's probably worth moving pg_xlog, because the benefit also
>> includes preventing a few users from shooting themselves somewhere
>> vital. That's imo much less the case for some of the other moves.  But I
>> still don't think think a largescale reorganization is a good idea,
>> it'll just stall and nothing will happen.
>
> OK, so let's focus only on the renaming mentioned in $subject. So far
> as I can see on this thread, here are the opinions of people who
> clearly gave one:
> - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
> David's input),  Magnus
> - Rename them, hard break not OK: Fujii-san (perhaps do nothing?)
> - Do nothing: Simon (add a README), Tom, Peter E

I vote for "do nothing".
First of all, I can't have much hope for that renaming the directories really
prevents "careless" users from wrongly deleting the important files.

Regards,

-- 
Fujii Masao


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Joshua D. Drake]

On 08/29/2016 08:07 AM, Tom Lane wrote:

"Joshua D. Drake"  writes:

Also as a note to the idea that we make break things for external user
space; the next version being v10 is the exact time to do that.


Let's please drop this meme that "v10 is a great time to break things".
We don't want this to be any worse than any other major-version upgrade.


The moment you break backward compatibility, it will be worse. We are 
talking about breaking backward compatibility. So let's just accept it 
as it is, there is a mentality about a major jump. A major jump 
(9.6->10) is *the* perfect time to make world changing, changes.



If we throw thirty different major incompatibilities in at once, we're
going to be hearing about how painful it was for the next decade, even if
any one of them individually would have been manageable.  Or, if we make
the pain factor too high, users will simply not upgrade, and we'll be
faced with demands that we support 9.6 forever.


Whiners always find a reason to whine.

Let's be on two feet here. I am not saying we should jump to using json 
notation for our next release. I am simply stating that any largish 
(even if it is a small patch) changes to expected behavior should be 
done with care. We have a window because no matter how much you yell, I 
yell, Magnus yells, or anybody else yells; the telling story will be, 
"10.0 is a huge jump from 9.6". Most people *WOULD NOT CARE* if we only 
changed one thing. They care that we jumped 4 releases. That type of 
communication implies BIG CHANGES.


Whether you like it or not. Whether I like it or not.

Sincerely,

JD



--
Command Prompt, Inc.  http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
Unless otherwise stated, opinions are my own.


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Andres Freund]

On 2016-08-29 12:07:51 -0400, David Steele wrote:
> >> pg_replslot -> pg_tmp/pg_repslot
> > 
> > That's most certainly not ephemeral. Just because something isn't
> > generally appropriate on a standby, doesn't, by far, mean it's ephemeral.
> 
> Yes, ephemeral was a poor choice of words.  I really meant "can be
> excluded from backups".

Then it most certainly can't be called pg_tmp, that'll just invite
people to rm -rf it. And then they'll be screwed.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[David Steele]

On 8/29/16 11:35 AM, Andres Freund wrote:
> On 2016-08-29 11:18:38 -0400, David Steele wrote:
>> pg_logical -> pg_replgcl
> 
> That seems considerably worse.

Fair enough.  I was just trying to throw something out there to get rid
of the "log" in logical.

>> pg_replslot -> pg_tmp/pg_repslot
> 
> That's most certainly not ephemeral. Just because something isn't
> generally appropriate on a standby, doesn't, by far, mean it's ephemeral.

Yes, ephemeral was a poor choice of words.  I really meant "can be
excluded from backups".

-- 
-David
da...@pgmasters.net


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Tom Lane]

Andres Freund  writes:
> On 2016-08-29 11:18:38 -0400, David Steele wrote:
>> pg_replslot -> pg_tmp/pg_repslot

> That's most certainly not ephemeral. Just because something isn't
> generally appropriate on a standby, doesn't, by far, mean it's ephemeral.

Do we need to account for both of those concepts explicitly?

The original point here was to simplify figuring out what needed to be
copied in base backups.  I'm not sure whether we really need a marker
for what's going to be flushed at restart.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Andres Freund]

Hi,

On 2016-08-29 11:18:38 -0400, David Steele wrote:
> pg_logical -> pg_replgcl

That seems considerably worse.

> pg_replslot -> pg_tmp/pg_repslot

That's most certainly not ephemeral. Just because something isn't
generally appropriate on a standby, doesn't, by far, mean it's ephemeral.

Greetings,

Andres Freund


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[David Steele]

On 8/27/16 4:33 AM, Michael Paquier wrote:
> On Sat, Aug 27, 2016 at 6:34 AM, Andres Freund  wrote:
>> On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote:
>>> I agree with all that.  But the subject line is specifically about
>>> moving pg_xlog.  So if your opinion is that we shouldn't move pg_xlog,
>>> then that is noted.  But if we were to move it, we can think about a
>>> good place to move it to.
>>
>> I think it's probably worth moving pg_xlog, because the benefit also
>> includes preventing a few users from shooting themselves somewhere
>> vital. That's imo much less the case for some of the other moves.  But I
>> still don't think think a largescale reorganization is a good idea,
>> it'll just stall and nothing will happen.
> 
> OK, so let's focus only on the renaming mentioned in $subject. So far
> as I can see on this thread, here are the opinions of people who
> clearly gave one:
> - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
> David's input),  Magnus
> - Rename them, hard break not OK: Fujii-san (perhaps do nothing?)
> - Do nothing: Simon (add a README), Tom, Peter E
> 
> As far as I can see, there is a consensus to not rename pg_xlog to
> pg_journal and avoid using a third meaning, but instead use pg_wal. I
> guess that now the other renaming would be pg_clog -> pg_xact. Other
> opinions? Forgot you here?

I'm definitely +1 for a hard break (internal links are a pain).  Ideally
I'd would like to see:

pg_xlog -> pg_wal
pg_clog -> pg_xact
pg_logical -> pg_replgcl

pg_logical is a special case because it contains both ephemeral and
persistent files.  I'd like to see the temporary files in
pg_tmp/pg_replgcl (or whatever it gets renamed to) but that means that
pg_tmp must be on the same device to get atomic renames.  It would be
enough if pg_replgcl had a pgsql_tmp subdirectory so temp files are
excluded from backups with the current logic.

I'm also in favor of leaving configuration files where they are because
these are the files that are most likely to be checked/manipulated in
various user scripts (other than pg_xlog) of course.

As Stephen mentioned I would like to see purely ephemeral data moved
into its own directory.  Maybe $PGDATA/pg_tmp?

pg_subtrans -> pg_tmp/pg_subxact
pg_stat_tamp -> pg_tmp/pg_stat
pg_dynshmem -> pg_tmp/pg_dynshmem (or pg_shmem?)
pg_notify -> pg_tmp/pg_notify
pg_replslot -> pg_tmp/pg_repslot
pg_serial -> pg_tmp/pg_serial
pg_snapshots -> pg_tmp/pg_snapshot

It would be helpful if we had consistent temp directory naming
everywhere so it's clear what to exclude when it is not practical to put
files in the root pg_tmp.  As such I would rename pgsql_tmp to pg_tmp in
base and pg_tblspc/*/*/.

-- 
-David
da...@pgmasters.net


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Tom Lane]

"Joshua D. Drake"  writes:
> Also as a note to the idea that we make break things for external user 
> space; the next version being v10 is the exact time to do that.

Let's please drop this meme that "v10 is a great time to break things".
We don't want this to be any worse than any other major-version upgrade.
If we throw thirty different major incompatibilities in at once, we're
going to be hearing about how painful it was for the next decade, even if
any one of them individually would have been manageable.  Or, if we make
the pain factor too high, users will simply not upgrade, and we'll be
faced with demands that we support 9.6 forever.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Joshua D. Drake]

On 08/29/2016 06:42 AM, Daniel Verite wrote:


Aside from that, we might also question how much of the excuse
"pg_xlog looked like it was just deletable logs" is a lie made up
after the fact, because anybody wrecking a database is not against
deflecting a bit of the blame to the software, that's human.

The truth being that they took the gamble that postgres
will somehow recover from the deletion, at the risk of possibly
loosing the latest transactions. That doesn't necessarily look
so bad when current transactions are failing anyway for lack of disk
space, users are yelling at you, and you're frantically searching for
anything to delete in the FS to come back online quickly.
Personally I'm quite skeptical of the *name* of the directory
playing that much of a role in this scenario.


Oh it makes perfect sense.

User who isn't a postgres/database person installs X software. X 
software uses PostgreSQL and you have read on the intertubes about how 
Postgres is awesome. So you install it, get everything up and running 
from the README.md on Github. You walk away.


A year later, after it becomes crucial to whatever it is you do the 
system crashes. You have no idea what is going on, you just set this up 
on some recycled server or VM. You log in, see that all the space and 
you find that you are using a ton of disk space. You look around for 
anything you can delete. You find a directory called pg_xlog, it says 
log, junior ignorant, don't want to be a sysadmin 101 says, "delete logs".


Boom, crushed server. Let us not forget that by far the number of 
PostgreSQL users we have, have never done ANYTHING with postgres except 
make it so something like Drupal can connect to it.


JD






Best regards,




--
Command Prompt, Inc.  http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Joshua D. Drake]

On 08/29/2016 12:04 AM, Magnus Hagander wrote:

On Mon, Aug 29, 2016 at 2:45 AM, Jim Nasby > wrote:

On 8/26/16 4:08 PM, Andres Freund wrote:

Splitting of ephemeral data seems to have a benefit, the rest
seems more
like rather noisy busywork to me.


People accidentally blowing away pg_clog or pg_xlog is a pretty
common occurrence, and I don't think there's all that many tools
that reference them. I think it's well worth renaming them.


Pretty sure every single backup tool or script out there is referencing
pg_xlog. If it's not, then it's broken...


No, not really. Consider a filesytem backup using archiving and base 
backups. It doesn't care one lick about pg_xlog. And I guarantee you 
that there are tons of people running a backup like that considering the 
same script would work all the way back to 8.2 (.1?).


Sincerely,

JD

--
Command Prompt, Inc.  http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Joshua D. Drake]

On 08/27/2016 11:11 AM, Tom Lane wrote:

Alvaro Herrera  writes:

I'm for renaming too, but I'd go with Peter E's suggestion: move pg_xlog
to something like $PGDATA/var/wal or $PGDATA/srv/wal or something like that.


I think that would make sense if we were to relocate *everything* under
PGDATA into some FHS-like subdirectory structure.  But I'm against moving
the config files for previously-stated reasons, and I doubt it makes sense
to adopt an FHS-like structure only in part.


I think that is a very reasonable suggestion.

Also as a note to the idea that we make break things for external user 
space; the next version being v10 is the exact time to do that.


JD



regards, tom lane





--
Command Prompt, Inc.  http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Daniel Verite]

Craig Ringer wrote:

> People won't see a README in amongst 5000 xlog segments while
> freaking out about the sever being down.

Maybe they're more likely to google "pg_xlog".
BTW, renaming it will not help with respect to that, as
there's a pretty good corpus on knowledge linked to that
particular keyword.

The first google results of "pg_xlog" are, for me:

- Solving pg_xlog out of disk space problem on Postgres 
- PostgreSQL: Documentation: 9.1: WAL Internals
- Why is my pg_xlog directory so huge? - PostgreSQL
- Database Soup: Don't delete pg_xlog
...

That's spot-on. For each person deleting stuff in pg_xlog and then
regretting it, how many look it up in the above results and avoid
making the mistake? Will they find comparable results if the
directory is "wal" ?

Aside from that, we might also question how much of the excuse
"pg_xlog looked like it was just deletable logs" is a lie made up
after the fact, because anybody wrecking a database is not against
deflecting a bit of the blame to the software, that's human.

The truth being that they took the gamble that postgres
will somehow recover from the deletion, at the risk of possibly
loosing the latest transactions. That doesn't necessarily look
so bad when current transactions are failing anyway for lack of disk
space, users are yelling at you, and you're frantically searching for
anything to delete in the FS to come back online quickly.
Personally I'm quite skeptical of the *name* of the directory
playing that much of a role in this scenario.

Best regards,
-- 
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Craig Ringer]

On 29 August 2016 at 20:28, Michael Paquier  wrote:
> On Mon, Aug 29, 2016 at 5:28 PM, Craig Ringer
>  wrote:
>> On 29 August 2016 at 14:30, Michael Paquier  
>> wrote:
>>> On Mon, Aug 29, 2016 at 2:36 PM, Craig Ringer
>>>  wrote:
 I don't care if it comes as part of some greater reorg or not but I'll be
 really annoyed if scope creep lands up killing the original proposal to 
 just
 rename these dirs. I think that a simple rename should be done first. Then
 if some greater reorg is to be done it can be done shortly after. The only
 people that'll upset are folks tracking early 10.0 dev and they'll be aware
 it's coming.
>>>
>>> Okay, so let's do it. Attached are two patches:
>>> - 0001 renames pg_clog to pg_trans. I have let clog.c with its current
>>> name, as well as its structures. That's the mechanical patch, the ony
>>> interesting part being in pg_upgrade.
>>> - 0002 renames pg_xlog to pg_wal.
>>
>> Is there any expectation that a 10.0 pg_basebackup should work on a
>> 9.x server, or fail to work gracefully? There doesn't look to be any
>> version specific handling of the rename there.
>
> Oops. Per the docs:
> pg_basebackup works with servers of the same or an older major
> version, down to 9.1. However, WAL streaming mode (-X stream) only
> works with server version 9.3 and later, and tar format mode
> (--format=tar) of the current version only works with server version
> 9.5 or later.
>
> So we need to do a bit better than what the patch proposes, but that's
> actually just tweaking the things with pg_xlog/pg_wal depending on the
> version of the target server.

Cool. I'll mark as waiting on author pending that.

It'll be good to get this footgun put away.

-- 
 Craig Ringer   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Michael Paquier]

On Mon, Aug 29, 2016 at 5:28 PM, Craig Ringer
 wrote:
> On 29 August 2016 at 14:30, Michael Paquier  wrote:
>> On Mon, Aug 29, 2016 at 2:36 PM, Craig Ringer
>>  wrote:
>>> I don't care if it comes as part of some greater reorg or not but I'll be
>>> really annoyed if scope creep lands up killing the original proposal to just
>>> rename these dirs. I think that a simple rename should be done first. Then
>>> if some greater reorg is to be done it can be done shortly after. The only
>>> people that'll upset are folks tracking early 10.0 dev and they'll be aware
>>> it's coming.
>>
>> Okay, so let's do it. Attached are two patches:
>> - 0001 renames pg_clog to pg_trans. I have let clog.c with its current
>> name, as well as its structures. That's the mechanical patch, the ony
>> interesting part being in pg_upgrade.
>> - 0002 renames pg_xlog to pg_wal.
>
> Is there any expectation that a 10.0 pg_basebackup should work on a
> 9.x server, or fail to work gracefully? There doesn't look to be any
> version specific handling of the rename there.

Oops. Per the docs:
pg_basebackup works with servers of the same or an older major
version, down to 9.1. However, WAL streaming mode (-X stream) only
works with server version 9.3 and later, and tar format mode
(--format=tar) of the current version only works with server version
9.5 or later.

So we need to do a bit better than what the patch proposes, but that's
actually just tweaking the things with pg_xlog/pg_wal depending on the
version of the target server.

> The patch does not update the translations. I wonder if it's worth
> doing so to save translators the hassle by sed'ing the following
> lines:
> sed -i 's/\/pg_wal/g' src/backend/po/*.po
> ?

Yes I was wondering about that... But concluded that normally the
translation updates would just do it. It is not complicated to update
if need be anyway.

> src/backend/access/transam/README should probably have a note about the 
> rename.

Good idea.

> Looks like changes in pg_upgrade for clog are a bit more than the
> mechanical changes elsewhere, but seem sensible to me. I haven't yet
> done a test pg_upgrade run.

I have tested that FWIW using 10devel -> 10devel, 9.5 -> 10devel, 9.4
-> 10devel. There are versions of 10devel using pg_xlog and others
pg_wal if this patch is introduced. I am aware of the fact that
pg_upgrade supports upgrades for the same major version though it
seems to me that we would not want to support this scenario, which is
why it would be good to get this patch at the beginning of the dev
cycle.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Petr Jelinek]

On 27/08/16 18:50, Tom Lane wrote:

Michael Paquier  writes:

OK, so let's focus only on the renaming mentioned in $subject. So far
as I can see on this thread, here are the opinions of people who
clearly gave one:
- Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
David's input),  Magnus
- Rename them, hard break not OK: Fujii-san (perhaps do nothing?)
- Do nothing: Simon (add a README), Tom, Peter E


I'm against moving/renaming the
configuration files, because I think that will break a lot of users'
scripts and habits without really buying much.


I don't agree with this part, mainly because there is significant number 
of installations (everything that uses debian/ubuntu scripts) that 
already don't have configuration files inside the data directory.


On the other matters:
+1 for renaming pg_xlog to pg_wal and pg_clog to pg_xact/trans (don't 
care really which one)


And +1 for renaming pg_logical to something more reasonable.

--
  Petr Jelinek  http://www.2ndQuadrant.com/
  PostgreSQL Development, 24x7 Support, Training & Services


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Renaming of pg_xlog and pg_clog

[Craig Ringer]

On 29 August 2016 at 14:30, Michael Paquier  wrote:
> On Mon, Aug 29, 2016 at 2:36 PM, Craig Ringer
>  wrote:
>> I don't care if it comes as part of some greater reorg or not but I'll be
>> really annoyed if scope creep lands up killing the original proposal to just
>> rename these dirs. I think that a simple rename should be done first. Then
>> if some greater reorg is to be done it can be done shortly after. The only
>> people that'll upset are folks tracking early 10.0 dev and they'll be aware
>> it's coming.
>
> Okay, so let's do it. Attached are two patches:
> - 0001 renames pg_clog to pg_trans. I have let clog.c with its current
> name, as well as its structures. That's the mechanical patch, the ony
> interesting part being in pg_upgrade.
> - 0002 renames pg_xlog to pg_wal.

Is there any expectation that a 10.0 pg_basebackup should work on a
9.x server, or fail to work gracefully? There doesn't look to be any
version specific handling of the rename there.

Otherwise looks good.

Doesn't upset 'make check' or the TAP recovery suite.

Works:

src/test/regress/tmp_check/data $ ls
basepg_commit_ts  pg_hba.confpg_logicalpg_notify
pg_serial pg_stat  pg_subtrans  pg_trans PG_VERSION
postgresql.auto.conf  postmaster.opts
global  pg_dynshmem   pg_ident.conf  pg_multixact  pg_replslot
pg_snapshots  pg_stat_tmp  pg_tblspcpg_twophase  pg_wal
postgresql.conf

It leaves pg_xlogfilename etc with original names as it IMO should.
There's no need to break more than we have to and it's still the xlog.

The documentation on backup/restore might benefit from a note saying
that pg_wal was named pg_xlog prior to 10.0 so tools intended to work
on older versions should check PG_VERSION. Though in practice most
people who write new tools will target 10.0+ and people maintaining
older tools will know, so it's not a big deal.

I don't know if the renaming in XLogFileRead of XLOG_FROM_PG_XLOG =>
XLOG_FROM_PG_WAL  is really necessary, but tend to think it's good
since that define explicitly refers to the directory name, not
transaction logs in general.

The patch does not update the translations. I wonder if it's worth
doing so to save translators the hassle by sed'ing the following
lines:

-errhint("The database server
will regularly poll the pg_xlog subdirectory to check for files placed
there.")));
+errhint("The database server
will regularly poll the pg_wal subdirectory to check for files placed
there.")));

-(errmsg("could not open directory
\"%s\": %m", "pg_xlog")));
+(errmsg("could not open directory
\"%s\": %m", "pg_wal")));

with

sed -i 's/\/pg_wal/g' src/backend/po/*.po

?



src/backend/access/transam/README should probably have a note about the rename.

Looks like changes in pg_upgrade for clog are a bit more than the
mechanical changes elsewhere, but seem sensible to me. I haven't yet
done a test pg_upgrade run.



-- 
-- 
 Craig Ringer   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers