Re: [HACKERS] [PATCH] add --progress option to pgbench (submission 3)

2013-06-01 Thread Fabien COELHO
New submission which put option help in alphabetical position, as per Peter Eisentraut f0ed3a8a99b052d2d5e0b6153a8907b90c486636 This is for reference to the next commitfest. Patch update after conflict induced by pg-indentation, for the next commitfest. -- Fabien.diff --git

Re: [HACKERS] detecting binary backup in progress

2013-06-01 Thread Michael Paquier
On Sat, Jun 1, 2013 at 4:55 AM, Alvaro Herrera alvhe...@2ndquadrant.comwrote: Heikki Linnakangas wrote: The manual says: pg_is_in_backup()boolTrue if an on-line exclusive backup is still in progress. So clearly that is intentional. That could use some rephrasing, though; a

Re: [HACKERS] detecting binary backup in progress

2013-06-01 Thread Andres Freund
On 2013-06-01 17:05:57 +0900, Michael Paquier wrote: On Sat, Jun 1, 2013 at 4:55 AM, Alvaro Herrera alvhe...@2ndquadrant.comwrote: Heikki Linnakangas wrote: The manual says: pg_is_in_backup()boolTrue if an on-line exclusive backup is still in progress. So clearly

[HACKERS] Combo xids

2013-06-01 Thread Simon Riggs
Currently, we hold xmin and xmax for each tuple. For xmax, we have the multixact mechanism that allows us to represent an array of xids with just a single pseudo xid. So why not hold xmin and xmax as part of a multixact? Instead of caching two xids on each tuple, why not hold just one and allow

[HACKERS] Deferring transaction wraparound

2013-06-01 Thread Simon Riggs
As we get faster, we'll need to wrap the xid counter more quickly. At current full speed, we could wrap the xid counter every 72 hours. This is a concern for very large tables, since currently we have to rescan the whole table. This has my interest for change in the next release. We could solve

[HACKERS] Optimising Foreign Key checks

2013-06-01 Thread Simon Riggs
FK checks can be expensive, especially when loading large volumes of data into an existing table or partition. A couple of ideas for improving performance are discussed here: 1. Use Case: Bulk loading COPY pgbench_accounts; -- references pgbench_branches with many repeated values Proposal:

Re: [HACKERS] [PATCH] pgbench --throttle (submission 7 - with lag measurement)

2013-06-01 Thread Fabien COELHO
New submission for the next commit fest. This new version also reports the average lag time, i.e. the delay between scheduled and actual transaction start times. This may help detect whether things went smothly, or if at some time some delay was introduced because of the load and some

Re: [HACKERS] Combo xids

2013-06-01 Thread Martijn van Oosterhout
On Sat, Jun 01, 2013 at 09:22:05AM +0100, Simon Riggs wrote: When would this make sense? Frequently. Most of the time a tuple needs only one xid set. In most cases, we set xmin and xmax a long time apart. Very few cases end with both of them set inside the *same* xmin horizon. In a heavy

Re: [HACKERS] detecting binary backup in progress

2013-06-01 Thread Christoph Moench-Tegeder
## Joe Conway (m...@joeconway.com): However there is a period of time after pg_start_backup() is first executed to when it completes, during which backup_label file does not exist yet, but the backup has essentially been started. Is there any way to detect this state? When I did some

[HACKERS] Which table stored in which file in PGDATA/base/[db-oid]

2013-06-01 Thread Soroosh Sardari
Dear Hackers I've created a new DB, and a bunch of files created in base/12054, 12054 is oid of the new DB. I want to find what table stored in each file. BTW, I read this http://www.postgresql.org/docs/9.2/interactive/storage-file-layout.html I have 156 files with numerical names, vm and fsm

Re: [HACKERS] Which table stored in which file in PGDATA/base/[db-oid]

2013-06-01 Thread Martijn van Oosterhout
On Sat, Jun 01, 2013 at 03:11:50PM +0430, Soroosh Sardari wrote: Dear Hackers I've created a new DB, and a bunch of files created in base/12054, 12054 is oid of the new DB. I want to find what table stored in each file. BTW, I read this

Re: [HACKERS] Which table stored in which file in PGDATA/base/[db-oid]

2013-06-01 Thread Soroosh Sardari
Yes, I have some files which is not in pg_class.relfilenode of any table or index. I want to know which table or index stored in such files. From that page: Each table and index is stored in a separate file. For ordinary relations, these files are named after the table or index's filenode

Re: [HACKERS] Which table stored in which file in PGDATA/base/[db-oid]

2013-06-01 Thread Martijn van Oosterhout
On Sat, Jun 01, 2013 at 03:27:40PM +0430, Soroosh Sardari wrote: Yes, I have some files which is not in pg_class.relfilenode of any table or index. I want to know which table or index stored in such files. That shouldn't happen. Are you sure you're looking in the right database? Kan you list

Re: [HACKERS] Which table stored in which file in PGDATA/base/[db-oid]

2013-06-01 Thread Victor Yegorov
2013/6/1 Martijn van Oosterhout klep...@svana.org On Sat, Jun 01, 2013 at 03:27:40PM +0430, Soroosh Sardari wrote: Yes, I have some files which is not in pg_class.relfilenode of any table or index. I want to know which table or index stored in such files. That shouldn't happen. Are you

Re: [HACKERS] Which table stored in which file in PGDATA/base/[db-oid]

2013-06-01 Thread Andres Freund
On 2013-06-01 13:04:55 +0200, Martijn van Oosterhout wrote: On Sat, Jun 01, 2013 at 03:27:40PM +0430, Soroosh Sardari wrote: Yes, I have some files which is not in pg_class.relfilenode of any table or index. I want to know which table or index stored in such files. That shouldn't happen.

Re: [HACKERS] Which table stored in which file in PGDATA/base/[db-oid]

2013-06-01 Thread Soroosh Sardari
Yes, I'm sure. Difference of filenodes and new files changed w.r.t my first mail, because I added a table. I attached 3 files, newfile.pg : list of numerical files in base/[db-oid], ls | grep '[[:digit:]]\' filenode.pg : select distinct relfilenode from pg_class newfile-filenode.pg : Set of oids

Re: [HACKERS] detecting binary backup in progress

2013-06-01 Thread Simon Riggs
On 31 May 2013 21:06, Andres Freund and...@2ndquadrant.com wrote: On 2013-05-31 22:53:14 +0300, Heikki Linnakangas wrote: On 31.05.2013 22:36, Andres Freund wrote: On 2013-05-31 22:29:45 +0300, Heikki Linnakangas wrote: Note that pg_is_in_backup() just checks for presence of

Re: [HACKERS] Freezing without write I/O

2013-06-01 Thread Simon Riggs
On 30 May 2013 14:33, Heikki Linnakangas hlinnakan...@vmware.com wrote: Since we're bashing around ideas around freezing, let me write down the idea I've been pondering and discussing with various people for years. I don't think I invented this myself, apologies to whoever did for not giving

Re: [HACKERS] Freezing without write I/O

2013-06-01 Thread Simon Riggs
On 30 May 2013 19:39, Robert Haas robertmh...@gmail.com wrote: On Thu, May 30, 2013 at 9:33 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: The reason we have to freeze is that otherwise our 32-bit XIDs wrap around and become ambiguous. The obvious solution is to extend XIDs to 64 bits,

Re: [HACKERS] Running pgindent

2013-06-01 Thread Stephen Frost
* Bruce Momjian (br...@momjian.us) wrote: OK. Done. Thanks, Stephen signature.asc Description: Digital signature

Re: [HACKERS] detecting binary backup in progress

2013-06-01 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: The way to resolve this is to have two functions: pg_is_in_backup() - which covers both/all kinds of backup pg_is_in_exclusive_backup() - which covers just the exclusive backup mode What will you do with pg_backup_start_time()?

[HACKERS] Placing hints in line pointers

2013-06-01 Thread Simon Riggs
Notes on a longer term idea... An item pointer (also called line pointer) is used to allow an external pointer to an item, while allowing us to place the tuple that anywhere on the page. An ItemId is 4 bytes long and currently consists of (see src/include/storage/itemid.h)... typedef struct

Re: [HACKERS] pg_dump with postgis extension dumps rules separately

2013-06-01 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/31/2013 08:46 PM, Robert Haas wrote: On Wed, May 29, 2013 at 6:55 PM, Joe Conway m...@joeconway.com wrote: OK, simple enough. New patch attached. I still need to do some testing to verify this does not break anything, but other than that,

Re: [HACKERS] pg_dump with postgis extension dumps rules separately

2013-06-01 Thread Tom Lane
Joe Conway m...@joeconway.com writes: On 05/31/2013 08:46 PM, Robert Haas wrote: Changing SQL syntax in the back-branches isn't normally something we do, but I confess I don't see any real reason not to do it in this case. That was part of my hesitation, but I don't see any better way to fix

Re: [HACKERS] detecting binary backup in progress

2013-06-01 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/01/2013 02:43 AM, Christoph Moench-Tegeder wrote: ## Joe Conway (m...@joeconway.com): However there is a period of time after pg_start_backup() is first executed to when it completes, during which backup_label file does not exist yet, but

Re: [HACKERS] detecting binary backup in progress

2013-06-01 Thread Andres Freund
On 2013-06-01 08:11:26 -0700, Joe Conway wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/01/2013 02:43 AM, Christoph Moench-Tegeder wrote: ## Joe Conway (m...@joeconway.com): However there is a period of time after pg_start_backup() is first executed to when it completes,

Re: [HACKERS] pg_dump with postgis extension dumps rules separately

2013-06-01 Thread Andres Freund
On 2013-06-01 11:07:53 -0400, Tom Lane wrote: Joe Conway m...@joeconway.com writes: On 05/31/2013 08:46 PM, Robert Haas wrote: Changing SQL syntax in the back-branches isn't normally something we do, but I confess I don't see any real reason not to do it in this case. That was part of

Re: [HACKERS] pg_dump with postgis extension dumps rules separately

2013-06-01 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/01/2013 08:07 AM, Tom Lane wrote: Joe Conway m...@joeconway.com writes: On 05/31/2013 08:46 PM, Robert Haas wrote: Changing SQL syntax in the back-branches isn't normally something we do, but I confess I don't see any real reason not to do

Re: [HACKERS] detecting binary backup in progress

2013-06-01 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/01/2013 08:14 AM, Andres Freund wrote: On 2013-06-01 08:11:26 -0700, Joe Conway wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/01/2013 02:43 AM, Christoph Moench-Tegeder wrote: ## Joe Conway (m...@joeconway.com): However

Re: [HACKERS] detecting binary backup in progress

2013-06-01 Thread Andres Freund
On 2013-06-01 08:27:42 -0700, Joe Conway wrote: Uh. Why would you do the lock(file) thingy *after* calling pg_start_backup? You should do lock before calling start backup and remove the lock after calling stop backup. In that case I don't see where the race condition is? No, the point

Re: [HACKERS] pg_dump with postgis extension dumps rules separately

2013-06-01 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2013-06-01 11:07:53 -0400, Tom Lane wrote: I don't like this approach much. 1. It does nothing to fix the issue in *existing* databases, which won't have pg_depend entries like this. Well, you can now write an extension upgrade script that

Re: [HACKERS] pg_dump with postgis extension dumps rules separately

2013-06-01 Thread Andres Freund
On 2013-06-01 11:31:05 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2013-06-01 11:07:53 -0400, Tom Lane wrote: I don't like this approach much. 1. It does nothing to fix the issue in *existing* databases, which won't have pg_depend entries like this. Well,

Re: [HACKERS] pg_dump with postgis extension dumps rules separately

2013-06-01 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/01/2013 08:32 AM, Andres Freund wrote: On 2013-06-01 11:31:05 -0400, Tom Lane wrote: But in any case, making rules act differently from other table properties for this purpose seems seriously wrong. What's your proposal to fix this

Re: [HACKERS] pg_dump with postgis extension dumps rules separately

2013-06-01 Thread Tom Lane
Joe Conway m...@joeconway.com writes: I can look at having pg_dump ignore these entries, but I suspect that will be quite a bit more invasive. Actually, I believe the answer is just that getSchemaData() is doing things in the wrong order: if (g_verbose) write_msg(NULL, reading

Re: [HACKERS] pg_dump with postgis extension dumps rules separately

2013-06-01 Thread Tom Lane
I wrote: Actually, I believe the answer is just that getSchemaData() is doing things in the wrong order: BTW, I'm inclined to think it's also wrong that the getEventTriggers() call was just added at the end; those things are certainly not table subsidiary objects. I don't know if we allow

Re: [HACKERS] pg_dump with postgis extension dumps rules separately

2013-06-01 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/01/2013 09:39 AM, Tom Lane wrote: I wrote: Actually, I believe the answer is just that getSchemaData() is doing things in the wrong order: BTW, I'm inclined to think it's also wrong that the getEventTriggers() call was just added at the

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-01 Thread Simon Riggs
On 30 May 2013 19:48, Josh Berkus j...@agliodbs.com wrote: There's currently some great ideas bouncing around about eliminating the overhead associated with FREEZE. However, I wanted to take a step back and take a look at the big picture for VACUUM, FREEZE and ANALYZE. That is a very

Re: [HACKERS] getting rid of freezing

2013-06-01 Thread Simon Riggs
On 28 May 2013 15:15, Robert Haas robertmh...@gmail.com wrote: On Sat, May 25, 2013 at 6:14 AM, Simon Riggs si...@2ndquadrant.com wrote: I think the right way is actually to rethink and simplify all this complexity of Freezing/Pruning/Hinting/Visibility I agree, but I think that's likely to

Re: [HACKERS] detecting binary backup in progress

2013-06-01 Thread Simon Riggs
On 1 June 2013 15:45, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: The way to resolve this is to have two functions: pg_is_in_backup() - which covers both/all kinds of backup pg_is_in_exclusive_backup() - which covers just the exclusive backup mode What

Re: [HACKERS] Combo xids

2013-06-01 Thread Heikki Linnakangas
On 01.06.2013 11:22, Simon Riggs wrote: What is the benefit? Merging xmin/xmax would save 4 bytes per row. On servers with 8 byte word length, that means that we'd save 8 bytes per row for tables that have between 9 and 40 columns. Which is the majority of tables. Hmm, it would probably be

Re: [HACKERS] Combo xids

2013-06-01 Thread Simon Riggs
On 1 June 2013 19:25, Heikki Linnakangas hlinnakan...@vmware.com wrote: Hmm, it would probably be much easier to squeeze, say, one byte from the tuple header, than full four bytes. Then you could store store a null bitmap for upto 16 columns, without crossing the 24 byte mark. That would

Re: [HACKERS] [BUGS] COPY .... (FORMAT binary) syntax doesn't work

2013-06-01 Thread Simon Riggs
On 27 May 2013 15:31, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: On 26 May 2013 17:10, Tom Lane t...@sss.pgh.pa.us wrote: More readable would be to invent an intermediate nonterminal falling between ColId and ColLabel, whose expansion would be IDENT |

Re: [HACKERS] Freezing without write I/O

2013-06-01 Thread Heikki Linnakangas
On 31.05.2013 06:02, Robert Haas wrote: On Thu, May 30, 2013 at 2:39 PM, Robert Haasrobertmh...@gmail.com wrote: Random thought: Could you compute the reference XID based on the page LSN? That would eliminate the storage overhead. After mulling this over a bit, I think this is definitely

Re: [HACKERS] Freezing without write I/O

2013-06-01 Thread Simon Riggs
On 1 June 2013 19:48, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 31.05.2013 06:02, Robert Haas wrote: On Thu, May 30, 2013 at 2:39 PM, Robert Haasrobertmh...@gmail.com wrote: Random thought: Could you compute the reference XID based on the page LSN? That would eliminate the

Re: [HACKERS] Freezing without write I/O

2013-06-01 Thread Robert Haas
On Sat, Jun 1, 2013 at 2:48 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: We define a new page-level bit, something like PD_RECENTLY_FROZEN. When this bit is set, it means there are no unfrozen tuples on the page with XIDs that predate the current half-epoch. Whenever we know this to

Re: [HACKERS] Freezing without write I/O

2013-06-01 Thread Robert Haas
On Sat, Jun 1, 2013 at 3:22 PM, Simon Riggs si...@2ndquadrant.com wrote: If we set a bit, surely we need to write the page. Isn't that what we were trying to avoid? No, the bit only gets set in situations when we were going to dirty the page for some other reason anyway. Specifically, if a

Re: [HACKERS] Optimising Foreign Key checks

2013-06-01 Thread Noah Misch
On Sat, Jun 01, 2013 at 09:41:13AM +0100, Simon Riggs wrote: FK checks can be expensive, especially when loading large volumes of data into an existing table or partition. A couple of ideas for improving performance are discussed here: 1. Use Case: Bulk loading COPY pgbench_accounts; --

Re: [HACKERS] Vacuum, Freeze and Analyze: the big picture

2013-06-01 Thread Robert Haas
On Sat, Jun 1, 2013 at 1:49 PM, Simon Riggs si...@2ndquadrant.com wrote: On 30 May 2013 19:48, Josh Berkus j...@agliodbs.com wrote: There's currently some great ideas bouncing around about eliminating the overhead associated with FREEZE. However, I wanted to take a step back and take a look

Re: [HACKERS] Freezing without write I/O

2013-06-01 Thread Simon Riggs
On 1 June 2013 21:26, Robert Haas robertmh...@gmail.com wrote: On Sat, Jun 1, 2013 at 3:22 PM, Simon Riggs si...@2ndquadrant.com wrote: If we set a bit, surely we need to write the page. Isn't that what we were trying to avoid? No, the bit only gets set in situations when we were going to