Re: [HACKERS] Move unused buffers to freelist

2013-05-24 Thread Amit Kapila
On Thursday, May 23, 2013 8:45 PM Robert Haas wrote: On Tue, May 21, 2013 at 3:06 AM, Amit Kapila amit.kap...@huawei.com wrote: Here are the results. The first field in each line is the number of clients. The second number is the scale factor. The numbers after master and patched are the

Re: [HACKERS] Move unused buffers to freelist

2013-05-24 Thread Amit Kapila
On Friday, May 24, 2013 2:47 AM Jim Nasby wrote: On 5/14/13 2:13 PM, Greg Smith wrote: It is possible that we are told to put something in the freelist that is already in it; don't screw up the list if so. I don't see where the code does anything to handle that though. What was your

Re: [HACKERS] Patch to .gitignore

2013-05-24 Thread amul sul
 ctags and etags be part of postgres source tree and its generate some output inside them, so I think we must ignore it.  +1 Regards, Amul Sul -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] WARNING : pgstat wait timeout - Postgres 9.1

2013-05-24 Thread Mathieu Guerin
Hello, Thanks a lot for your answers. You should get it... stats_temp_directory | pg_stat_tmp | Writes temporary statistics files to the specified directory. I don't know why i don't get it. I am in 9.1 version... Moreover, when I mount pg_stat_tmp

Re: [HACKERS] WARNING : pgstat wait timeout - Postgres 9.1

2013-05-24 Thread Mathieu Guerin
About the stats_temp_directory, I didn't run as root... Now I'm sure the configurations are correct. I think, I have too much IO to use stats. I will ever have this message... Maybe I can disable this option. Do you know what it really impact ? Thanks. Math 2013/5/24 Mathieu Guerin

Re: [HACKERS] Removal of pageinspect--1.0.sql

2013-05-24 Thread Heikki Linnakangas
On 20.05.2013 19:50, Michael Paquier wrote: The contrib module pageinspect has been upgraded to 1.1, but pageinspect--1.0.sql is still present in source code. Shouldn't it be removed? Please find patch attached. Yep. Removed, thanks. - Heikki -- Sent via pgsql-hackers mailing list

Re: [HACKERS] WAL segments (names) not in a sequence

2013-05-24 Thread German Becker
Hi Sergey, Maybe I didn't explain correctly. I am using COPY/pg_dump/pg_restore for migration (and it is working fine). The streaming replication is for hot-standby replication *once migrated*. Thing is I disbable archving and set wal_level to minimal, when migrating the large portion of data, to

Re: [HACKERS] Patch to .gitignore

2013-05-24 Thread Tom Lane
=?ISO-8859-1?Q?Fabr=EDzio_de_Royes_Mello?= fabriziome...@gmail.com writes: On Fri, May 24, 2013 at 12:04 AM, Christopher Browne cbbro...@gmail.comwrote: There hasn't been general agreement on the merits of particular .gitignore rules of this sort. I agree with you about vim-oriented

Re: [HACKERS] WAL segments (names) not in a sequence

2013-05-24 Thread Amit Langote
Maybe I didn't explain correctly. I am using COPY/pg_dump/pg_restore for migration (and it is working fine). The streaming replication is for hot-standby replication *once migrated*. Thing is I disbable archving and set wal_level to minimal, when migrating the large portion of data, to make

Re: [HACKERS] Cost limited statements RFC

2013-05-24 Thread Robert Haas
On Thu, May 23, 2013 at 7:27 PM, Greg Smith g...@2ndquadrant.com wrote: I'm working on a new project here that I wanted to announce, just to keep from duplicating effort in this area. I've started to add a cost limit delay for regular statements. The idea is that you set a new

[HACKERS] background processes vs. hot standby

2013-05-24 Thread Robert Haas
CheckRequiredParameterValues() has some code that, when hot standby is in use, checks the values of max_connections, max_prepared_transactions, and max_locks_per_transaction against the master. The comment says we must have at least as many backend slots as the primary ... but the code no longer

Re: [HACKERS] background processes vs. hot standby

2013-05-24 Thread Andres Freund
Hi, On 2013-05-24 09:48:03 -0400, Robert Haas wrote: CheckRequiredParameterValues() has some code that, when hot standby is in use, checks the values of max_connections, max_prepared_transactions, and max_locks_per_transaction against the master. The comment says we must have at least as

Re: [HACKERS] Cost limited statements RFC

2013-05-24 Thread Jim Nasby
On 5/24/13 8:21 AM, Robert Haas wrote: On Thu, May 23, 2013 at 7:27 PM, Greg Smithg...@2ndquadrant.com wrote: I'm working on a new project here that I wanted to announce, just to keep from duplicating effort in this area. I've started to add a cost limit delay for regular statements. The

Re: [HACKERS] Move unused buffers to freelist

2013-05-24 Thread Jim Nasby
On 5/14/13 8:42 AM, Amit Kapila wrote: In the attached patch, bgwriter/checkpointer moves unused (usage_count =0 refcount = 0) buffer’s to end of freelist. I have implemented a new API StrategyMoveBufferToFreeListEnd() to move buffer’s to end of freelist. Instead of a separate function,

Re: [HACKERS] getting rid of freezing

2013-05-24 Thread Andres Freund
On 2013-05-23 22:09:02 -0400, Robert Haas wrote: On Thu, May 23, 2013 at 1:51 PM, Andres Freund and...@2ndquadrant.com wrote: So, what I propose instead is basically: 1) only vacuum non-all-visible pages, even when doing it for anti-wraparound Check. We might want an option to force a

Re: [HACKERS] pg_rewind, a tool for resynchronizing an old master after failover

2013-05-24 Thread Thom Brown
On 23 May 2013 10:03, Thom Brown t...@linux.com wrote: On 23 May 2013 07:10, Heikki Linnakangas hlinnakan...@vmware.com wrote: Hi, I've been hacking on a tool to allow resynchronizing an old master server after failover. The need to do a full backup/restore has been a common complaint ever

Re: [HACKERS] pg_rewind, a tool for resynchronizing an old master after failover

2013-05-24 Thread Andres Freund
On 2013-05-24 10:57:22 -0400, Thom Brown wrote: By the way, without any data inserted I get: thom@swift /tmp $ pg_rewind --target-pgdata=/tmp/primary --source-server='host=localhost port=5531 dbname=postgres' -v connected to remote server fetched file global/pg_control, length 8192 fetched

[HACKERS] Patch to add support of IF NOT EXISTS to others CREATE statements

2013-05-24 Thread Fabrízio de Royes Mello
Hi all, I working in a patch to include support of IF NOT EXISTS into CREATE statements that not have it yet. I started with DefineStmt section from src/backend/parser/gram.y: - CREATE AGGREGATE [ IF NOT EXISTS ] ... - CREATE OPERATOR [ IF NOT EXISTS ] ... - CREATE TYPE [ IF NOT EXISTS ] ... [AS

Re: [HACKERS] pg_rewind, a tool for resynchronizing an old master after failover

2013-05-24 Thread Jim Nasby
On 5/23/13 12:51 PM, Pavan Deolasee wrote: On Thu, May 23, 2013 at 11:10 PM, Heikki Linnakangas hlinnakan...@vmware.com mailto:hlinnakan...@vmware.com wrote: On 23.05.2013 07:55, Robert Haas wrote: On Thu, May 23, 2013 at 7:10 AM, Heikki Linnakangas

Re: [HACKERS] background processes vs. hot standby

2013-05-24 Thread Alvaro Herrera
Andres Freund escribió: I wonder if we shouldn't make background workers use connections slots from max_connections similar to how superuser_reserved_connections work. That would mean we don't need to care about it for HS. I remember considering this and concluding that it's messy. Suppose

Re: [HACKERS] WAL segments (names) not in a sequence

2013-05-24 Thread German Becker
On Fri, May 24, 2013 at 10:01 AM, Amit Langote amitlangot...@gmail.comwrote: Maybe I didn't explain correctly. I am using COPY/pg_dump/pg_restore for migration (and it is working fine). The streaming replication is for hot-standby replication *once migrated*. Thing is I disbable archving

Re: [HACKERS] getting rid of freezing

2013-05-24 Thread Robert Haas
On Fri, May 24, 2013 at 10:53 AM, Andres Freund and...@2ndquadrant.com wrote: [all-visible cannot restore hint bits without FPI because of torn pages] I haven't yet thought about this sufficiently yet. I think we might have a chance of working around this, let me ponder a bit. Yeah. I too

Re: [HACKERS] WAL segments (names) not in a sequence

2013-05-24 Thread Amit Langote
I didn't quite understand what you mean by that... But anyways so do you people think this sequence number overlap is normal ? There is no overlap at all. The newer segments that you see are pre-allocated ones. They have not been written to yet. From the ls -l pg_xlog output that you sent, it

Re: [HACKERS] getting rid of freezing

2013-05-24 Thread Andres Freund
On 2013-05-24 11:29:10 -0400, Robert Haas wrote: But even if that means needing a full page write via the usual mechanism for all visible if any hint bits needed to be set we are still out far ahead of the current state imo. * cleanup would quite possibly do an FPI shortly after in vacuum

Re: [HACKERS] getting rid of freezing

2013-05-24 Thread Robert Haas
On Fri, May 24, 2013 at 11:29 AM, Robert Haas robertmh...@gmail.com wrote: On Fri, May 24, 2013 at 10:53 AM, Andres Freund and...@2ndquadrant.com wrote: [all-visible cannot restore hint bits without FPI because of torn pages] I haven't yet thought about this sufficiently yet. I think we

Re: [HACKERS] getting rid of freezing

2013-05-24 Thread Robert Haas
On Fri, May 24, 2013 at 11:52 AM, Andres Freund and...@2ndquadrant.com wrote: The basic problem is that if the data is going to be removed before it would have gotten frozen, then the extra FPIs are just overhead. In effect, we're just deciding to freeze a lot sooner. Well, freezing without

Re: [HACKERS] background processes vs. hot standby

2013-05-24 Thread Robert Haas
On Fri, May 24, 2013 at 11:25 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Andres Freund escribió: I wonder if we shouldn't make background workers use connections slots from max_connections similar to how superuser_reserved_connections work. That would mean we don't need to care about

Re: [HACKERS] getting rid of freezing

2013-05-24 Thread Jim Nasby
On 5/24/13 9:53 AM, Andres Freund wrote: We don't even necessarily need to log the hint bits for all items since the redo for all_visible could make sure all items are hinted. The only problem is knowing up to where we can truncate pg_clog... [all-visible cannot restore hint bits without FPI

Re: [HACKERS] PGCON meetup FreeNAS/FreeBSD: In Ottawa Tue Wed.

2013-05-24 Thread Jim Nasby
I'm moving this to -advocacy, as it seems more appropriate there... On 5/20/13 10:31 AM, Alfred Perlstein wrote: Hello PostgreSQL Hackers, I am now in Ottawa, last week we wrapped up the BSDCon and I was hoping to chat with a few Postgresql developers in person about using Postgresql in

[HACKERS] visibilitymap_set and checksums

2013-05-24 Thread Andres Freund
Hi, while thinking about vacuum freeze I noticed that since the checksums patch visibilitymap_set() does: /* * If data checksums are enabled, we need to protect the heap * page from being torn. */ if (DataChecksumsEnabled()) {

Re: [HACKERS] WAL segments (names) not in a sequence

2013-05-24 Thread German Becker
Thanks Amit, I understand now. Is there a way to know/predict how many prealocated segments will there be in a certain moment? What does it deppend on? On Fri, May 24, 2013 at 12:46 PM, Amit Langote amitlangot...@gmail.comwrote: I didn't quite understand what you mean by that... But anyways

Re: [HACKERS] visibilitymap_set and checksums

2013-05-24 Thread Simon Riggs
On 24 May 2013 18:40, Andres Freund and...@2ndquadrant.com wrote: That pattern looks dangerous. Setting the lsn of the heap page will prevent the next action from doing a FPI even if it would be required. Can you be more specific about the danger you see? -- Simon Riggs

Re: [HACKERS] Parallel Sort

2013-05-24 Thread Jim Nasby
On 5/13/13 9:28 AM, Noah Misch wrote: It would be great if one client session could take advantage of multiple CPU cores. EnterpriseDB wishes to start the trek into this problem space for 9.4 by implementing parallel internal (i.e. not spilling to disk) sort. This touches on a notable subset

Re: [HACKERS] WAL segments (names) not in a sequence

2013-05-24 Thread Amit Langote
On Sat, May 25, 2013 at 3:08 AM, German Becker german.bec...@gmail.com wrote: Thanks Amit, I understand now. Is there a way to know/predict how many prealocated segments will there be in a certain moment? What does it deppend on? Upthread, Fujii Masao-san suggested what might have happened

Re: [HACKERS] Incomplete description of pg_start_backup?

2013-05-24 Thread Dmitry Koterov
I don't get still. Suppose we have a data file with blocks with important (non-empty) data: A B C D 1. I call pg_start_backup(). 2. Tar starts to copy A block to the destination archive... 3. During this copying, somebody removes data from a table which is situated in B block. So this data is a

Re: [HACKERS] Cost limited statements RFC

2013-05-24 Thread Greg Smith
On 5/24/13 10:36 AM, Jim Nasby wrote: Instead of KB/s, could we look at how much time one process is spending waiting on IO vs the rest of the cluster? Is it reasonable for us to measure IO wait time for every request, at least on the most popular OSes? It's not just an OS specific issue. The

Re: [HACKERS] WAL segments (names) not in a sequence

2013-05-24 Thread German Becker
On Fri, May 24, 2013 at 3:24 PM, Amit Langote amitlangot...@gmail.comwrote: On Sat, May 25, 2013 at 3:08 AM, German Becker german.bec...@gmail.com wrote: Thanks Amit, I understand now. Is there a way to know/predict how many prealocated segments will there be in a certain moment? What does

Re: [HACKERS] Cost limited statements RFC

2013-05-24 Thread Greg Smith
On 5/24/13 9:21 AM, Robert Haas wrote: But I wonder if we wouldn't be better off coming up with a little more user-friendly API. Instead of exposing a cost delay, a cost limit, and various charges, perhaps we should just provide limits measured in KB/s, like dirty_rate_limit = amount of data

Re: [HACKERS] Parallel Sort

2013-05-24 Thread james
Have you considered GPU-based sorting? I know there's been discussion in the past. If you use OpenCL, then you can use a CPU driver if there is no GPU, and that can allow you to leverage all the CPU cores without having to do the multi-thread stuff in the backend. While the compilation of

[HACKERS] adding import in pl/python function

2013-05-24 Thread Szymon Guz
Hi, I'm wondering if it would be OK to change the procedure code before execution. I'm thinking about adding magically an import at the beginning of a function. Currently numeric arguments passed to the procedure are converted into floats. This is not good, as it causes loss of information. The

Re: [HACKERS] Cost limited statements RFC

2013-05-24 Thread Robert Haas
On Fri, May 24, 2013 at 10:36 AM, Jim Nasby j...@nasby.net wrote: Doesn't that hit the old issue of not knowing if a read came from FS cache or disk? I realize that the current cost_delay mechanism suffers from that too, but since the API is lower level that restriction is much more apparent.

Re: [HACKERS] adding import in pl/python function

2013-05-24 Thread Claudio Freire
On Fri, May 24, 2013 at 4:10 PM, Szymon Guz mabew...@gmail.com wrote: I'm thinking about something else. We could convert it into Decimal (http://docs.python.org/2/library/decimal.html) class in Python. Unfortunately this class requires import like `from decimal import Decimal` from a

[HACKERS] shmem startup and shutdown hooks

2013-05-24 Thread Alvaro Herrera
Hi, While going through Andres' BDR code I noticed that it has a shmem startup hook, which installs a on_shmem_exit() callback to write stuff at shutdown time. This works fine but seems somewhat of a hazard: it is having postmaster do the actual write, which has to access shared memory while

Re: [HACKERS] adding import in pl/python function

2013-05-24 Thread Szymon Guz
On 24 May 2013 21:14, Claudio Freire klaussfre...@gmail.com wrote: On Fri, May 24, 2013 at 4:10 PM, Szymon Guz mabew...@gmail.com wrote: I'm thinking about something else. We could convert it into Decimal (http://docs.python.org/2/library/decimal.html) class in Python. Unfortunately this

Re: [HACKERS] visibilitymap_set and checksums

2013-05-24 Thread Andres Freund
On 2013-05-24 19:09:57 +0100, Simon Riggs wrote: On 24 May 2013 18:40, Andres Freund and...@2ndquadrant.com wrote: That pattern looks dangerous. Setting the lsn of the heap page will prevent the next action from doing a FPI even if it would be required. Can you be more specific about the

Re: [HACKERS] adding import in pl/python function

2013-05-24 Thread Claudio Freire
On Fri, May 24, 2013 at 4:22 PM, Szymon Guz mabew...@gmail.com wrote: Hm... maybe you're right. I think I don't understand fully how the procedures are executed, and I need to read more to get it. Well, it's easy. Instead of PLyFloat_FromNumeric[0], you can make a PLyDecimal_FromNumeric.

Re: [HACKERS] getting rid of freezing

2013-05-24 Thread Josh Berkus
Andres, If I understand your solution correctly, though, this doesn't really help the pathological case for freezing, which is the time-oriented append-only table. For data which isn't being used, allvisible won't be set either because it won't have been read, no? Is it still cheaper to set

Re: [HACKERS] adding import in pl/python function

2013-05-24 Thread Szymon Guz
On 24 May 2013 21:46, Claudio Freire klaussfre...@gmail.com wrote: On Fri, May 24, 2013 at 4:22 PM, Szymon Guz mabew...@gmail.com wrote: Hm... maybe you're right. I think I don't understand fully how the procedures are executed, and I need to read more to get it. Well, it's easy.

Re: [HACKERS] getting rid of freezing

2013-05-24 Thread Andres Freund
On 2013-05-24 15:49:31 -0400, Josh Berkus wrote: If I understand your solution correctly, though, this doesn't really help the pathological case for freezing, which is the time-oriented append-only table. For data which isn't being used, allvisible won't be set either because it won't have

Re: [HACKERS] Parallel Sort

2013-05-24 Thread Kohei KaiGai
Let me introduce one thing we discussed in the developer meeting at Ottawa. We got a consensus that pluggable exec-node may be useful to replace a part of exec-node tree with an alternative one being implemented by extensions; which will allow to run something like GpuSort instead of existing

Re: [HACKERS] visibilitymap_set and checksums

2013-05-24 Thread Simon Riggs
On 24 May 2013 20:26, Andres Freund and...@2ndquadrant.com wrote: On 2013-05-24 19:09:57 +0100, Simon Riggs wrote: On 24 May 2013 18:40, Andres Freund and...@2ndquadrant.com wrote: That pattern looks dangerous. Setting the lsn of the heap page will prevent the next action from doing a FPI

Re: [HACKERS] Incomplete description of pg_start_backup?

2013-05-24 Thread Heikki Linnakangas
On 24.05.2013 14:33, Dmitry Koterov wrote: I don't get still. Suppose we have a data file with blocks with important (non-empty) data: A B C D 1. I call pg_start_backup(). 2. Tar starts to copy A block to the destination archive... 3. During this copying, somebody removes data from a table