Re: [HACKERS] Patch to .gitignore

2013-05-24 Thread amul sul
 >"ctags" and "etags" be part of postgres source tree and its generate some output inside them, so I think we must ignore it.  +1 Regards, Amul Sul -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/p

Re: [HACKERS] WARNING : pgstat wait timeout - Postgres 9.1

2013-05-24 Thread Mathieu Guerin
Hello, Thanks a lot for your answers. > You should get it... > stats_temp_directory | pg_stat_tmp | Writes temporary statistics files to the specified directory. I don't know why i don't get it. I am in 9.1 version... Moreover, when I mount pg_stat_tm

Re: [HACKERS] WARNING : pgstat wait timeout - Postgres 9.1

2013-05-24 Thread Mathieu Guerin
About the stats_temp_directory, I didn't run as root... Now I'm sure the configurations are correct. I think, I have too much IO to use stats. I will ever have this message... Maybe I can disable this option. Do you know what it really impact ? Thanks. Math 2013/5/24 Mathieu Guerin > Hello,

Re: [HACKERS] Removal of pageinspect--1.0.sql

2013-05-24 Thread Heikki Linnakangas
On 20.05.2013 19:50, Michael Paquier wrote: The contrib module pageinspect has been upgraded to 1.1, but pageinspect--1.0.sql is still present in source code. Shouldn't it be removed? Please find patch attached. Yep. Removed, thanks. - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hac

Re: [HACKERS] WAL segments (names) not in a sequence

2013-05-24 Thread German Becker
Hi Sergey, Maybe I didn't explain correctly. I am using COPY/pg_dump/pg_restore for migration (and it is working fine). The streaming replication is for hot-standby replication *once migrated*. Thing is I disbable archving and set wal_level to minimal, when migrating the large portion of data, to

Re: [HACKERS] Patch to .gitignore

2013-05-24 Thread Tom Lane
=?ISO-8859-1?Q?Fabr=EDzio_de_Royes_Mello?= writes: > On Fri, May 24, 2013 at 12:04 AM, Christopher Browne > wrote: >> There hasn't been general agreement on the merits of particular .gitignore >> rules of this sort. > I agree with you about vim-oriented patterns, because its a particular > tool,

Re: [HACKERS] WAL segments (names) not in a sequence

2013-05-24 Thread Amit Langote
> Maybe I didn't explain correctly. I am using COPY/pg_dump/pg_restore for > migration (and it is working fine). The streaming replication is for > hot-standby replication *once migrated*. Thing is I disbable archving and > set wal_level to minimal, when migrating the large portion of data, to make

Re: [HACKERS] Cost limited statements RFC

2013-05-24 Thread Robert Haas
On Thu, May 23, 2013 at 7:27 PM, Greg Smith wrote: > I'm working on a new project here that I wanted to announce, just to keep > from duplicating effort in this area. I've started to add a cost limit > delay for regular statements. The idea is that you set a new > statement_cost_delay setting be

[HACKERS] background processes vs. hot standby

2013-05-24 Thread Robert Haas
CheckRequiredParameterValues() has some code that, when hot standby is in use, checks the values of max_connections, max_prepared_transactions, and max_locks_per_transaction against the master. The comment says "we must have at least as many backend slots as the primary" ... but the code no longe

Re: [HACKERS] background processes vs. hot standby

2013-05-24 Thread Andres Freund
Hi, On 2013-05-24 09:48:03 -0400, Robert Haas wrote: > CheckRequiredParameterValues() has some code that, when hot standby is > in use, checks the values of max_connections, > max_prepared_transactions, and max_locks_per_transaction against the > master. The comment says "we must have at least a

Re: [HACKERS] Cost limited statements RFC

2013-05-24 Thread Jim Nasby
On 5/24/13 8:21 AM, Robert Haas wrote: On Thu, May 23, 2013 at 7:27 PM, Greg Smith wrote: >I'm working on a new project here that I wanted to announce, just to keep >from duplicating effort in this area. I've started to add a cost limit >delay for regular statements. The idea is that you set

Re: [HACKERS] Move unused buffers to freelist

2013-05-24 Thread Jim Nasby
On 5/14/13 8:42 AM, Amit Kapila wrote: In the attached patch, bgwriter/checkpointer moves unused (usage_count =0 && refcount = 0) buffer’s to end of freelist. I have implemented a new API StrategyMoveBufferToFreeListEnd() to move buffer’s to end of freelist. Instead of a separate function,

Re: [HACKERS] getting rid of freezing

2013-05-24 Thread Andres Freund
On 2013-05-23 22:09:02 -0400, Robert Haas wrote: > On Thu, May 23, 2013 at 1:51 PM, Andres Freund wrote: > > So, what I propose instead is basically: > > 1) only vacuum non-all-visible pages, even when doing it for > >anti-wraparound > > Check. We might want an option to force a scan of the w

Re: [HACKERS] pg_rewind, a tool for resynchronizing an old master after failover

2013-05-24 Thread Thom Brown
On 23 May 2013 10:03, Thom Brown wrote: > On 23 May 2013 07:10, Heikki Linnakangas wrote: >> Hi, >> >> I've been hacking on a tool to allow resynchronizing an old master server >> after failover. The need to do a full backup/restore has been a common >> complaint ever since we've had streaming re

Re: [HACKERS] pg_rewind, a tool for resynchronizing an old master after failover

2013-05-24 Thread Andres Freund
On 2013-05-24 10:57:22 -0400, Thom Brown wrote: > By the way, without any data inserted I get: > > thom@swift /tmp $ pg_rewind --target-pgdata=/tmp/primary > --source-server='host=localhost port=5531 dbname=postgres' -v > connected to remote server > fetched file "global/pg_control", length 8192 >

[HACKERS] Patch to add support of "IF NOT EXISTS" to others "CREATE" statements

2013-05-24 Thread Fabrízio de Royes Mello
Hi all, I working in a patch to include support of "IF NOT EXISTS" into "CREATE" statements that not have it yet. I started with "DefineStmt" section from "src/backend/parser/gram.y": - CREATE AGGREGATE [ IF NOT EXISTS ] ... - CREATE OPERATOR [ IF NOT EXISTS ] ... - CREATE TYPE [ IF NOT EXISTS ]

Re: [HACKERS] pg_rewind, a tool for resynchronizing an old master after failover

2013-05-24 Thread Jim Nasby
On 5/23/13 12:51 PM, Pavan Deolasee wrote: On Thu, May 23, 2013 at 11:10 PM, Heikki Linnakangas mailto:hlinnakan...@vmware.com>> wrote: On 23.05.2013 07:55, Robert Haas wrote: On Thu, May 23, 2013 at 7:10 AM, Heikki Linnakangas mailto:hlinnakan...@vmware.com>> wrote:

Re: [HACKERS] background processes vs. hot standby

2013-05-24 Thread Alvaro Herrera
Andres Freund escribió: > I wonder if we shouldn't make background workers use connections slots > from max_connections similar to how superuser_reserved_connections > work. That would mean we don't need to care about it for HS. I remember considering this and concluding that it's messy. Suppose

Re: [HACKERS] WAL segments (names) not in a sequence

2013-05-24 Thread German Becker
On Fri, May 24, 2013 at 10:01 AM, Amit Langote wrote: > > Maybe I didn't explain correctly. I am using COPY/pg_dump/pg_restore for > > migration (and it is working fine). The streaming replication is for > > hot-standby replication *once migrated*. Thing is I disbable archving and > > set wal_leve

Re: [HACKERS] getting rid of freezing

2013-05-24 Thread Robert Haas
On Fri, May 24, 2013 at 10:53 AM, Andres Freund wrote: >> [all-visible cannot restore hint bits without FPI because of torn pages] > > I haven't yet thought about this sufficiently yet. I think we might have > a chance of working around this, let me ponder a bit. Yeah. I too feel like there migh

Re: [HACKERS] WAL segments (names) not in a sequence

2013-05-24 Thread Amit Langote
> I didn't quite understand what you mean by that... But anyways so do you > people think this sequence number overlap is "normal" ? There is "no overlap" at all. The newer segments that you see are "pre-allocated" ones. They have not been written to yet. >From the "ls -l pg_xlog" output that you

Re: [HACKERS] getting rid of freezing

2013-05-24 Thread Andres Freund
On 2013-05-24 11:29:10 -0400, Robert Haas wrote: > > But even if that means needing a full page write via the usual mechanism > > for all visible if any hint bits needed to be set we are still out far > > ahead of the current state imo. > > * cleanup would quite possibly do an FPI shortly after in

Re: [HACKERS] getting rid of freezing

2013-05-24 Thread Robert Haas
On Fri, May 24, 2013 at 11:29 AM, Robert Haas wrote: > On Fri, May 24, 2013 at 10:53 AM, Andres Freund > wrote: >>> [all-visible cannot restore hint bits without FPI because of torn pages] >> >> I haven't yet thought about this sufficiently yet. I think we might have >> a chance of working aroun

Re: [HACKERS] getting rid of freezing

2013-05-24 Thread Robert Haas
On Fri, May 24, 2013 at 11:52 AM, Andres Freund wrote: >> The basic problem is that if the data is going to be removed before it >> would have gotten frozen, then the extra FPIs are just overhead. In >> effect, we're just deciding to freeze a lot sooner. > > Well, freezing without removing inform

Re: [HACKERS] background processes vs. hot standby

2013-05-24 Thread Robert Haas
On Fri, May 24, 2013 at 11:25 AM, Alvaro Herrera wrote: > Andres Freund escribió: >> I wonder if we shouldn't make background workers use connections slots >> from max_connections similar to how superuser_reserved_connections >> work. That would mean we don't need to care about it for HS. > > I re

Re: [HACKERS] getting rid of freezing

2013-05-24 Thread Jim Nasby
On 5/24/13 9:53 AM, Andres Freund wrote: We don't even necessarily need to log the hint bits for all items since > >the redo for all_visible could make sure all items are hinted. The only > >problem is knowing up to where we can truncate pg_clog... >[all-visible cannot restore hint bits without

Re: [HACKERS] PGCON meetup FreeNAS/FreeBSD: In Ottawa Tue & Wed.

2013-05-24 Thread Jim Nasby
I'm moving this to -advocacy, as it seems more appropriate there... On 5/20/13 10:31 AM, Alfred Perlstein wrote: Hello PostgreSQL Hackers, I am now in Ottawa, last week we wrapped up the BSDCon and I was hoping to chat with a few Postgresql developers in person about using Postgresql in FreeNA

[HACKERS] visibilitymap_set and checksums

2013-05-24 Thread Andres Freund
Hi, while thinking about vacuum freeze I noticed that since the checksums patch visibilitymap_set() does: /* * If data checksums are enabled, we need to protect the heap * page from being torn. */ if (DataChecksumsEnabled()) { Page

Re: [HACKERS] WAL segments (names) not in a sequence

2013-05-24 Thread German Becker
Thanks Amit, I understand now. Is there a way to know/predict how many prealocated segments will there be in a certain moment? What does it deppend on? On Fri, May 24, 2013 at 12:46 PM, Amit Langote wrote: > > I didn't quite understand what you mean by that... But anyways so do you > > people th

Re: [HACKERS] visibilitymap_set and checksums

2013-05-24 Thread Simon Riggs
On 24 May 2013 18:40, Andres Freund wrote: > That pattern looks dangerous. Setting the lsn of the heap page will > prevent the next action from doing a FPI even if it would be required. Can you be more specific about the danger you see? -- Simon Riggs http://www.2ndQuadrant.c

Re: [HACKERS] Parallel Sort

2013-05-24 Thread Jim Nasby
On 5/13/13 9:28 AM, Noah Misch wrote: It would be great if one client session could take advantage of multiple CPU cores. EnterpriseDB wishes to start the trek into this problem space for 9.4 by implementing parallel internal (i.e. not spilling to disk) sort. This touches on a notable subset of

Re: [HACKERS] WAL segments (names) not in a sequence

2013-05-24 Thread Amit Langote
On Sat, May 25, 2013 at 3:08 AM, German Becker wrote: > Thanks Amit, I understand now. Is there a way to know/predict how many > prealocated segments will there be in a certain moment? What does it deppend > on? Upthread, Fujii Masao-san suggested what might have happened that caused these pre-al

Re: [HACKERS] Incomplete description of pg_start_backup?

2013-05-24 Thread Dmitry Koterov
I don't get still. Suppose we have a data file with blocks with important (non-empty) data: A B C D 1. I call pg_start_backup(). 2. Tar starts to copy A block to the destination archive... 3. During this copying, somebody removes data from a table which is situated in B block. So this data is a

Re: [HACKERS] Cost limited statements RFC

2013-05-24 Thread Greg Smith
On 5/24/13 10:36 AM, Jim Nasby wrote: Instead of KB/s, could we look at how much time one process is spending waiting on IO vs the rest of the cluster? Is it reasonable for us to measure IO wait time for every request, at least on the most popular OSes? It's not just an OS specific issue. The

Re: [HACKERS] WAL segments (names) not in a sequence

2013-05-24 Thread German Becker
On Fri, May 24, 2013 at 3:24 PM, Amit Langote wrote: > On Sat, May 25, 2013 at 3:08 AM, German Becker > wrote: > > Thanks Amit, I understand now. Is there a way to know/predict how many > > prealocated segments will there be in a certain moment? What does it > deppend > > on? > > Upthread, Fujii

Re: [HACKERS] Cost limited statements RFC

2013-05-24 Thread Greg Smith
On 5/24/13 9:21 AM, Robert Haas wrote: But I wonder if we wouldn't be better off coming up with a little more user-friendly API. Instead of exposing a cost delay, a cost limit, and various charges, perhaps we should just provide limits measured in KB/s, like dirty_rate_limit = and read_rate_li

Re: [HACKERS] Parallel Sort

2013-05-24 Thread james
> Have you considered GPU-based sorting? I know there's been discussion in the past. If you use OpenCL, then you can use a CPU driver if there is no GPU, and that can allow you to leverage all the CPU cores without having to do the multi-thread stuff in the backend. While the compilation of

[HACKERS] adding import in pl/python function

2013-05-24 Thread Szymon Guz
Hi, I'm wondering if it would be OK to change the procedure code before execution. I'm thinking about adding magically an import at the beginning of a function. Currently numeric arguments passed to the procedure are converted into floats. This is not good, as it causes loss of information. The p

Re: [HACKERS] Cost limited statements RFC

2013-05-24 Thread Robert Haas
On Fri, May 24, 2013 at 10:36 AM, Jim Nasby wrote: > Doesn't that hit the old issue of not knowing if a read came from FS cache > or disk? I realize that the current cost_delay mechanism suffers from that > too, but since the API is lower level that restriction is much more > apparent. Sure, but

Re: [HACKERS] adding import in pl/python function

2013-05-24 Thread Claudio Freire
On Fri, May 24, 2013 at 4:10 PM, Szymon Guz wrote: > > I'm thinking about something else. We could convert it into Decimal > (http://docs.python.org/2/library/decimal.html) class in Python. > Unfortunately this class requires import like `from decimal import Decimal` > from a standard Python libra

[HACKERS] shmem startup and shutdown hooks

2013-05-24 Thread Alvaro Herrera
Hi, While going through Andres' BDR code I noticed that it has a shmem startup hook, which installs a on_shmem_exit() callback to write stuff at shutdown time. This works fine but seems somewhat of a hazard: it is having postmaster do the actual write, which has to access shared memory while doin

Re: [HACKERS] adding import in pl/python function

2013-05-24 Thread Szymon Guz
On 24 May 2013 21:14, Claudio Freire wrote: > On Fri, May 24, 2013 at 4:10 PM, Szymon Guz wrote: > > > > I'm thinking about something else. We could convert it into Decimal > > (http://docs.python.org/2/library/decimal.html) class in Python. > > Unfortunately this class requires import like `fro

Re: [HACKERS] visibilitymap_set and checksums

2013-05-24 Thread Andres Freund
On 2013-05-24 19:09:57 +0100, Simon Riggs wrote: > On 24 May 2013 18:40, Andres Freund wrote: > > > That pattern looks dangerous. Setting the lsn of the heap page will > > prevent the next action from doing a FPI even if it would be required. > > Can you be more specific about the danger you see

Re: [HACKERS] adding import in pl/python function

2013-05-24 Thread Claudio Freire
On Fri, May 24, 2013 at 4:22 PM, Szymon Guz wrote: > Hm... maybe you're right. I think I don't understand fully how the > procedures are executed, and I need to read more to get it. Well, it's easy. Instead of PLyFloat_FromNumeric[0], you can make a PLyDecimal_FromNumeric. There, you'd do with

Re: [HACKERS] getting rid of freezing

2013-05-24 Thread Josh Berkus
Andres, If I understand your solution correctly, though, this doesn't really help the pathological case for freezing, which is the time-oriented append-only table. For data which isn't being used, allvisible won't be set either because it won't have been read, no? Is it still cheaper to set allv

Re: [HACKERS] adding import in pl/python function

2013-05-24 Thread Szymon Guz
On 24 May 2013 21:46, Claudio Freire wrote: > On Fri, May 24, 2013 at 4:22 PM, Szymon Guz wrote: > > Hm... maybe you're right. I think I don't understand fully how the > > procedures are executed, and I need to read more to get it. > > > Well, it's easy. > > Instead of PLyFloat_FromNumeric[0], y

Re: [HACKERS] getting rid of freezing

2013-05-24 Thread Andres Freund
On 2013-05-24 15:49:31 -0400, Josh Berkus wrote: > If I understand your solution correctly, though, this doesn't really > help the pathological case for freezing, which is the time-oriented > append-only table. For data which isn't being used, allvisible won't be > set either because it won't have

Re: [HACKERS] Parallel Sort

2013-05-24 Thread Kohei KaiGai
Let me introduce one thing we discussed in the developer meeting at Ottawa. We got a consensus that pluggable exec-node may be useful to replace a part of exec-node tree with an alternative one being implemented by extensions; which will allow to run something like "GpuSort" instead of existing Sor

Re: [HACKERS] visibilitymap_set and checksums

2013-05-24 Thread Simon Riggs
On 24 May 2013 20:26, Andres Freund wrote: > On 2013-05-24 19:09:57 +0100, Simon Riggs wrote: >> On 24 May 2013 18:40, Andres Freund wrote: >> >> > That pattern looks dangerous. Setting the lsn of the heap page will >> > prevent the next action from doing a FPI even if it would be required. >> >>

Re: [HACKERS] Incomplete description of pg_start_backup?

2013-05-24 Thread Heikki Linnakangas
On 24.05.2013 14:33, Dmitry Koterov wrote: I don't get still. Suppose we have a data file with blocks with important (non-empty) data: A B C D 1. I call pg_start_backup(). 2. Tar starts to copy A block to the destination archive... 3. During this copying, somebody removes data from a table whi

Re: [HACKERS] getting rid of freezing

2013-05-24 Thread Hannu Krosing
On 05/24/2013 07:00 PM, Robert Haas wrote: > On Fri, May 24, 2013 at 11:29 AM, Robert Haas wrote: >> On Fri, May 24, 2013 at 10:53 AM, Andres Freund >> wrote: [all-visible cannot restore hint bits without FPI because of torn pages] >>> I haven't yet thought about this sufficiently yet. I th

Re: [HACKERS] Move unused buffers to freelist

2013-05-24 Thread Amit kapila
On Friday, May 24, 2013 8:22 PM Jim Nasby wrote: On 5/14/13 8:42 AM, Amit Kapila wrote: >> In the attached patch, bgwriter/checkpointer moves unused (usage_count =0 && >> refcount = 0) buffer’s to end of freelist. I have implemented a new API >> StrategyMoveBufferToFreeListEnd() to >> >> move buf