[HACKERS] Redundancy in comment within lock.c

2013-04-24 Thread Peter Geoghegan
Silly typo report. Line 3774 of lmgr's lock.c says: * Re-acquire a lock belonging to a transaction that was prepared, when * when starting up into hot standby mode. This has been the case since the original hot standby commit. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsq

Re: [HACKERS] Failing start-up archive recovery at Standby mode in PG9.2.4

2013-04-24 Thread Amit Langote
I also had a similar observation when I could reproduce this. I tried to find why restartpoint causes the recycled segment to be named after timeline 3, but have not been able to determine that. When I looked at the source, I found that, the function XLogFileReadAnyTLI which returns a segment file

[HACKERS] Please add discussion topics for cluster-hackers meeting

2013-04-24 Thread Josh Berkus
Folks, The 2013 cluster-hackers meeting is less than a month away. If you are attending, and have topics of common interest to discuss, such as core APIs (i.e. event triggers are in, at least partway), please add your topic to the wiki here: https://wiki.postgresql.org/wiki/PgCon2013CanadaCluste

Re: [HACKERS] danger of stats_temp_directory = /dev/shm

2013-04-24 Thread Tom Lane
Alvaro Herrera writes: > Jeff Janes escribió: >> With the stats file split patch 187492b6c2e8cafc5 introduced in 9.3dev, now >> after a crash the postmaster will try to delete all files in the directory >> stats_temp_directory. When that is just a subdirectory of PGDATA, this is >> fine. But it s

Re: [HACKERS] Enabling Checksums

2013-04-24 Thread Jeff Davis
On Wed, 2013-04-24 at 21:09 +0100, Simon Riggs wrote: > On 24 April 2013 21:06, Jeff Davis wrote: > > > What goal are you trying to accomplish with this patch? > > That we might need to patch the checksum version on a production release. Oh, I see. I don't think we need two output fields from

Re: [HACKERS] Failing start-up archive recovery at Standby mode in PG9.2.4

2013-04-24 Thread KONDO Mitsumasa
Hi, I find problem about failing start-up achive recovery at Standby mode in PG9.2.4 streaming replication. I test same problem in PG9.2.3. But it is not occerd... cp: cannot stat `../arc/00030013': そのようなファイルやディレクトリはありません [Standby] 2013-04-22 01:27:25 EDTLOG: 0: restored

Re: [HACKERS] danger of stats_temp_directory = /dev/shm

2013-04-24 Thread Alvaro Herrera
Jeff Janes escribió: > With the stats file split patch 187492b6c2e8cafc5 introduced in 9.3dev, now > after a crash the postmaster will try to delete all files in the directory > stats_temp_directory. When that is just a subdirectory of PGDATA, this is > fine. But it seems rather hostile when it is

Re: [HACKERS] Failing start-up archive recovery at Standby mode in PG9.2.4

2013-04-24 Thread Kyotaro HORIGUCHI
I had a bit look on it and came up with an hypothesis.. umm or a scenario. == Just after restartpoint, too old xlog files are recycled but its page header has old content, specifically, xlogid and xrecoff. Plus, if the master's LSN is at the head of new segment file, the file for the segment ma

[HACKERS] danger of stats_temp_directory = /dev/shm

2013-04-24 Thread Jeff Janes
With the stats file split patch 187492b6c2e8cafc5 introduced in 9.3dev, now after a crash the postmaster will try to delete all files in the directory stats_temp_directory. When that is just a subdirectory of PGDATA, this is fine. But it seems rather hostile when it is set to a shared directory, l

Re: [HACKERS] Proposal to add --single-row to psql

2013-04-24 Thread Christopher Manning
Fabrízio and Tom, I know that you can use --variable="FETCH_COUNT=1" from the psql command line, but internally that uses a CURSOR to batch the rows and [Redshift doesn't support CURSOR]( https://forums.aws.amazon.com/thread.jspa?threadID=122664&tstart=0) so it's not an option when using psql

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Claudio Freire
On Wed, Apr 24, 2013 at 6:47 PM, Joachim Wieland wrote: > On Wed, Apr 24, 2013 at 4:05 PM, Stefan Kaltenbrunner > wrote: >> >> > What might make sense is something like pg_dump_restore which would have >> > no intermediate storage at all, just pump the data etc from one source >> > to another in

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Joachim Wieland
On Wed, Apr 24, 2013 at 4:05 PM, Stefan Kaltenbrunner < ste...@kaltenbrunner.cc> wrote: > > What might make sense is something like pg_dump_restore which would have > > no intermediate storage at all, just pump the data etc from one source > > to another in parallel. But I pity the poor guy who ha

Re: [HACKERS] Enabling Checksums

2013-04-24 Thread Simon Riggs
On 24 April 2013 21:06, Jeff Davis wrote: > What goal are you trying to accomplish with this patch? That we might need to patch the checksum version on a production release. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Enabling Checksums

2013-04-24 Thread Jeff Davis
On Wed, 2013-04-24 at 08:20 +0100, Simon Riggs wrote: > On 24 April 2013 01:10, Jeff Davis wrote: > > I'd prefer that it was some kind of a checksum ID code -- e.g. 0 for no > > checksum, 1 for FNV-1a-SR3, etc. That would allow us to release 9.4 with > > a new algorithm without forcing existing us

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Stefan Kaltenbrunner
On 04/24/2013 09:51 PM, Andrew Dunstan wrote: > > On 04/24/2013 03:49 PM, Andrew Dunstan wrote: >> >> On 04/24/2013 03:40 PM, Dimitri Fontaine wrote: >>> Andrew Dunstan writes: On 04/23/2013 07:53 PM, Timothy Garnett wrote: > Anyways, the question is if people think this is generally use

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Andrew Dunstan
On 04/24/2013 03:49 PM, Andrew Dunstan wrote: On 04/24/2013 03:40 PM, Dimitri Fontaine wrote: Andrew Dunstan writes: On 04/23/2013 07:53 PM, Timothy Garnett wrote: Anyways, the question is if people think this is generally useful. If so I can clean up the preferred choice a bit and rebase

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Andrew Dunstan
On 04/24/2013 03:40 PM, Dimitri Fontaine wrote: Andrew Dunstan writes: On 04/23/2013 07:53 PM, Timothy Garnett wrote: Anyways, the question is if people think this is generally useful. If so I can clean up the preferred choice a bit and rebase it off of master, etc. I find this idea very us

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Dimitri Fontaine
Andrew Dunstan writes: > On 04/23/2013 07:53 PM, Timothy Garnett wrote: >> Anyways, the question is if people think this is generally useful. If so >> I can clean up the preferred choice a bit and rebase it off of master, >> etc. I find this idea very useful yes. Another idea would be to allow

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Andrew Dunstan
On 04/23/2013 07:53 PM, Timothy Garnett wrote: Hi All, Currently the -j option to pg_restore, which allows for parallelization in the restore, can only be used if the input file is a regular file and not, for ex., a pipe. However this is a pretty common occurrence for us (usually in the for

Re: [HACKERS] GSOC Student Project Idea

2013-04-24 Thread Michael Schuh
On Wed, Apr 24, 2013 at 5:31 AM, Florian Pflug wrote: > On Apr23, 2013, at 23:25 , Alexander Korotkov > wrote: > > I've taken a brief look on the paper and implementation. As I can see > iDistance implements some global building strategy. I mean, for example, it > selects some point, calculates

[HACKERS] Re: [HACKERS] Re[3]: [HACKERS] high io BUT huge amount of free memory

2013-04-24 Thread Миша Тюрин
vm state root@avi-sql09:~# /sbin/sysctl -a|grep vm vm.overcommit_memory = 0 vm.panic_on_oom = 0 vm.oom_kill_allocating_task = 0 vm.oom_dump_tasks = 0 vm.overcommit_ratio = 50 vm.page-cluster = 3 vm.dirty_background_ratio = 10 vm.dirty_background_bytes = 0 vm.dirty_ratio = 20 vm.dirty_bytes = 0 vm.

[HACKERS] Re[3]: [HACKERS] high io BUT huge amount of free memory

2013-04-24 Thread Миша Тюрин
typo > if ( user cpu + io wait ) is ~140% then i have ~140GB free. 140% ===>> 1400% if ~14 cores are busy then ~140GB is free 10GB per process hmmm... -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mail

[HACKERS] Re[2]: [HACKERS] high io BUT huge amount of free memory

2013-04-24 Thread Миша Тюрин
thanks a lot for responses 1) just remind my case Intel 32 core = 2*8 *2threads Linux 2.6.32-5-amd64 #1 SMP Sun May 6 04:00:17 UTC 2012 x86_64 GNU/Linux PostgreSQL 9.2.2 on x86_64-unknown-linux-gnu, compiled by gcc-4.4.real (Debian 4.4.5-8) 4.4.5, 64-bit shared_buffers 64GB / constant hit ra

Examples Re: [HACKERS] Bug Fix: COLLATE with multiple ORDER BYs in aggregates

2013-04-24 Thread David Fetter
On Tue, Apr 23, 2013 at 09:57:27AM -0700, David Fetter wrote: > Folks, > > While testing the upcoming FILTER clause for aggregates, Erik Rijkers > uncovered a long-standing bug in $subject, namely that this case > wasn't handled. Please find attached a patch by Andrew Gierth and > myself which fi

Re: [HACKERS] putting a bgworker to rest

2013-04-24 Thread Robert Haas
On Tue, Apr 23, 2013 at 1:22 PM, Andres Freund wrote: >> So a "done" worker would never be restarted, until postmaster sees a >> crash or is itself restarted? I guess that'd be useful for workers >> running during recovery, which terminate when recovery completes. Is >> that your use case? > > W

Re: [HACKERS] 9.3 Beta1 status report

2013-04-24 Thread Heikki Linnakangas
On 24.04.2013 06:22, Bruce Momjian wrote: On Tue, Apr 23, 2013 at 06:56:34PM -0300, Alvaro Herrera wrote: Bruce Momjian wrote: On Tue, Apr 23, 2013 at 05:04:15PM -0400, Bruce Momjian wrote: Do we usually repeat the changes listed in the backwards compatibility section later, in the "Changes" s

Re: [HACKERS] putting a bgworker to rest

2013-04-24 Thread Dimitri Fontaine
Andres Freund writes: >> How would postmaster know when to restart a worker that stopped? > > I had imagined we would assign some return codes special > meaning. Currently 0 basically means "restart immediately", 1 means > "crashed, wait for some time", everything else results in a postmaster > re

Re: [HACKERS] missing time.h include in psql/command.c since the addition of \watch

2013-04-24 Thread Heikki Linnakangas
On 24.04.2013 16:27, Andres Freund wrote: Hi, our internal testbuilds show a new warning on windows: src\bin\psql\command.c(2617): warning C4013: 'time' undefined; assuming extern returning int [C:\jenkins\workspace\andres_git.postgresql.org_windows\BT\release\SL_OS\windows\TA\x86\TO\xp\psql.v

[HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Timothy Garnett
Hi All, Currently the -j option to pg_restore, which allows for parallelization in the restore, can only be used if the input file is a regular file and not, for ex., a pipe. However this is a pretty common occurrence for us (usually in the form of pg_dump | pg_restore to copy an individual datab

Re: [HACKERS] high io BUT huge amount of free memory

2013-04-24 Thread Andres Freund
On 2013-04-24 09:06:39 -0500, Shaun Thomas wrote: > On 04/24/2013 08:49 AM, Andres Freund wrote: > > >Uh. Ranting can be rather healthy thing every now and then and it good > >for the soul and such. But. Did you actually try reporting those issues? > > That's actually part of the problem. How do

Re: [HACKERS] high io BUT huge amount of free memory

2013-04-24 Thread Shaun Thomas
On 04/24/2013 08:49 AM, Andres Freund wrote: Uh. Ranting can be rather healthy thing every now and then and it good for the soul and such. But. Did you actually try reporting those issues? That's actually part of the problem. How do you report: Throwing a lot of processes at a high-memory sys

Re: [HACKERS] high io BUT huge amount of free memory

2013-04-24 Thread Andres Freund
On 2013-04-24 08:39:09 -0500, Shaun Thomas wrote: > The memory pressure code in Linux is extremely fucked up. I can't find it > right now, but the memory management algorithm makes some pretty ridiculous > assumptions once you pass half memory usage, regarding what is in active and > inactive cache

Re: [HACKERS] REFRESH MATERIALIZED VIEW command in PL block hitting Assert

2013-04-24 Thread Kevin Grittner
Jeevan Chalke wrote: > On Wed, Apr 24, 2013 at 3:04 AM, Kevin Grittner wrote: >> Any objections to the attached to fix this issue? > Nope. Fine with me. Pushed.  Thanks for the report! -- Kevin Grittner EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via

Re: [HACKERS] high io BUT huge amount of free memory

2013-04-24 Thread Shaun Thomas
On 04/24/2013 08:24 AM, Robert Haas wrote: Are you referring to the fact that vm.zone_reclaim_mode = 1 is an idiotic default? Well... it is. But even on systems where it's not the default or is explicitly disabled, there's just something hideously wrong with NUMA in general. Take a look at o

Re: [HACKERS] 9.3 release notes suggestions

2013-04-24 Thread Robert Haas
On Tue, Apr 23, 2013 at 11:41 PM, Bruce Momjian wrote: > Thanks for the many suggestions on improving the 9.3 release notes. > There were many ideas I would have never thought of. Please keep the > suggestions coming. Bruce, Thanks for writing them! -- Robert Haas EnterpriseDB: http://www.ente

[HACKERS] missing time.h include in psql/command.c since the addition of \watch

2013-04-24 Thread Andres Freund
Hi, our internal testbuilds show a new warning on windows: src\bin\psql\command.c(2617): warning C4013: 'time' undefined; assuming extern returning int [C:\jenkins\workspace\andres_git.postgresql.org_windows\BT\release\SL_OS\windows\TA\x86\TO\xp\psql.vcxproj] src\bin\psql\command.c(2619): warnin

Re: [HACKERS] high io BUT huge amount of free memory

2013-04-24 Thread Robert Haas
On Tue, Apr 23, 2013 at 10:50 AM, Shaun Thomas wrote: > This is most likely a NUMA issue. There really seems to be some kind of > horrible flaw in the Linux kernel when it comes to properly handling NUMA on > large memory systems. Are you referring to the fact that vm.zone_reclaim_mode = 1 is an

Re: [HACKERS] REFRESH MATERIALIZED VIEW command in PL block hitting Assert

2013-04-24 Thread Jeevan Chalke
On Wed, Apr 24, 2013 at 3:04 AM, Kevin Grittner wrote: > Jeevan Chalke wrote: > > On Mon, Apr 22, 2013 at 6:41 PM, Andres Freund > wrote: > >> On 2013-04-22 18:35:04 +0530, Jeevan Chalke wrote: > > >>> I have observed that following sequence is causing server crash. > >>> > >>> CREATE MATERIALI

Re: [HACKERS] Proposal to add --single-row to psql

2013-04-24 Thread Fabrízio de Royes Mello
On Tue, Apr 23, 2013 at 1:05 PM, Tom Lane wrote: > > Isn't there already a way to set FETCH_COUNT from the command line? > (ie, I think there's a generic variable-assignment facility that could > do this) > Christopher, Tom is all right... from psql [1] command line we can do that: $ bin/psql -

Re: [HACKERS] GSOC Student Project Idea

2013-04-24 Thread Florian Pflug
On Apr23, 2013, at 23:25 , Alexander Korotkov wrote: > I've taken a brief look on the paper and implementation. As I can see > iDistance implements some global building strategy. I mean, for example, it > selects some point, calculates distances from selected point to all points in > dataset et

Re: [HACKERS] Failing start-up archive recovery at Standby mode in PG9.2.4

2013-04-24 Thread Kyotaro HORIGUCHI
Oops, > But thats not what happening here, afaics the "restore log file ..." > message is only printed if the returncode is 0. You're right. 'cp ' exits with the status code 1 (or 256?). The quoted log lines simply show that segment for tli=3 did not exist and that for tli=2 had been gotten fin

Re: [HACKERS] Failing start-up archive recovery at Standby mode in PG9.2.4

2013-04-24 Thread Andres Freund
On 2013-04-24 19:16:12 +0900, Kyotaro HORIGUCHI wrote: > Hello, > > > > cp: cannot stat `../arc/00030013': そのようなファイルやディレクトリはありません > > > [Standby] 2013-04-22 01:27:25 EDTLOG: 0: restored log file > > > "00020013" from archive > > > I can't read the error messa

Re: [HACKERS] Failing start-up archive recovery at Standby mode in PG9.2.4

2013-04-24 Thread Kyotaro HORIGUCHI
Sorry, > caller XLogFileOpen successfully ets and returns fd for the filename The caller is XLogFileRead in this case. # and 'ets' is gets, of course. regards, -- Kyotaro Horiguchi -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: ht

Re: [HACKERS] Failing start-up archive recovery at Standby mode in PG9.2.4

2013-04-24 Thread Kyotaro HORIGUCHI
Hello, > > cp: cannot stat `../arc/00030013': そのようなファイルやディレクトリはありません > > [Standby] 2013-04-22 01:27:25 EDTLOG: 0: restored log file > > "00020013" from archive > I can't read the error message here, but this looks suspicious. The error message is "No such fi

Re: [HACKERS] Fast promotion, loose ends

2013-04-24 Thread Simon Riggs
On 24 April 2013 09:53, Heikki Linnakangas wrote: > On 24.04.2013 11:46, Simon Riggs wrote: >> >> On 24 April 2013 09:32, Heikki Linnakangas >> wrote: >> pg_ctl already checks versions, so I don't see the point. >>> >>> >>> >>> The point is, if you do "pgsql93/bin/pg_ctl -D $92DATADIR promote

Re: [HACKERS] Failing start-up archive recovery at Standby mode in PG9.2.4

2013-04-24 Thread Andres Freund
ello, On 2013-04-24 17:43:39 +0900, KONDO Mitsumasa wrote: > Hi, > > I find problem about failing start-up achive recovery at Standby mode in > PG9.2.4 streaming replication. > I test same problem in PG9.2.3. But it is not occerd... > > cp: cannot stat `../arc/00030013': そのようなファ

Re: [HACKERS] Fast promotion, loose ends

2013-04-24 Thread Heikki Linnakangas
On 24.04.2013 11:46, Simon Riggs wrote: On 24 April 2013 09:32, Heikki Linnakangas wrote: pg_ctl already checks versions, so I don't see the point. The point is, if you do "pgsql93/bin/pg_ctl -D $92DATADIR promote", it will create fast_promote file and return success. But it won't actually

Re: [HACKERS] Fast promotion, loose ends

2013-04-24 Thread Simon Riggs
On 24 April 2013 09:32, Heikki Linnakangas wrote: >> pg_ctl already checks versions, so I don't see the point. > > > The point is, if you do "pgsql93/bin/pg_ctl -D $92DATADIR promote", it will > create fast_promote file and return success. But it won't actually promote > the server. I think that'

Re: [HACKERS] Fast promotion, loose ends

2013-04-24 Thread Heikki Linnakangas
On 24.04.2013 11:23, Simon Riggs wrote: On 24 April 2013 09:10, Heikki Linnakangas wrote: Regarding the change in pg_ctl: /* -* Use two different kinds of promotion file so we can understand -* the difference between smart and fast promotion. +* For 9.3 onward

Re: [HACKERS] Fast promotion, loose ends

2013-04-24 Thread Simon Riggs
On 24 April 2013 09:10, Heikki Linnakangas wrote: > Regarding the change in pg_ctl: > >> /* >> -* Use two different kinds of promotion file so we can understand >> -* the difference between smart and fast promotion. >> +* For 9.3 onwards, use fast promotion as the d

Re: [HACKERS] Fast promotion, loose ends

2013-04-24 Thread Heikki Linnakangas
On 24.04.2013 10:57, Simon Riggs wrote: On 24 April 2013 08:23, Heikki Linnakangas wrote: On 22.04.2013 18:44, Simon Riggs wrote: On 22 April 2013 09:29, Heikki Linnakangas wrote: Hmm. That requires write access to $DATADIR, so that's not quite the same thing as the trigger_file recovery.co

Re: [HACKERS] Fast promotion, loose ends

2013-04-24 Thread Simon Riggs
On 24 April 2013 08:23, Heikki Linnakangas wrote: > On 22.04.2013 18:44, Simon Riggs wrote: >> >> On 22 April 2013 09:29, Heikki Linnakangas >> wrote: >> >>> Hmm. That requires write access to $DATADIR, so that's not quite the same >>> thing as the trigger_file recovery.conf option. >> >> >> Well,

Re: [HACKERS] Fast promotion, loose ends

2013-04-24 Thread Heikki Linnakangas
On 22.04.2013 18:44, Simon Riggs wrote: On 22 April 2013 09:29, Heikki Linnakangas wrote: Hmm. That requires write access to $DATADIR, so that's not quite the same thing as the trigger_file recovery.conf option. Well, you also (elsewhere) requested that we must keep recovery.conf in $DATADIR

Re: [HACKERS] Enabling Checksums

2013-04-24 Thread Simon Riggs
On 24 April 2013 01:10, Jeff Davis wrote: > On Tue, 2013-04-23 at 16:28 +0100, Simon Riggs wrote: >> * make the pg_control.data_checksums field into a version number, for >> future flexibility... >> patch attached > > Commenting on this separately because it's a separate issue. > > I'd prefer that