Re: [HACKERS] REFRESH MATERIALIZED VIEW command in PL block hitting Assert

2013-04-24 Thread Michael Paquier
On Tue, Apr 23, 2013 at 11:05 PM, Andres Freund and...@2ndquadrant.comwrote: On 2013-04-23 19:33:24 +0530, Jeevan Chalke wrote: On Tue, Apr 23, 2013 at 7:01 PM, Merlin Moncure mmonc...@gmail.com wrote: On Tue, Apr 23, 2013 at 7:18 AM, Jeevan Chalke jeevan.cha...@enterprisedb.com

Re: [HACKERS] Enabling Checksums

2013-04-24 Thread Simon Riggs
On 24 April 2013 01:10, Jeff Davis pg...@j-davis.com wrote: On Tue, 2013-04-23 at 16:28 +0100, Simon Riggs wrote: * make the pg_control.data_checksums field into a version number, for future flexibility... patch attached Commenting on this separately because it's a separate issue. I'd

Re: [HACKERS] Fast promotion, loose ends

2013-04-24 Thread Heikki Linnakangas
On 22.04.2013 18:44, Simon Riggs wrote: On 22 April 2013 09:29, Heikki Linnakangashlinnakan...@vmware.com wrote: Hmm. That requires write access to $DATADIR, so that's not quite the same thing as the trigger_file recovery.conf option. Well, you also (elsewhere) requested that we must keep

Re: [HACKERS] Fast promotion, loose ends

2013-04-24 Thread Simon Riggs
On 24 April 2013 08:23, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 22.04.2013 18:44, Simon Riggs wrote: On 22 April 2013 09:29, Heikki Linnakangashlinnakan...@vmware.com wrote: Hmm. That requires write access to $DATADIR, so that's not quite the same thing as the trigger_file

Re: [HACKERS] Fast promotion, loose ends

2013-04-24 Thread Heikki Linnakangas
On 24.04.2013 10:57, Simon Riggs wrote: On 24 April 2013 08:23, Heikki Linnakangashlinnakan...@vmware.com wrote: On 22.04.2013 18:44, Simon Riggs wrote: On 22 April 2013 09:29, Heikki Linnakangashlinnakan...@vmware.com wrote: Hmm. That requires write access to $DATADIR, so that's not quite

Re: [HACKERS] Fast promotion, loose ends

2013-04-24 Thread Simon Riggs
On 24 April 2013 09:10, Heikki Linnakangas hlinnakan...@vmware.com wrote: Regarding the change in pg_ctl: /* -* Use two different kinds of promotion file so we can understand -* the difference between smart and fast promotion. +* For 9.3 onwards, use fast

Re: [HACKERS] Fast promotion, loose ends

2013-04-24 Thread Heikki Linnakangas
On 24.04.2013 11:23, Simon Riggs wrote: On 24 April 2013 09:10, Heikki Linnakangashlinnakan...@vmware.com wrote: Regarding the change in pg_ctl: /* -* Use two different kinds of promotion file so we can understand -* the difference between smart and fast promotion. +

Re: [HACKERS] Fast promotion, loose ends

2013-04-24 Thread Simon Riggs
On 24 April 2013 09:32, Heikki Linnakangas hlinnakan...@vmware.com wrote: pg_ctl already checks versions, so I don't see the point. The point is, if you do pgsql93/bin/pg_ctl -D $92DATADIR promote, it will create fast_promote file and return success. But it won't actually promote the

Re: [HACKERS] Fast promotion, loose ends

2013-04-24 Thread Heikki Linnakangas
On 24.04.2013 11:46, Simon Riggs wrote: On 24 April 2013 09:32, Heikki Linnakangashlinnakan...@vmware.com wrote: pg_ctl already checks versions, so I don't see the point. The point is, if you do pgsql93/bin/pg_ctl -D $92DATADIR promote, it will create fast_promote file and return success.

Re: [HACKERS] Failing start-up archive recovery at Standby mode in PG9.2.4

2013-04-24 Thread Andres Freund
ello, On 2013-04-24 17:43:39 +0900, KONDO Mitsumasa wrote: Hi, I find problem about failing start-up achive recovery at Standby mode in PG9.2.4 streaming replication. I test same problem in PG9.2.3. But it is not occerd... cp: cannot stat `../arc/00030013':

Re: [HACKERS] Fast promotion, loose ends

2013-04-24 Thread Simon Riggs
On 24 April 2013 09:53, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 24.04.2013 11:46, Simon Riggs wrote: On 24 April 2013 09:32, Heikki Linnakangashlinnakan...@vmware.com wrote: pg_ctl already checks versions, so I don't see the point. The point is, if you do pgsql93/bin/pg_ctl

Re: [HACKERS] Failing start-up archive recovery at Standby mode in PG9.2.4

2013-04-24 Thread Kyotaro HORIGUCHI
Hello, cp: cannot stat `../arc/00030013': そのようなファイルやディレクトリはありません [Standby] 2013-04-22 01:27:25 EDTLOG: 0: restored log file 00020013 from archive I can't read the error message here, but this looks suspicious. The error message is No such file or

Re: [HACKERS] Failing start-up archive recovery at Standby mode in PG9.2.4

2013-04-24 Thread Kyotaro HORIGUCHI
Sorry, caller XLogFileOpen successfully ets and returns fd for the filename The caller is XLogFileRead in this case. # and 'ets' is gets, of course. regards, -- Kyotaro Horiguchi -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] Failing start-up archive recovery at Standby mode in PG9.2.4

2013-04-24 Thread Andres Freund
On 2013-04-24 19:16:12 +0900, Kyotaro HORIGUCHI wrote: Hello, cp: cannot stat `../arc/00030013': そのようなファイルやディレクトリはありません [Standby] 2013-04-22 01:27:25 EDTLOG: 0: restored log file 00020013 from archive I can't read the error message here, but

Re: [HACKERS] Failing start-up archive recovery at Standby mode in PG9.2.4

2013-04-24 Thread Kyotaro HORIGUCHI
Oops, But thats not what happening here, afaics the restore log file ... message is only printed if the returncode is 0. You're right. 'cp nonexistent somewhere' exits with the status code 1 (or 256?). The quoted log lines simply show that segment for tli=3 did not exist and that for tli=2

Re: [HACKERS] GSOC Student Project Idea

2013-04-24 Thread Florian Pflug
On Apr23, 2013, at 23:25 , Alexander Korotkov aekorot...@gmail.com wrote: I've taken a brief look on the paper and implementation. As I can see iDistance implements some global building strategy. I mean, for example, it selects some point, calculates distances from selected point to all

Re: [HACKERS] Proposal to add --single-row to psql

2013-04-24 Thread Fabrízio de Royes Mello
On Tue, Apr 23, 2013 at 1:05 PM, Tom Lane t...@sss.pgh.pa.us wrote: Isn't there already a way to set FETCH_COUNT from the command line? (ie, I think there's a generic variable-assignment facility that could do this) Christopher, Tom is all right... from psql [1] command line we can do that:

Re: [HACKERS] REFRESH MATERIALIZED VIEW command in PL block hitting Assert

2013-04-24 Thread Jeevan Chalke
On Wed, Apr 24, 2013 at 3:04 AM, Kevin Grittner kgri...@ymail.com wrote: Jeevan Chalke jeevan.cha...@enterprisedb.com wrote: On Mon, Apr 22, 2013 at 6:41 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-04-22 18:35:04 +0530, Jeevan Chalke wrote: I have observed that following

Re: [HACKERS] high io BUT huge amount of free memory

2013-04-24 Thread Robert Haas
On Tue, Apr 23, 2013 at 10:50 AM, Shaun Thomas stho...@optionshouse.com wrote: This is most likely a NUMA issue. There really seems to be some kind of horrible flaw in the Linux kernel when it comes to properly handling NUMA on large memory systems. Are you referring to the fact that

[HACKERS] missing time.h include in psql/command.c since the addition of \watch

2013-04-24 Thread Andres Freund
Hi, our internal testbuilds show a new warning on windows: src\bin\psql\command.c(2617): warning C4013: 'time' undefined; assuming extern returning int [C:\jenkins\workspace\andres_git.postgresql.org_windows\BT\release\SL_OS\windows\TA\x86\TO\xp\psql.vcxproj] src\bin\psql\command.c(2619):

Re: [HACKERS] 9.3 release notes suggestions

2013-04-24 Thread Robert Haas
On Tue, Apr 23, 2013 at 11:41 PM, Bruce Momjian br...@momjian.us wrote: Thanks for the many suggestions on improving the 9.3 release notes. There were many ideas I would have never thought of. Please keep the suggestions coming. Bruce, Thanks for writing them! -- Robert Haas EnterpriseDB:

Re: [HACKERS] high io BUT huge amount of free memory

2013-04-24 Thread Shaun Thomas
On 04/24/2013 08:24 AM, Robert Haas wrote: Are you referring to the fact that vm.zone_reclaim_mode = 1 is an idiotic default? Well... it is. But even on systems where it's not the default or is explicitly disabled, there's just something hideously wrong with NUMA in general. Take a look at

Re: [HACKERS] REFRESH MATERIALIZED VIEW command in PL block hitting Assert

2013-04-24 Thread Kevin Grittner
Jeevan Chalke jeevan.cha...@enterprisedb.com wrote: On Wed, Apr 24, 2013 at 3:04 AM, Kevin Grittner kgri...@ymail.com wrote: Any objections to the attached to fix this issue? Nope. Fine with me. Pushed.  Thanks for the report! -- Kevin Grittner EnterpriseDB: http://www.enterprisedb.com The

Re: [HACKERS] high io BUT huge amount of free memory

2013-04-24 Thread Andres Freund
On 2013-04-24 08:39:09 -0500, Shaun Thomas wrote: The memory pressure code in Linux is extremely fucked up. I can't find it right now, but the memory management algorithm makes some pretty ridiculous assumptions once you pass half memory usage, regarding what is in active and inactive cache.

Re: [HACKERS] high io BUT huge amount of free memory

2013-04-24 Thread Shaun Thomas
On 04/24/2013 08:49 AM, Andres Freund wrote: Uh. Ranting can be rather healthy thing every now and then and it good for the soul and such. But. Did you actually try reporting those issues? That's actually part of the problem. How do you report: Throwing a lot of processes at a high-memory

Re: [HACKERS] high io BUT huge amount of free memory

2013-04-24 Thread Andres Freund
On 2013-04-24 09:06:39 -0500, Shaun Thomas wrote: On 04/24/2013 08:49 AM, Andres Freund wrote: Uh. Ranting can be rather healthy thing every now and then and it good for the soul and such. But. Did you actually try reporting those issues? That's actually part of the problem. How do you

[HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Timothy Garnett
Hi All, Currently the -j option to pg_restore, which allows for parallelization in the restore, can only be used if the input file is a regular file and not, for ex., a pipe. However this is a pretty common occurrence for us (usually in the form of pg_dump | pg_restore to copy an individual

Re: [HACKERS] missing time.h include in psql/command.c since the addition of \watch

2013-04-24 Thread Heikki Linnakangas
On 24.04.2013 16:27, Andres Freund wrote: Hi, our internal testbuilds show a new warning on windows: src\bin\psql\command.c(2617): warning C4013: 'time' undefined; assuming extern returning int

Re: [HACKERS] putting a bgworker to rest

2013-04-24 Thread Dimitri Fontaine
Andres Freund and...@2ndquadrant.com writes: How would postmaster know when to restart a worker that stopped? I had imagined we would assign some return codes special meaning. Currently 0 basically means restart immediately, 1 means crashed, wait for some time, everything else results in a

Re: [HACKERS] 9.3 Beta1 status report

2013-04-24 Thread Heikki Linnakangas
On 24.04.2013 06:22, Bruce Momjian wrote: On Tue, Apr 23, 2013 at 06:56:34PM -0300, Alvaro Herrera wrote: Bruce Momjian wrote: On Tue, Apr 23, 2013 at 05:04:15PM -0400, Bruce Momjian wrote: Do we usually repeat the changes listed in the backwards compatibility section later, in the Changes

Re: [HACKERS] putting a bgworker to rest

2013-04-24 Thread Robert Haas
On Tue, Apr 23, 2013 at 1:22 PM, Andres Freund and...@2ndquadrant.com wrote: So a done worker would never be restarted, until postmaster sees a crash or is itself restarted? I guess that'd be useful for workers running during recovery, which terminate when recovery completes. Is that your

Examples Re: [HACKERS] Bug Fix: COLLATE with multiple ORDER BYs in aggregates

2013-04-24 Thread David Fetter
On Tue, Apr 23, 2013 at 09:57:27AM -0700, David Fetter wrote: Folks, While testing the upcoming FILTER clause for aggregates, Erik Rijkers uncovered a long-standing bug in $subject, namely that this case wasn't handled. Please find attached a patch by Andrew Gierth and myself which fixes

[HACKERS] Re[2]: [HACKERS] high io BUT huge amount of free memory

2013-04-24 Thread Миша Тюрин
thanks a lot for responses 1) just remind my case Intel 32 core = 2*8 *2threads Linux 2.6.32-5-amd64 #1 SMP Sun May 6 04:00:17 UTC 2012 x86_64 GNU/Linux PostgreSQL 9.2.2 on x86_64-unknown-linux-gnu, compiled by gcc-4.4.real (Debian 4.4.5-8) 4.4.5, 64-bit shared_buffers 64GB / constant hit

[HACKERS] Re[3]: [HACKERS] high io BUT huge amount of free memory

2013-04-24 Thread Миша Тюрин
typo if ( user cpu + io wait ) is ~140% then i have ~140GB free. 140% === 1400% if ~14 cores are busy then ~140GB is free 10GB per process hmmm... -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

[HACKERS] Re: [HACKERS] Re[3]: [HACKERS] high io BUT huge amount of free memory

2013-04-24 Thread Миша Тюрин
vm state root@avi-sql09:~# /sbin/sysctl -a|grep vm vm.overcommit_memory = 0 vm.panic_on_oom = 0 vm.oom_kill_allocating_task = 0 vm.oom_dump_tasks = 0 vm.overcommit_ratio = 50 vm.page-cluster = 3 vm.dirty_background_ratio = 10 vm.dirty_background_bytes = 0 vm.dirty_ratio = 20 vm.dirty_bytes = 0

Re: [HACKERS] GSOC Student Project Idea

2013-04-24 Thread Michael Schuh
On Wed, Apr 24, 2013 at 5:31 AM, Florian Pflug f...@phlo.org wrote: On Apr23, 2013, at 23:25 , Alexander Korotkov aekorot...@gmail.com wrote: I've taken a brief look on the paper and implementation. As I can see iDistance implements some global building strategy. I mean, for example, it

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Andrew Dunstan
On 04/23/2013 07:53 PM, Timothy Garnett wrote: Hi All, Currently the -j option to pg_restore, which allows for parallelization in the restore, can only be used if the input file is a regular file and not, for ex., a pipe. However this is a pretty common occurrence for us (usually in the

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Dimitri Fontaine
Andrew Dunstan and...@dunslane.net writes: On 04/23/2013 07:53 PM, Timothy Garnett wrote: Anyways, the question is if people think this is generally useful. If so I can clean up the preferred choice a bit and rebase it off of master, etc. I find this idea very useful yes. Another idea would

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Andrew Dunstan
On 04/24/2013 03:40 PM, Dimitri Fontaine wrote: Andrew Dunstan and...@dunslane.net writes: On 04/23/2013 07:53 PM, Timothy Garnett wrote: Anyways, the question is if people think this is generally useful. If so I can clean up the preferred choice a bit and rebase it off of master, etc. I

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Andrew Dunstan
On 04/24/2013 03:49 PM, Andrew Dunstan wrote: On 04/24/2013 03:40 PM, Dimitri Fontaine wrote: Andrew Dunstan and...@dunslane.net writes: On 04/23/2013 07:53 PM, Timothy Garnett wrote: Anyways, the question is if people think this is generally useful. If so I can clean up the preferred

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Stefan Kaltenbrunner
On 04/24/2013 09:51 PM, Andrew Dunstan wrote: On 04/24/2013 03:49 PM, Andrew Dunstan wrote: On 04/24/2013 03:40 PM, Dimitri Fontaine wrote: Andrew Dunstan and...@dunslane.net writes: On 04/23/2013 07:53 PM, Timothy Garnett wrote: Anyways, the question is if people think this is generally

Re: [HACKERS] Enabling Checksums

2013-04-24 Thread Jeff Davis
On Wed, 2013-04-24 at 08:20 +0100, Simon Riggs wrote: On 24 April 2013 01:10, Jeff Davis pg...@j-davis.com wrote: I'd prefer that it was some kind of a checksum ID code -- e.g. 0 for no checksum, 1 for FNV-1a-SR3, etc. That would allow us to release 9.4 with a new algorithm without forcing

Re: [HACKERS] Enabling Checksums

2013-04-24 Thread Simon Riggs
On 24 April 2013 21:06, Jeff Davis pg...@j-davis.com wrote: What goal are you trying to accomplish with this patch? That we might need to patch the checksum version on a production release. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support,

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Joachim Wieland
On Wed, Apr 24, 2013 at 4:05 PM, Stefan Kaltenbrunner ste...@kaltenbrunner.cc wrote: What might make sense is something like pg_dump_restore which would have no intermediate storage at all, just pump the data etc from one source to another in parallel. But I pity the poor guy who has to

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Claudio Freire
On Wed, Apr 24, 2013 at 6:47 PM, Joachim Wieland j...@mcknight.de wrote: On Wed, Apr 24, 2013 at 4:05 PM, Stefan Kaltenbrunner ste...@kaltenbrunner.cc wrote: What might make sense is something like pg_dump_restore which would have no intermediate storage at all, just pump the data etc from

Re: [HACKERS] Proposal to add --single-row to psql

2013-04-24 Thread Christopher Manning
Fabrízio and Tom, I know that you can use --variable=FETCH_COUNT=1 from the psql command line, but internally that uses a CURSOR to batch the rows and [Redshift doesn't support CURSOR]( https://forums.aws.amazon.com/thread.jspa?threadID=122664tstart=0) so it's not an option when using psql

[HACKERS] danger of stats_temp_directory = /dev/shm

2013-04-24 Thread Jeff Janes
With the stats file split patch 187492b6c2e8cafc5 introduced in 9.3dev, now after a crash the postmaster will try to delete all files in the directory stats_temp_directory. When that is just a subdirectory of PGDATA, this is fine. But it seems rather hostile when it is set to a shared directory,

Re: [HACKERS] Failing start-up archive recovery at Standby mode in PG9.2.4

2013-04-24 Thread Kyotaro HORIGUCHI
I had a bit look on it and came up with an hypothesis.. umm or a scenario. == Just after restartpoint, too old xlog files are recycled but its page header has old content, specifically, xlogid and xrecoff. Plus, if the master's LSN is at the head of new segment file, the file for the segment

Re: [HACKERS] danger of stats_temp_directory = /dev/shm

2013-04-24 Thread Alvaro Herrera
Jeff Janes escribió: With the stats file split patch 187492b6c2e8cafc5 introduced in 9.3dev, now after a crash the postmaster will try to delete all files in the directory stats_temp_directory. When that is just a subdirectory of PGDATA, this is fine. But it seems rather hostile when it is

Re: [HACKERS] Failing start-up archive recovery at Standby mode in PG9.2.4

2013-04-24 Thread KONDO Mitsumasa
Hi, I find problem about failing start-up achive recovery at Standby mode in PG9.2.4 streaming replication. I test same problem in PG9.2.3. But it is not occerd... cp: cannot stat `../arc/00030013': そのようなファイルやディレクトリはありません [Standby] 2013-04-22 01:27:25 EDTLOG: 0:

Re: [HACKERS] Enabling Checksums

2013-04-24 Thread Jeff Davis
On Wed, 2013-04-24 at 21:09 +0100, Simon Riggs wrote: On 24 April 2013 21:06, Jeff Davis pg...@j-davis.com wrote: What goal are you trying to accomplish with this patch? That we might need to patch the checksum version on a production release. Oh, I see. I don't think we need two output

Re: [HACKERS] danger of stats_temp_directory = /dev/shm

2013-04-24 Thread Tom Lane
Alvaro Herrera alvhe...@2ndquadrant.com writes: Jeff Janes escribió: With the stats file split patch 187492b6c2e8cafc5 introduced in 9.3dev, now after a crash the postmaster will try to delete all files in the directory stats_temp_directory. When that is just a subdirectory of PGDATA, this is

[HACKERS] Please add discussion topics for cluster-hackers meeting

2013-04-24 Thread Josh Berkus
Folks, The 2013 cluster-hackers meeting is less than a month away. If you are attending, and have topics of common interest to discuss, such as core APIs (i.e. event triggers are in, at least partway), please add your topic to the wiki here: