Re: [HACKERS] [REVIEW] Generate column names for subquery expressions

2011-09-29 Thread Kyotaro HORIGUCHI
Hi, PS: When you send a review, you should add the author's email to the To: line to make sure they see it. I noticed your email only today because it was in a new thread and not addressed to me directly. Thanks for the advise. I will do so after this. Good catch, a new patch is attached.

Re: [HACKERS] [REVIEW] pg_last_xact_insert_timestamp

2011-09-29 Thread Kyotaro HORIGUCHI
Sorry for late to re-review. One question is remaind, Q1: The shmem entry for timestamp is not initialized on allocating. Is this OK? (I don't know that for OSs other than Linux) And zeroing double field is OK for all OSs? CreateSharedBackendStatus() initializes that shmem entries by

Re: [HACKERS] Feature proposal: www_fdw

2011-09-29 Thread Alexander Soudakov
On Thu, Sep 29, 2011 at 1:20 AM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Florian Pflug f...@phlo.org wrote: On Sep28, 2011, at 15:32 , Alexander Soudakov wrote: Here you can find www_fdw feature documentation: http://wiki.postgresql.org/wiki/WWW_FDW Certainly looks useful (as a

Re: [HACKERS] [v9.2] make_greater_string() does not return a string in some cases

2011-09-29 Thread Kyotaro HORIGUCHI
This is new version of make_greater_string patch. 1. wchar.c:1532 pg_wchar_table: Restore the pg_wchar_table. 2. wchar.c:1371 pg_utf8_increment: Remove dangerous memcpy, but one memcpy is left because it's safe. Remove code check after increment. 3. wchar.c:1429 pg_eucjp_increment: Remove

Re: [HACKERS] bug of recovery?

2011-09-29 Thread Fujii Masao
On Tue, Sep 27, 2011 at 8:06 PM, Florian Pflug f...@phlo.org wrote: On Sep27, 2011, at 07:59 , Heikki Linnakangas wrote: On 27.09.2011 00:28, Florian Pflug wrote: On Sep26, 2011, at 22:39 , Tom Lane wrote: It might be worthwhile to invoke XLogCheckInvalidPages() as soon as we (think we) have

Re: [HACKERS] bug of recovery?

2011-09-29 Thread Simon Riggs
On Thu, Sep 29, 2011 at 12:31 PM, Fujii Masao masao.fu...@gmail.com wrote: On Tue, Sep 27, 2011 at 8:06 PM, Florian Pflug f...@phlo.org wrote: On Sep27, 2011, at 07:59 , Heikki Linnakangas wrote: On 27.09.2011 00:28, Florian Pflug wrote: On Sep26, 2011, at 22:39 , Tom Lane wrote: It might be

Re: [HACKERS] pg_upgrade - add config directory setting

2011-09-29 Thread Bruce Momjian
Peter Eisentraut wrote: On ons, 2011-09-28 at 11:53 -0300, Alvaro Herrera wrote: Excerpts from Peter Eisentraut's message of mi? sep 28 04:49:43 -0300 2011: On tis, 2011-09-27 at 16:13 -0700, Steve Crawford wrote: It would perhaps be useful to add optional --old-confdir and

Re: [HACKERS] Removing savepointLevel from TransactionState

2011-09-29 Thread Alvaro Herrera
Excerpts from Tom Lane's message of jue sep 29 02:11:52 -0300 2011: Gurjeet Singh singh.gurj...@gmail.com writes: I noticed that the savepointLevel member of TransactionStateData struct is initialized to 0 from TopTransactionStateData, and never incremented or decremented afterwards.

Re: [HACKERS] [REVIEW] pg_last_xact_insert_timestamp

2011-09-29 Thread Fujii Masao
On Thu, Sep 29, 2011 at 5:20 PM, Kyotaro HORIGUCHI horiguchi.kyot...@oss.ntt.co.jp wrote: Sorry for late to re-review. Thanks! Nevertheless this is ok for all OSs, I don't know whether initializing TimestampTz(double, int64 is ok) field with 8 bytes zeros is OK or not, for all platforms.

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread MauMau
From: Merlin Moncure mmonc...@gmail.com can we see all of your memory settings plus physical memory? the solution is probably going to be reducing shared buffers an/or adding physical memory. Thank you for your response. The amount of physical memory is 8GB, which is enough for the workload. I

Re: [HACKERS] Feature proposal: www_fdw

2011-09-29 Thread Dickson S. Guedes
2011/9/28 Florian Pflug f...@phlo.org: On Sep28, 2011, at 15:32 , Alexander Soudakov wrote: Here you can find www_fdw feature documentation: http://wiki.postgresql.org/wiki/WWW_FDW Certainly looks useful (as a third-party extension, as others have already pointed out) +1. What I didn't

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread Alvaro Herrera
Excerpts from MauMau's message of jue sep 29 09:23:48 -0300 2011: The amount of physical memory is 8GB, which is enough for the workload. I asked the customer for the output of SHOW ALL, but I haven't received it yet. However, shared_buffers should be less than 1.6GB because, as I wrote

Re: [HACKERS] pg_upgrade - add config directory setting

2011-09-29 Thread Bruce Momjian
Bruce Momjian wrote: Peter Eisentraut wrote: On ons, 2011-09-28 at 11:53 -0300, Alvaro Herrera wrote: Excerpts from Peter Eisentraut's message of mi? sep 28 04:49:43 -0300 2011: On tis, 2011-09-27 at 16:13 -0700, Steve Crawford wrote: It would perhaps be useful to add optional

Re: [HACKERS] Removing savepointLevel from TransactionState

2011-09-29 Thread Gurjeet Singh
On Thu, Sep 29, 2011 at 8:10 AM, Alvaro Herrera alvhe...@commandprompt.comwrote: Excerpts from Tom Lane's message of jue sep 29 02:11:52 -0300 2011: Gurjeet Singh singh.gurj...@gmail.com writes: I noticed that the savepointLevel member of TransactionStateData struct is initialized to 0

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread MauMau
From: Alvaro Herrera alvhe...@commandprompt.com You don't really know this; some operating systems (Linux in particular) does not show shared memory as in use by a process until it is accessed. It may very well have well over 1.6 GB of shared_buffers, yet not show that in VIRT. Oh, really?

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread Merlin Moncure
On Thu, Sep 29, 2011 at 7:23 AM, MauMau maumau...@gmail.com wrote: From: Merlin Moncure mmonc...@gmail.com can we see all of your memory settings plus physical memory?  the solution is probably going to be reducing shared buffers an/or adding physical memory. Thank you for your response.

Re: [HACKERS] pg_upgrade - add config directory setting

2011-09-29 Thread Alvaro Herrera
Excerpts from Bruce Momjian's message of jue sep 29 09:56:09 -0300 2011: Thinking some more, I don't need to know the data directory while the server is down --- I already am starting it. pg_upgrade starts both old and new servers during its check phase, and it could look up the

Re: [HACKERS] Feature proposal: www_fdw

2011-09-29 Thread Florian Pflug
On Sep29, 2011, at 14:45 , Dickson S. Guedes wrote: I'm working on a google_contacts_fdw to google contacts api [1] but stopped in the authentication design. As you can see in [2], for google api, you should get an authorization token and store the Auth value to use latter on the same session.

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread Robert Haas
On Thu, Sep 29, 2011 at 9:39 AM, Merlin Moncure mmonc...@gmail.com wrote: Like I said, this doesn't really come up this often but the 'real' solution in terms of postgrs is probably some kind of upper bound in the amount of cache memory used plus some intelligence in the cache implementation.  

Re: [HACKERS] bug of recovery?

2011-09-29 Thread Florian Pflug
On Sep29, 2011, at 13:49 , Simon Riggs wrote: This worries me slightly now though because the patch makes us PANIC in a place we didn't used to and once we do that we cannot restart the server at all. Are we sure we want that? It's certainly a great way to shake down errors in other code...

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread MauMau
From: Merlin Moncure mmonc...@gmail.com -- Oh -- I missed earlier that this was 32 bit o/s. Well, I'd consider drastically reducing shared buffers, down to say 256-512mb range. Postgres function plans and various other structures, tables,

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread Merlin Moncure
On Thu, Sep 29, 2011 at 8:59 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, Sep 29, 2011 at 9:39 AM, Merlin Moncure mmonc...@gmail.com wrote: Like I said, this doesn't really come up this often but the 'real' solution in terms of postgrs is probably some kind of upper bound in the amount

Re: [HACKERS] pg_upgrade - add config directory setting

2011-09-29 Thread Mr. Aaron W. Swenson
On Thu, Sep 29, 2011 at 10:44:29AM -0300, Alvaro Herrera wrote: Excerpts from Bruce Momjian's message of jue sep 29 09:56:09 -0300 2011: Thinking some more, I don't need to know the data directory while the server is down --- I already am starting it. pg_upgrade starts both old and new

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread Tom Lane
MauMau maumau...@gmail.com writes: Anyway, I'd appreciate if anyone could tell me about RelCache/SysCache. As far as I read the code, PostgreSQL seems to use memory for RelCache/SysCache without limit until the relations are dropped. That's correct. We used to have a limit on the size of

Re: [HACKERS] Feature proposal: www_fdw

2011-09-29 Thread Dickson S. Guedes
2011/9/29 Florian Pflug f...@phlo.org: You could use a hash table, allocated in the top-level memory context, to store one authentication token per combination of server and local user. In fact I started something in this way, with ldap_fdw, stashing the connection away using memory context and

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: ... It seems that we used to have some kind of LRU algorithm to prevent excessive memory usage, but we rippped it out because it was too expensive (see commit 8b9bc234ad43dfa788bde40ebf12e94f16556b7f). Not only was it too expensive, but performance

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread MauMau
From: Tom Lane t...@sss.pgh.pa.us That's correct. We used to have a limit on the size of catcache (if memory serves, it was something like 5000 entries). We got rid of it after observing that performance fell off a cliff as soon as you had a working set larger than the cache limit. Trust me,

Re: [HACKERS] Feature proposal: www_fdw

2011-09-29 Thread Florian Pflug
On Sep29, 2011, at 16:43 , Dickson S. Guedes wrote: 2011/9/29 Florian Pflug f...@phlo.org: You could use a hash table, allocated in the top-level memory context, to store one authentication token per combination of server and local user. In fact I started something in this way, with

Re: [HACKERS] pg_upgrade - add config directory setting

2011-09-29 Thread Bruce Momjian
Mr. Aaron W. Swenson wrote: -- Start of PGP signed section. On Thu, Sep 29, 2011 at 10:44:29AM -0300, Alvaro Herrera wrote: Excerpts from Bruce Momjian's message of jue sep 29 09:56:09 -0300 2011: Thinking some more, I don't need to know the data directory while the server is down

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread Robert Haas
On Thu, Sep 29, 2011 at 10:45 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: ... It seems that we used to have some kind of LRU algorithm to prevent excessive memory usage, but we rippped it out because it was too expensive (see commit

Re: [HACKERS] pg_regress input/output directory option

2011-09-29 Thread Robert Haas
On Wed, Sep 28, 2011 at 12:21 AM, Michael Paquier michael.paqu...@gmail.com wrote: Why are there no options in_regress to specify the directory where input and output data are located? Such options would bring more flexibility when running regressions without make check/installcheck for an

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread Merlin Moncure
On Thu, Sep 29, 2011 at 10:22 AM, Robert Haas robertmh...@gmail.com wrote: I can't really explain why people seem to keep wanting to create hundreds of thousands or even millions of tables, but it's not like MauMau's customer is the first one to try to do this, and I'm sure they won't be the

Re: [HACKERS] Hot Backup with rsync fails at pg_clog if under load

2011-09-29 Thread Linas Virbalas
Linas, could you capture the output of pg_controldata *and* increase the log level to DEBUG1 on the standby? We should then see nextXid value of the checkpoint the recovery is starting from. I'll try to do that whenever I'm in that territory again... Incidentally, recently there was a lot of

[HACKERS] have SLRU truncation use callbacks

2011-09-29 Thread Alvaro Herrera
Hi, Currently, the mechanism that SLRU uses to truncate directory entries is hardcoded in SlruScanDirectory. It receives a cutoff page number and a boolean flag; segments found in the SLRU directory that are below the cutoff are deleted -- iff the flag is true. This is fine for most current

[HACKERS] Temporary tables and in-memory use

2011-09-29 Thread Marios Vodas
Hello, If I'm not wrong, temporary tables stay in memory if they do not go over temp_buffers limit (e.g. if temp_buffers is 2GB and the size of the table is 300MB the table will remain in memory). What if a column is variable length (e.g. text), how does this column stay in-memory since it should

Re: [HACKERS] Temporary tables and in-memory use

2011-09-29 Thread Tom Lane
Marios Vodas mvo...@gmail.com writes: If I'm not wrong, temporary tables stay in memory if they do not go over temp_buffers limit (e.g. if temp_buffers is 2GB and the size of the table is 300MB the table will remain in memory). What if a column is variable length (e.g. text), how does this

Re: [HACKERS] pg_upgrade - add config directory setting

2011-09-29 Thread Steve Crawford
On 09/29/2011 08:20 AM, Bruce Momjian wrote: ... 1 document the limitation and require users to use symlinks 2 add a --old/new-configdir parameter to pg_upgrade 3 have pg_upgrade find the real data dir by starting the server 4 add a flag to some tool to return the real data dir, and

Re: [HACKERS] Temporary tables and in-memory use

2011-09-29 Thread Marios Vodas
Thank you. The setup is intended for one user environment for complex queries and operations that's why I wrote 2GB temp_buffers! Thank you again, I really appreciate it. Marios On 29/9/2011 7:55 μμ, Tom Lane wrote: Marios Vodasmvo...@gmail.com writes: If I'm not wrong, temporary tables stay

Re: [HACKERS] Displaying accumulated autovacuum cost

2011-09-29 Thread Alvaro Herrera
I reviewed this patch. My question for you is: does it make sense to enable to reporting of write rate even when vacuum cost accounting is enabled? In my opinion it would be useful to do so. If you agree, please submit an updated patch. -- Álvaro Herrera alvhe...@commandprompt.com The

Re: [HACKERS] have SLRU truncation use callbacks

2011-09-29 Thread Kevin Grittner
Alvaro Herrera alvhe...@alvh.no-ip.org wrote: But I think these changes stand on their own, merely on code clarity grounds). After a quick scan, I think it helps with that. This was a messy area to deal with in SSI given the old API; with this change I think we could make that part of the

Re: [HACKERS] Hot Backup with rsync fails at pg_clog if under load

2011-09-29 Thread Florian Pflug
On Sep29, 2011, at 17:44 , Linas Virbalas wrote: I also checked what rsync does when a file vanishes after rsync computed the file list, but before it is sent. rsync 3.0.7 on OSX, at least, complains loudly, and doesn't sync the file. It BTW also exits non-zero, with a special exit code for

Re: [HACKERS] Re: Optimizing pg_trgm makesign() (was Re: WIP: Fast GiST index build)

2011-09-29 Thread Kevin Grittner
Kevin Grittner kevin.gritt...@wicourts.gov wrote: Tom Lane wrote: Hmm, why is that patch the one posted for review, when several better versions were already discussed? See thread starting here: http://archives.postgresql.org/pgsql-hackers/2011-07/msg00028.php The patch I reviewed was

Re: [HACKERS] Updated version of pg_receivexlog

2011-09-29 Thread Magnus Hagander
On Thu, Sep 29, 2011 at 01:55, Jaime Casanova ja...@2ndquadrant.com wrote: On Wed, Sep 28, 2011 at 12:50 PM, Magnus Hagander mag...@hagander.net wrote: pg_receivexlog worked good in my tests. pg_basebackup with --xlog=stream gives me an already recycled wal segment message (note that the

Re: [HACKERS] [v9.2] Object access hooks with arguments support (v1)

2011-09-29 Thread Kohei KaiGai
I noticed that the previous revision does not provide any way to inform the modules name of foreign server, even if foreign table was created, on the OAT_POST_CREATE hook. So, I modified the invocation at heap_create_with_catalog to deliver this information to the modules. Rest of parts were

Re: [HACKERS] pg_upgrade - add config directory setting

2011-09-29 Thread Bruce Momjian
Steve Crawford wrote: On 09/29/2011 08:20 AM, Bruce Momjian wrote: ... 1 document the limitation and require users to use symlinks 2 add a --old/new-configdir parameter to pg_upgrade 3 have pg_upgrade find the real data dir by starting the server 4 add a flag to some tool to return

Re: [HACKERS] Re: Optimizing pg_trgm makesign() (was Re: WIP: Fast GiST index build)

2011-09-29 Thread Heikki Linnakangas
On 29.09.2011 20:27, Kevin Grittner wrote: Heikki's second version, a more radical revision optimized for 64 bit systems, blows up on a 32 bit compile, writing off the end of the structure. Personally, I'd be OK with sacrificing some performance for 32 bit systems to get better performance on

Re: [HACKERS] Re: Optimizing pg_trgm makesign() (was Re: WIP: Fast GiST index build)

2011-09-29 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Looking at the big picture, however, the real problem with all those makesign() calls is that they happen in the first place. They happen when gist needs to choose which child page to place a new tuple on. It calls the penalty

Re: [HACKERS] Re: Optimizing pg_trgm makesign() (was Re: WIP: Fast GiST index build)

2011-09-29 Thread Alexander Korotkov
On Fri, Sep 30, 2011 at 1:08 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: At every call, gtrgm_penalty() has to calculate the signature for newitem, using makesign(). That's an enormous waste of effort, but there's currently no way gtrgm_penalty() to avoid that. If we

Re: [HACKERS] pg_upgrade - add config directory setting

2011-09-29 Thread Tom Lane
Bruce Momjian br...@momjian.us writes: pg_upgrade is not about to start reading through postgresql.conf looking for a definition for data_directory --- there are too many cases where this could go wrong. It would need a full postgresql.conf parser. Yeah. I think the only sensible way to do

Re: [HACKERS] fix for pg_upgrade

2011-09-29 Thread Bruce Momjian
Alvaro Herrera wrote: Excerpts from Bruce Momjian's message of miC3A9 sep 28 13:48:28 -0300 2011: Bruce Momjian wrote: OK, so it fails for all tables and you are using the newest version. Thanks for all your work. I am now guessing that pg_upgrade 9.1.X is just broken on

Re: [HACKERS] Re: Optimizing pg_trgm makesign() (was Re: WIP: Fast GiST index build)

2011-09-29 Thread Alexander Korotkov
On Fri, Sep 30, 2011 at 1:16 AM, Tom Lane t...@sss.pgh.pa.us wrote: Hmm. Are there any other datatypes for which the penalty function has to duplicate effort? I'm disinclined to fool with this if pg_trgm is the only example ... but if it's not, maybe we should do something about that

Re: [HACKERS] pg_upgrade - add config directory setting

2011-09-29 Thread Bruce Momjian
Tom Lane wrote: Bruce Momjian br...@momjian.us writes: pg_upgrade is not about to start reading through postgresql.conf looking for a definition for data_directory --- there are too many cases where this could go wrong. It would need a full postgresql.conf parser. Yeah. I think the

Re: [HACKERS] pg_upgrade - add config directory setting

2011-09-29 Thread Tom Lane
Bruce Momjian br...@momjian.us writes: Tom Lane wrote: Yeah. I think the only sensible way to do this would be to provide an operating mode for the postgres executable that would just parse the config file and spit out requested values. That would certainly solve the problem, though it

Re: [HACKERS] Re: Optimizing pg_trgm makesign() (was Re: WIP: Fast GiST index build)

2011-09-29 Thread Tom Lane
Alexander Korotkov aekorot...@gmail.com writes: On Fri, Sep 30, 2011 at 1:08 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: At every call, gtrgm_penalty() has to calculate the signature for newitem, using makesign(). That's an enormous waste of effort, but there's currently

Re: [HACKERS] pg_regress input/output directory option

2011-09-29 Thread Michael Paquier
On Fri, Sep 30, 2011 at 12:37 AM, Robert Haas robertmh...@gmail.com wrote: On Wed, Sep 28, 2011 at 12:21 AM, Michael Paquier michael.paqu...@gmail.com wrote: Why are there no options in_regress to specify the directory where input and output data are located? Such options would bring

Re: [HACKERS] pg_upgrade - add config directory setting

2011-09-29 Thread Bruce Momjian
Tom Lane wrote: Bruce Momjian br...@momjian.us writes: Tom Lane wrote: Yeah. I think the only sensible way to do this would be to provide an operating mode for the postgres executable that would just parse the config file and spit out requested values. That would certainly solve the

Re: [HACKERS] REVIEW proposal: a validator for configuration files

2011-09-29 Thread Alexander
On 09/10/2011 11:39 AM, Alexey Klyukin wrote: Hi Andy, On Sep 7, 2011, at 6:40 AM, Andy Colson wrote: Hi Alexey, I was taking a quick look at this patch, and have a question for ya. ... Where did the other warnings go? Its right though, line 570 is bad. It also seems to have killed the

Re: [HACKERS] bug of recovery?

2011-09-29 Thread Fujii Masao
On Thu, Sep 29, 2011 at 11:12 PM, Florian Pflug f...@phlo.org wrote: On Sep29, 2011, at 13:49 , Simon Riggs wrote: This worries me slightly now though because the patch makes us PANIC in a place we didn't used to and once we do that we cannot restart the server at all. Are we sure we want

Re: [HACKERS] [REVIEW] pg_last_xact_insert_timestamp

2011-09-29 Thread Kyotaro HORIGUCHI
Hello, I understand that it has been at least practically no problem. Ok, I send this patch to comitters. Thanks for your dealing with nuisance questions. At Thu, 29 Sep 2011 21:21:32 +0900, Fujii Masao masao.fu...@gmail.com wrote in