Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-03 Thread Simon Riggs
On Fri, 2010-09-03 at 12:50 +0900, Fujii Masao wrote: On Thu, Sep 2, 2010 at 11:32 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I understand what you're after, the idea of being able to set synchronization level on a per-transaction basis is cool. But I haven't seen a

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-03 Thread Fujii Masao
On Thu, Sep 2, 2010 at 7:24 PM, Fujii Masao masao.fu...@gmail.com wrote: I propose a configuration file standbys.conf, in the master: # STANDBY NAME    SYNCHRONOUS   TIMEOUT importantreplica  yes           100ms tempcopy          no            10s Seems good. In fact, instead of yes/no,

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-03 Thread Heikki Linnakangas
On 03/09/10 09:36, Simon Riggs wrote: On Fri, 2010-09-03 at 12:50 +0900, Fujii Masao wrote: That design would affect what the standby should reply. If we choose async/recv/fsync/replay on a per-transaction basis, the standby should send multiple LSNs and the master needs to decide when

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-03 Thread Fujii Masao
On Fri, Sep 3, 2010 at 3:36 PM, Simon Riggs si...@2ndquadrant.com wrote: The former seems to be more useful, but triples the number of ACK from the standby. I'm not sure whether its overhead is ignorable, especially when the distance between the master and the standby is very long. No, it

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-03 Thread Simon Riggs
On Fri, 2010-09-03 at 09:55 +0300, Heikki Linnakangas wrote: On 03/09/10 09:36, Simon Riggs wrote: On Fri, 2010-09-03 at 12:50 +0900, Fujii Masao wrote: That design would affect what the standby should reply. If we choose async/recv/fsync/replay on a per-transaction basis, the standby

Re: [HACKERS] thousand unrelated data files in pg_default tablespace

2010-09-03 Thread Pavel Stehule
hello 2010/8/31 Tom Lane t...@sss.pgh.pa.us: Pavel Stehule pavel.steh...@gmail.com writes: there is a dump from 8KB files Well, those certainly look like tables/indexes not temp files. So we can rule out one theory. You're *certain* these aren't referenced from pg_class.relfilenode of any

[HACKERS] regclass without error?

2010-09-03 Thread Tatsuo Ishii
Hi, Is there any way to use regclass without having ERROR? pgpool-II needs to find the oid from table name and for the purpose it issues something like SELECT 'table_name'::regproc::oid. Problem is, if the table does not exist, an error occured and the transaction aborts. Ideally if the table

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-03 Thread Heikki Linnakangas
On 03/09/10 10:45, Simon Riggs wrote: On Fri, 2010-09-03 at 09:55 +0300, Heikki Linnakangas wrote: On 03/09/10 09:36, Simon Riggs wrote: On Fri, 2010-09-03 at 12:50 +0900, Fujii Masao wrote: That design would affect what the standby should reply. If we choose async/recv/fsync/replay on a

[HACKERS] plan time of MASSIVE partitioning ...

2010-09-03 Thread PostgreSQL - Hans-Jürgen Schönig
hello everybody, we came across an issue which turned out to be more serious than previously expected. imagine a system with, say, 1000 partitions (heavily indexed) or so. the time taken by the planner is already fairly heavy in this case. i tried this one with 5000 unindexed tables (just one

Re: [HACKERS] Synchronization levels in SR

2010-09-03 Thread Boszormenyi Zoltan
Hi, Dimitri Fontaine írta: Simon Riggs si...@2ndquadrant.com writes: Seems strange. If you have 2 standbys and you say you would like node1 to be the preferred candidate, then you load it so heavily that a remote server with by-definition much larger network delay responds first, then I

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-03 Thread Simon Riggs
On Fri, 2010-09-03 at 12:33 +0300, Heikki Linnakangas wrote: On 03/09/10 10:45, Simon Riggs wrote: On Fri, 2010-09-03 at 09:55 +0300, Heikki Linnakangas wrote: On 03/09/10 09:36, Simon Riggs wrote: On Fri, 2010-09-03 at 12:50 +0900, Fujii Masao wrote: That design would affect what the

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-03 Thread Heikki Linnakangas
On 03/09/10 13:20, Simon Riggs wrote: On Fri, 2010-09-03 at 12:33 +0300, Heikki Linnakangas wrote: On 03/09/10 10:45, Simon Riggs wrote: On Fri, 2010-09-03 at 09:55 +0300, Heikki Linnakangas wrote: On 03/09/10 09:36, Simon Riggs wrote: On Fri, 2010-09-03 at 12:50 +0900, Fujii Masao wrote:

Re: Interruptible sleeps (was Re: [HACKERS] CommitFest 2009-07: Yay, Kevin! Thanks, reviewers!)

2010-09-03 Thread Heikki Linnakangas
On 02/09/10 23:13, Tom Lane wrote: The WaitLatch ...timeout API could use a bit of refinement. I'd suggest defining negative timeout as meaning wait forever, so that timeout = 0 can be used for check but don't wait. Also, it seems like the function shouldn't just return void but should return

Re: [HACKERS] thousand unrelated data files in pg_default tablespace

2010-09-03 Thread Heikki Linnakangas
On 03/09/10 11:16, Pavel Stehule wrote: 2010/8/31 Tom Lanet...@sss.pgh.pa.us: Pavel Stehulepavel.steh...@gmail.com writes: there is a dump from 8KB files Well, those certainly look like tables/indexes not temp files. So we can rule out one theory. You're *certain* these aren't referenced

[HACKERS] Streaming a base backup from master

2010-09-03 Thread Heikki Linnakangas
It's been discussed before that it would be cool if you could stream a new base backup from the master server, via libpq. That way you would not need low-level filesystem access to initialize a new standby. Magnus mentioned today that he started hacking on that, and coincidentally I just

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Thom Brown
On 3 September 2010 12:19, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: TODO: * We need a smarter way to do pg_start/stop_backup() with this. At the moment, you can only have one backup running at a time, but we shouldn't have that limitation with this built-in mechanism.

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Dave Page
On Fri, Sep 3, 2010 at 12:19 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Here's a WIP patch. It adds a new TAKE_BACKUP command to the replication command set. Upon receiving that command, the master starts a COPY, and streams a tarred copy of the data directory to the

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Magnus Hagander
On Fri, Sep 3, 2010 at 13:19, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: It's been discussed before that it would be cool if you could stream a new base backup from the master server, via libpq. That way you would not need low-level filesystem access to initialize a new

Re: [HACKERS] Synchronization levels in SR

2010-09-03 Thread Fujii Masao
On Fri, Sep 3, 2010 at 6:43 PM, Boszormenyi Zoltan z...@cybertec.at wrote: In my patch, when the transactions were waiting for ack from the standby, they have already released all their locks, the wait happened at the latest possible point in CommitTransaction(). In Fujii's patch (I am

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Magnus Hagander
On Fri, Sep 3, 2010 at 13:25, Thom Brown t...@linux.com wrote: On 3 September 2010 12:19, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: TODO: * We need a smarter way to do pg_start/stop_backup() with this. At the moment, you can only have one backup running at a time, but we

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Thom Brown
On 3 September 2010 12:30, Magnus Hagander mag...@hagander.net wrote: On Fri, Sep 3, 2010 at 13:25, Thom Brown t...@linux.com wrote: On 3 September 2010 12:19, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: TODO: * We need a smarter way to do pg_start/stop_backup() with this.

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Heikki Linnakangas
On 03/09/10 14:25, Thom Brown wrote: On 3 September 2010 12:19, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: TODO: * We need a smarter way to do pg_start/stop_backup() with this. At the moment, you can only have one backup running at a time, but we shouldn't have that

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Heikki Linnakangas
On 03/09/10 14:28, Dave Page wrote: On Fri, Sep 3, 2010 at 12:19 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Here's a WIP patch. It adds a new TAKE_BACKUP command to the replication command set. Upon receiving that command, the master starts a COPY, and streams a tarred

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Magnus Hagander
On Fri, Sep 3, 2010 at 13:48, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 03/09/10 14:28, Dave Page wrote: On Fri, Sep 3, 2010 at 12:19 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com  wrote: Here's a WIP patch. It adds a new TAKE_BACKUP command to the

Re: [HACKERS] Cost estimates for parameterized paths

2010-09-03 Thread Robert Haas
On Thu, Sep 2, 2010 at 5:31 PM, Tom Lane t...@sss.pgh.pa.us wrote: Awhile back I ranted about replacing the planner's concept of inner indexscans with a more generalized notion of parameterized paths: http://archives.postgresql.org/pgsql-hackers/2009-10/msg00994.php The executor fixes for

Re: [HACKERS] plan time of MASSIVE partitioning ...

2010-09-03 Thread Stephen Frost
* PostgreSQL - Hans-Jürgen Schönig (postg...@cybertec.at) wrote: did anybody think of a solution to this problem. or more precisely: can there be a solution to this problem? Please post to the correct list (-performance) and provide information like PG version, postgresql.conf, the actual table

Re: [HACKERS] plan time of MASSIVE partitioning ...

2010-09-03 Thread PostgreSQL - Hans-Jürgen Schönig
On Sep 3, 2010, at 2:04 PM, Stephen Frost wrote: * PostgreSQL - Hans-Jürgen Schönig (postg...@cybertec.at) wrote: did anybody think of a solution to this problem. or more precisely: can there be a solution to this problem? Please post to the correct list (-performance) and provide

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Greg Stark
On Fri, Sep 3, 2010 at 12:19 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: * We need a smarter way to do pg_start/stop_backup() with this. At the moment, you can only have one backup running at a time, but we shouldn't have that limitation with this built-in mechanism. Well

Re: [HACKERS] plan time of MASSIVE partitioning ...

2010-09-03 Thread Stephen Frost
* PostgreSQL - Hans-Jürgen Schönig (postg...@cybertec.at) wrote: this seems like more a developer question to me than a pre performance one. it is not related to the table structure at all - it is basically an issue with incredibly large inheritance lists. it applies to postgres 9 and most

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Heikki Linnakangas
On 03/09/10 15:16, Greg Stark wrote: On Fri, Sep 3, 2010 at 12:19 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: * We need a smarter way to do pg_start/stop_backup() with this. At the moment, you can only have one backup running at a time, but we shouldn't have that

Re: [HACKERS] thousand unrelated data files in pg_default tablespace

2010-09-03 Thread Pavel Stehule
2010/9/3 Heikki Linnakangas heikki.linnakan...@enterprisedb.com: On 03/09/10 11:16, Pavel Stehule wrote: 2010/8/31 Tom Lanet...@sss.pgh.pa.us: Pavel Stehulepavel.steh...@gmail.com  writes: there is a dump from 8KB files Well, those certainly look like tables/indexes not temp files. So we

Re: [HACKERS] Synchronization levels in SR

2010-09-03 Thread Boszormenyi Zoltan
Fujii Masao írta: On Fri, Sep 3, 2010 at 6:43 PM, Boszormenyi Zoltan z...@cybertec.at wrote: In my patch, when the transactions were waiting for ack from the standby, they have already released all their locks, the wait happened at the latest possible point in CommitTransaction(). In

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Robert Haas
On Fri, Sep 3, 2010 at 7:28 AM, Dave Page dp...@pgadmin.org wrote: On Fri, Sep 3, 2010 at 12:19 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Here's a WIP patch. It adds a new TAKE_BACKUP command to the replication command set. Upon receiving that command, the master starts

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Magnus Hagander
On Fri, Sep 3, 2010 at 15:24, Robert Haas robertmh...@gmail.com wrote: On Fri, Sep 3, 2010 at 7:28 AM, Dave Page dp...@pgadmin.org wrote: On Fri, Sep 3, 2010 at 12:19 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Here's a WIP patch. It adds a new TAKE_BACKUP command to the

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Dave Page
On Fri, Sep 3, 2010 at 2:24 PM, Robert Haas robertmh...@gmail.com wrote: On Fri, Sep 3, 2010 at 7:28 AM, Dave Page dp...@pgadmin.org wrote: On Fri, Sep 3, 2010 at 12:19 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Here's a WIP patch. It adds a new TAKE_BACKUP command to

Re: [HACKERS] plan time of MASSIVE partitioning ...

2010-09-03 Thread Robert Haas
2010/9/3 PostgreSQL - Hans-Jürgen Schönig postg...@cybertec.at: i tried this one with 5000 unindexed tables (just one col): test=# \timing Timing is on. test=# prepare x(int4) AS select * from t_data order by id desc; PREPARE Time: 361.552 ms you will see similar or higher runtimes in

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Robert Haas
On Fri, Sep 3, 2010 at 9:26 AM, Dave Page dp...@pgadmin.org wrote: rsync? Might be easier to use that from day 1 (well, day 2) than to retrofit later. I'm not sure we want to depend on an external utility like that, particularly one that users may not have installed. And I'm not sure if that

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Dave Page
On Fri, Sep 3, 2010 at 2:29 PM, Robert Haas robertmh...@gmail.com wrote: On Fri, Sep 3, 2010 at 9:26 AM, Dave Page dp...@pgadmin.org wrote: rsync? Might be easier to use that from day 1 (well, day 2) than to retrofit later. I'm not sure we want to depend on an external utility like that,

Re: [HACKERS] regclass without error?

2010-09-03 Thread Robert Haas
On Fri, Sep 3, 2010 at 4:28 AM, Tatsuo Ishii is...@postgresql.org wrote: Is there any way to use regclass without having ERROR? pgpool-II needs to find the oid from table name and for the purpose it issues something like SELECT 'table_name'::regproc::oid. Problem is, if the table does not

Re: [HACKERS] Path question

2010-09-03 Thread Hans-Jürgen Schönig
On Sep 2, 2010, at 1:20 AM, Robert Haas wrote: On Sep 1, 2010, at 10:21 AM, Greg Stark gsst...@mit.edu wrote: For what it's worth I disagree with Tom. I think this is a situation where we need *both* types of solution. Ideally we will be able to use a plain Append node for cases where we

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Robert Haas
On Fri, Sep 3, 2010 at 9:32 AM, Dave Page dp...@pgadmin.org wrote: No, I agree we don't want an external dependency (I was just bleating about needing tar on Windows). I was assuming/hoping there's a librsync somewhere... The rsync code itself is not modular, I believe. I think the author

Re: [HACKERS] english parser in text search: support for multiple words in the same position

2010-09-03 Thread Robert Haas
On Wed, Sep 1, 2010 at 2:42 AM, Sushant Sinha sushant...@gmail.com wrote: I have attached a patch that emits parts of a host token, a url token, an email token and a file token. Further, it makes sure that a host/url/email/file token and the first part-token are at the same position in

Re: [HACKERS] regclass without error?

2010-09-03 Thread Tatsuo Ishii
You can write a query against the system catalog tables. That was pretty hard than I though(schema search path etc.). Or you could install a function that wraps the regclass cast in an exception handler. That requires users to install the function. Annoying for users. -- Tatsuo Ishii SRA

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: The rsync code itself is not modular, I believe. I think the author thereof kind of took the approach of placing efficiency before all. Yeah, I looked into this when discussing this same concept at PGCon with folks. There doesn't appear to be a

Re: Interruptible sleeps (was Re: [HACKERS] CommitFest 2009-07: Yay, Kevin! Thanks, reviewers!)

2010-09-03 Thread Tom Lane
Fujii Masao masao.fu...@gmail.com writes: On Fri, Sep 3, 2010 at 11:08 AM, Tom Lane t...@sss.pgh.pa.us wrote: elog(FATAL) is *certainly* not a better idea.  I think there's really nothing that can be done, you just have to silently ignore the error. Hmm.. some functions called by a signal

Re: [HACKERS] regclass without error?

2010-09-03 Thread Marko Tiikkaja
On 2010-09-03 4:46 PM, Tatsuo Ishii wrote: You can write a query against the system catalog tables. That was pretty hard than I though(schema search path etc.). Or you could install a function that wraps the regclass cast in an exception handler. That requires users to install the

Re: [HACKERS] regclass without error?

2010-09-03 Thread David Fetter
On Fri, Sep 03, 2010 at 05:10:44PM +0300, Marko Tiikkaja wrote: On 2010-09-03 4:46 PM, Tatsuo Ishii wrote: You can write a query against the system catalog tables. That was pretty hard than I though(schema search path etc.). Or you could install a function that wraps the regclass cast in

Re: [HACKERS] regclass without error?

2010-09-03 Thread Tom Lane
Tatsuo Ishii is...@postgresql.org writes: Is there any way to use regclass without having ERROR? pgpool-II needs to find the oid from table name and for the purpose it issues something like SELECT 'table_name'::regproc::oid. Problem is, if the table does not exist, an error occured and the

Re: [HACKERS] regclass without error?

2010-09-03 Thread Pavel Golub
Hello, guys. You wrote: TI Hi, TI Is there any way to use regclass without having ERROR? TI pgpool-II needs to find the oid from table name and for the purpose it TI issues something like SELECT 'table_name'::regproc::oid. Problem is, TI if the table does not exist, an error occured and the

Re: [HACKERS] plan time of MASSIVE partitioning ...

2010-09-03 Thread Tom Lane
=?iso-8859-1?Q?PostgreSQL_-_Hans-J=FCrgen_Sch=F6nig?= postg...@cybertec.at writes: imagine a system with, say, 1000 partitions (heavily indexed) or so. the time taken by the planner is already fairly heavy in this case. As the fine manual points out, the current scheme for managing

Re: [HACKERS] plan time of MASSIVE partitioning ...

2010-09-03 Thread PostgreSQL - Hans-Jürgen Schönig
On Sep 3, 2010, at 4:40 PM, Tom Lane wrote: =?iso-8859-1?Q?PostgreSQL_-_Hans-J=FCrgen_Sch=F6nig?= postg...@cybertec.at writes: imagine a system with, say, 1000 partitions (heavily indexed) or so. the time taken by the planner is already fairly heavy in this case. As the fine manual

Re: Interruptible sleeps (was Re: [HACKERS] CommitFest 2009-07: Yay, Kevin! Thanks, reviewers!)

2010-09-03 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: On 02/09/10 23:13, Tom Lane wrote: (Yeah, I realize the caller could look into the latch to find that out, but callers really ought to treat latches as opaque structs.) Hmm, maybe we need a TestLatch function to check if a latch

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: On 03/09/10 15:16, Greg Stark wrote: On Fri, Sep 3, 2010 at 12:19 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: * We need a smarter way to do pg_start/stop_backup() with this. At the moment, you can only have

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Kevin Grittner
Stephen Frost sfr...@snowman.net wrote: there's a heck of alot of complexity there that we *don't* need. rsync is a great tool, don't get me wrong, but let's not try to go over our heads here. Right -- among other things, it checks for portions of a new file which match the old file at a

[HACKERS] Windows Tools

2010-09-03 Thread David Fetter
Hello, At work, I've been tasked with providing some Windows connection libraries, etc. Are people still using MinGW, or should I just recommend using a proprietary toolkit to do the builds? If so, which one(s)? Cheers, David (Windows n00b) -- David Fetter da...@fetter.org http://fetter.org/

Re: [HACKERS] Windows Tools

2010-09-03 Thread Dave Page
On Fri, Sep 3, 2010 at 4:11 PM, David Fetter da...@fetter.org wrote: Hello, At work, I've been tasked with providing some Windows connection libraries, etc.  Are people still using MinGW, or should I just recommend using a proprietary toolkit to do the builds?  If so, which one(s)? Visual

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Stephen Frost
Kevin, * Kevin Grittner (kevin.gritt...@wicourts.gov) wrote: While 1GB granularity would be OK, I doubt it's optimal; I think CRC checks for smaller chunks might be worthwhile. My gut feel is that somewhere in the 64kB to 1MB range would probably be optimal for us, although the sweet spot

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Thom Brown
On 3 September 2010 16:01, Tom Lane t...@sss.pgh.pa.us wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: On 03/09/10 15:16, Greg Stark wrote: On Fri, Sep 3, 2010 at 12:19 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com  wrote: * We need a smarter way to do

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Kevin Grittner
Stephen Frost sfr...@snowman.net wrote: We have something much better, called WAL. If people want to keep their backup current, they should use that after getting the base backup up and working. Unless you want to provide support for Point In Time Recovery without excessive recovery times.

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Heikki Linnakangas
On 03/09/10 18:01, Tom Lane wrote: Heikki Linnakangasheikki.linnakan...@enterprisedb.com writes: On 03/09/10 15:16, Greg Stark wrote: On Fri, Sep 3, 2010 at 12:19 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: * We need a smarter way to do pg_start/stop_backup() with

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Robert Haas
On Fri, Sep 3, 2010 at 11:20 AM, Stephen Frost sfr...@snowman.net wrote: Kevin, * Kevin Grittner (kevin.gritt...@wicourts.gov) wrote: While 1GB granularity would be OK, I doubt it's optimal; I think CRC checks for smaller chunks might be worthwhile.  My gut feel is that somewhere in the 64kB

Re: Interruptible sleeps (was Re: [HACKERS] CommitFest 2009-07: Yay, Kevin! Thanks, reviewers!)

2010-09-03 Thread Robert Haas
On Fri, Sep 3, 2010 at 10:07 AM, Tom Lane t...@sss.pgh.pa.us wrote: Fujii Masao masao.fu...@gmail.com writes: On Fri, Sep 3, 2010 at 11:08 AM, Tom Lane t...@sss.pgh.pa.us wrote: elog(FATAL) is *certainly* not a better idea.  I think there's really nothing that can be done, you just have to

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Tom Lane
Kevin Grittner kevin.gritt...@wicourts.gov writes: Stephen Frost sfr...@snowman.net wrote: In any case, it's certainly not something required for an initial implementation.. No disagreement there; but sometimes it pays to know where you might want to go, so you don't do something to make

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Robert Haas
On Fri, Sep 3, 2010 at 11:47 AM, Tom Lane t...@sss.pgh.pa.us wrote: Kevin Grittner kevin.gritt...@wicourts.gov writes: Stephen Frost sfr...@snowman.net wrote: In any case, it's certainly not something required for an initial implementation.. No disagreement there; but sometimes it pays to

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread David Blewett
On Fri, Sep 3, 2010 at 11:47 AM, Tom Lane t...@sss.pgh.pa.us wrote: IOW, what I'd like to see is protocol extensions that allow an external copy of rsync to be invoked; not build in rsync, or tar, or anything else that we could get off-the-shelf. Personally, I would love to see protocol-level

Re: [HACKERS] regclass without error?

2010-09-03 Thread David E. Wheeler
On Sep 3, 2010, at 7:31 AM, Tom Lane wrote: I don't think the cast should act that way, but I could see providing a separate conversion function that returns 0 ... or perhaps better NULL ... if no match. +1 I could use this in pgTAP. David -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: IOW, what I'd like to see is protocol extensions that allow an external copy of rsync to be invoked; not build in rsync, or tar, or anything else that we could get off-the-shelf. I'd much rather use an existing library to implement it than call out to

Re: Interruptible sleeps (was Re: [HACKERS] CommitFest 2009-07: Yay, Kevin! Thanks, reviewers!)

2010-09-03 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Fri, Sep 3, 2010 at 10:07 AM, Tom Lane t...@sss.pgh.pa.us wrote: [ shrug... ]  I stated before that the Hot Standby patch is doing utterly unsafe things in signal handlers.  Simon rejected that. I am waiting for irrefutable evidence to emerge from

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: what I'd like to see is protocol extensions that allow an external copy of rsync to be invoked; not build in rsync, or tar, or anything else that we could get off-the-shelf. The complexities of dealing with properly invoking rsync externally could well

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Heikki Linnakangas
On 03/09/10 19:09, Stephen Frost wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: IOW, what I'd like to see is protocol extensions that allow an external copy of rsync to be invoked; not build in rsync, or tar, or anything else that we could get off-the-shelf. I'd much rather use an existing

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Heikki Linnakangas
On 03/09/10 18:53, David Blewett wrote: On Fri, Sep 3, 2010 at 11:47 AM, Tom Lanet...@sss.pgh.pa.us wrote: IOW, what I'd like to see is protocol extensions that allow an external copy of rsync to be invoked; not build in rsync, or tar, or anything else that we could get off-the-shelf.

Re: [HACKERS] Windows Tools

2010-09-03 Thread Magnus Hagander
On Fri, Sep 3, 2010 at 17:18, Dave Page dp...@pgadmin.org wrote: On Fri, Sep 3, 2010 at 4:11 PM, David Fetter da...@fetter.org wrote: Hello, At work, I've been tasked with providing some Windows connection libraries, etc.  Are people still using MinGW, or should I just recommend using a

Re: Interruptible sleeps (was Re: [HACKERS] CommitFest 2009-07: Yay, Kevin! Thanks, reviewers!)

2010-09-03 Thread Robert Haas
On Fri, Sep 3, 2010 at 12:10 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Fri, Sep 3, 2010 at 10:07 AM, Tom Lane t...@sss.pgh.pa.us wrote: [ shrug... ]  I stated before that the Hot Standby patch is doing utterly unsafe things in signal handlers.  Simon

Re: [HACKERS] regclass without error?

2010-09-03 Thread Alvaro Herrera
Excerpts from David Fetter's message of vie sep 03 10:21:00 -0400 2010: How about using a SAVEPOINT before the cast? Wouldn't fail your transaction.. For unattended operation, there are some issues: * Generating appropriate SAVEPOINT names * Keeping track of same * Detecting errors

Re: [HACKERS] regclass without error?

2010-09-03 Thread David Fetter
On Fri, Sep 03, 2010 at 01:28:15PM -0400, Alvaro Herrera wrote: Excerpts from David Fetter's message of vie sep 03 10:21:00 -0400 2010: How about using a SAVEPOINT before the cast? Wouldn't fail your transaction.. For unattended operation, there are some issues: * Generating

Re: [HACKERS] regclass without error?

2010-09-03 Thread Alvaro Herrera
Excerpts from David Fetter's message of vie sep 03 13:31:00 -0400 2010: We have no mechanism to do this, or any other check, during a bulk load. It'd be a great feature to have :) I'm not sure what kind of bulk load you are talking about, nor what does it have to do with pgpool-II. --

Re: Interruptible sleeps (was Re: [HACKERS] CommitFest 2009-07: Yay, Kevin! Thanks, reviewers!)

2010-09-03 Thread Heikki Linnakangas
On 03/09/10 17:51, Tom Lane wrote: Heikki Linnakangasheikki.linnakan...@enterprisedb.com writes: On 02/09/10 23:13, Tom Lane wrote: Also, using sig_atomic_t for owner_pid is entirely not sane. Hmm, true, it doesn't need to be set from signal handler, but is there an atomicity problem if

Re: [HACKERS] Cost estimates for parameterized paths

2010-09-03 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Thu, Sep 2, 2010 at 5:31 PM, Tom Lane t...@sss.pgh.pa.us wrote: The best idea I can come up with at the moment is to compute best case and worst case costs for a parameterized path, Interestingly, I previously proposed almost exactly this approach

Re: Interruptible sleeps (was Re: [HACKERS] CommitFest 2009-07: Yay, Kevin! Thanks, reviewers!)

2010-09-03 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: On 03/09/10 17:51, Tom Lane wrote: If there is *any* possibility of that happening then you have far worse problems than whether the field is atomically readable or not: the behavior will be unpredictable at just slightly larger

[HACKERS] ps buffer is incorrectly padded on the (latest) OS X

2010-09-03 Thread Alexey Klyukin
Hi, I always wondered why ps ax|grep postgres shows several extra blank lines after the process name, i.e. 972 ?? Ss 0:00.69 postgres: writer process 973 ?? Ss 0:00.51 postgres: wal writer process (I put newlines instead of spaces there). By looking into the code I've

Re: [HACKERS] Windows Tools

2010-09-03 Thread David Fetter
On Fri, Sep 03, 2010 at 06:24:06PM +0200, Magnus Hagander wrote: On Fri, Sep 3, 2010 at 17:18, Dave Page dp...@pgadmin.org wrote: On Fri, Sep 3, 2010 at 4:11 PM, David Fetter da...@fetter.org wrote: Hello, At work, I've been tasked with providing some Windows connection libraries, etc.  

Re: Interruptible sleeps (was Re: [HACKERS] CommitFest 2009-07: Yay, Kevin! Thanks, reviewers!)

2010-09-03 Thread Heikki Linnakangas
On 03/09/10 21:16, Tom Lane wrote: Heikki Linnakangasheikki.linnakan...@enterprisedb.com writes: WaitLatch had to set the pid on the Latch struct to allow other processes to send the signal. Another process could call SetLatch and read the pid field, while WaitLatch is just setting it. I think

Re: Interruptible sleeps (was Re: [HACKERS] CommitFest 2009-07: Yay, Kevin! Thanks, reviewers!)

2010-09-03 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: On 03/09/10 21:16, Tom Lane wrote: It's probably not too unreasonable to assume that pid_t assignment is atomic. But I'm still thinking that we have bigger problems than that if there are really cases where SetLatch can execute at

Re: [HACKERS] ps buffer is incorrectly padded on the (latest) OS X

2010-09-03 Thread Tom Lane
Alexey Klyukin al...@commandprompt.com writes: I always wondered why ps ax|grep postgres shows several extra blank lines after the process name, i.e. AFAIR it's always done that on OSX. I thought we'd tried the '\0' padding way back when and it didn't work nicely, but maybe Apple fixed that.

Re: Interruptible sleeps (was Re: [HACKERS] CommitFest 2009-07: Yay, Kevin! Thanks, reviewers!)

2010-09-03 Thread Ron Mayer
Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Fri, Sep 3, 2010 at 10:07 AM, Tom Lane t...@sss.pgh.pa.us wrote: [ shrug... ] I stated before that the Hot Standby patch is doing utterly unsafe things in signal handlers. Simon rejected that. I am waiting for irrefutable evidence

Re: [HACKERS] returning multiple result sets from a stored procedure

2010-09-03 Thread Merlin Moncure
On Thu, Sep 2, 2010 at 7:17 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from John Adams's message of jue sep 02 18:25:45 -0400 2010: I noticed in postgres you cannot return multiple result sets from a stored procedure (surprisingly as it looks like a very good dbms). If

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Martijn van Oosterhout
On Fri, Sep 03, 2010 at 09:56:12AM -0400, Stephen Frost wrote: * Robert Haas (robertmh...@gmail.com) wrote: The rsync code itself is not modular, I believe. I think the author thereof kind of took the approach of placing efficiency before all. Yeah, I looked into this when discussing this

[HACKERS] The other major HS TODO: standby promotion

2010-09-03 Thread Josh Berkus
All, As far as I'm concerned, the other big missing feature for HS is the ability to promote standbys to become the new master. If we had that feature, then HS can be the backbone of a large-scale PostgreSQL cloud; if we don't have it, then HS does not contribute very much to scalability beyond

Re: [HACKERS] returning multiple result sets from a stored procedure

2010-09-03 Thread Josh Berkus
I noticed in postgres you cannot return multiple result sets from a stored procedure (surprisingly as it looks like a very good dbms). That feature has been on the TODO list for years. However, nobody has stepped forward to either write it, or to fund working on it. If your company has

Re: Interruptible sleeps (was Re: [HACKERS] CommitFest 2009-07: Yay, Kevin! Thanks, reviewers!)

2010-09-03 Thread Robert Haas
On Fri, Sep 3, 2010 at 3:11 PM, Ron Mayer rm...@cheapcomplexdevices.com wrote: Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Fri, Sep 3, 2010 at 10:07 AM, Tom Lane t...@sss.pgh.pa.us wrote: [ shrug... ]  I stated before that the Hot Standby patch is doing utterly unsafe things

Re: [HACKERS] returning multiple result sets from a stored procedure

2010-09-03 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: I noticed in postgres you cannot return multiple result sets from a stored procedure (surprisingly as it looks like a very good dbms). That feature has been on the TODO list for years. However, nobody has stepped forward to either write it, or to fund

Re: Interruptible sleeps (was Re: [HACKERS] CommitFest 2009-07: Yay, Kevin! Thanks, reviewers!)

2010-09-03 Thread Heikki Linnakangas
On 03/09/10 19:38, Robert Haas wrote: On Fri, Sep 3, 2010 at 12:10 PM, Tom Lanet...@sss.pgh.pa.us wrote: Robert Haasrobertmh...@gmail.com writes: On Fri, Sep 3, 2010 at 10:07 AM, Tom Lanet...@sss.pgh.pa.us wrote: [ shrug... ] I stated before that the Hot Standby patch is doing utterly

Re: [HACKERS] Cost estimates for parameterized paths

2010-09-03 Thread Robert Haas
On Fri, Sep 3, 2010 at 2:04 PM, Tom Lane t...@sss.pgh.pa.us wrote: On reflection I think that for parameterized paths the problem won't be too bad, because (a) we'll ignore parameterized paths except when considering a join to the right outer rel, so their presence in the rel's pathlist won't

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread David Blewett
On Fri, Sep 3, 2010 at 12:23 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 03/09/10 18:53, David Blewett wrote: On Fri, Sep 3, 2010 at 11:47 AM, Tom Lanet...@sss.pgh.pa.us  wrote: IOW, what I'd like to see is protocol extensions that allow an external copy of rsync to

Re: OT: OFF TOPIC: [HACKERS] returning multiple result sets from a stored procedure

2010-09-03 Thread Josh Berkus
John, I honestly do not mean any offence, just out of curiosity. If you guys care about money and time why would you spend the best years of your life basically copying commercial products for free? We don't do it to copy commercial products. We do it to build something better than them.

Re: OT: OFF TOPIC: [HACKERS] returning multiple result sets from a stored procedure

2010-09-03 Thread Josh Berkus
On 9/3/10 2:20 PM, Josh Berkus wrote: However, if you look at any mature, large open source project you will find that at least 1/4 of its code contributors are paid to work on the project by *someone*, and that those paid developers account for 70% to 95% of the code. Relevant link for this:

Re: Interruptible sleeps (was Re: [HACKERS] CommitFest 2009-07: Yay, Kevin! Thanks, reviewers!)

2010-09-03 Thread Robert Haas
On Fri, Sep 3, 2010 at 4:20 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Maybe that's ok, if I'm reading the deadlock checker code correctly, it also calls semop() to increment the another process' semaphore, and the deadlock checker can be invoked from a signal handler

Re: Interruptible sleeps (was Re: [HACKERS] CommitFest 2009-07: Yay, Kevin! Thanks, reviewers!)

2010-09-03 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: A safer approach would be to just PGSemaphoreUnlock() in the signal handler, and do all the other processing outside it. I don't see any particularly good reason to assume that PGSemaphoreUnlock is safe either: you're still talking

Re: [HACKERS] Cost estimates for parameterized paths

2010-09-03 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Fri, Sep 3, 2010 at 2:04 PM, Tom Lane t...@sss.pgh.pa.us wrote: On reflection I think that for parameterized paths the problem won't be too bad, because (a) we'll ignore parameterized paths except when considering a join to the right outer rel, so

Re: Interruptible sleeps (was Re: [HACKERS] CommitFest 2009-07: Yay, Kevin! Thanks, reviewers!)

2010-09-03 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: Color me confused; I may need to backpedal rapidly here. I had thought that what Tom was complaining about was the fact that the signal handler was taking LWLocks, which I would have thought to be totally unsafe. Well, it's unsafe if the signal could

  1   2   >