Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-19 Thread Kohei KaiGai
2015-08-19 20:12 GMT+09:00 Simon Riggs si...@2ndquadrant.com: On 12 June 2015 at 00:29, Tomas Vondra tomas.von...@2ndquadrant.com wrote: I see two ways to fix this: (1) enforce the 1GB limit (probably better for back-patching, if that's necessary) (2) make it work with hash tables

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread Oleg Bartunov
On Wed, Aug 19, 2015 at 4:46 PM, Andres Freund and...@anarazel.de wrote: On 2015-08-19 09:41:32 -0400, Tom Lane wrote: In fact, they'd still need to use DNS balancing for Postgres, because not everything connects with libpq (think JDBC for instance). It already does support this though.

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-19 Thread Simon Riggs
On 19 August 2015 at 12:55, Kohei KaiGai kai...@kaigai.gr.jp wrote: 2015-08-19 20:12 GMT+09:00 Simon Riggs si...@2ndquadrant.com: On 12 June 2015 at 00:29, Tomas Vondra tomas.von...@2ndquadrant.com wrote: I see two ways to fix this: (1) enforce the 1GB limit (probably better for

Re: [HACKERS] allowing wal_level change at run time

2015-08-19 Thread Andres Freund
On 2015-08-19 10:49:46 +0200, Magnus Hagander wrote: What happens the first time? Meaning I'm on wal_level=minimal and take a base backup. Then when the replica first connects 10 minutes later, it needs WAL back in time, which was logged at wal_level=minimal. So you'd need to bump it up

Re: [HACKERS] allowing wal_level change at run time

2015-08-19 Thread Andres Freund
On 2015-08-18 21:47:51 -0400, Peter Eisentraut wrote: On 8/18/15 1:46 PM, Andres Freund wrote: I don't think not requiring restarts is sufficient, having to twiddle a bunch of parameters manually still is a lot more effort than people see as necessary. I agree that we want both. But

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-19 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On 19 August 2015 at 12:55, Kohei KaiGai kai...@kaigai.gr.jp wrote: Please don't be rush. :-) Please explain what rush you see? Yours. You appear to be in a hurry to apply patches that there's no consensus on. It is not difficult to replace

Re: [HACKERS] postgres_fdw join pushdown (was Re: Custom/Foreign-Join-APIs)

2015-08-19 Thread Ashutosh Bapat
On Tue, Aug 4, 2015 at 2:20 PM, Shigeru Hanada shigeru.han...@gmail.com wrote: Hi Ashutosh, Sorry for leaving the thread. 2015-07-20 16:09 GMT+09:00 Ashutosh Bapat ashutosh.ba...@enterprisedb.com : In find_user_mapping(), if the first cache search returns a valid tuple, it is checked

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread Victor Wagner
On 2015.08.19 at 09:21:50 +, Albe Laurenz wrote: Yes, but that will only work reliably if the (read-only) standby does not allow connections before it is promoted. It would just take a bit more time for client and a bit more load for server - to make sure that this connection is

Re: [HACKERS] Declarative partitioning

2015-08-19 Thread David Fetter
On Wed, Aug 19, 2015 at 04:30:39PM +0900, Amit Langote wrote: On 2015-08-18 PM 10:43, David Fetter wrote: After the first command is done, the second command would take exclusive lock on table_name, scan the table to check if it contains any values outside the boundaries defined by FOR

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread Andres Freund
On 2015-08-19 09:41:32 -0400, Tom Lane wrote: In fact, they'd still need to use DNS balancing for Postgres, because not everything connects with libpq (think JDBC for instance). It already does support this though. https://jdbc.postgresql.org/documentation/head/connect.html : Connection

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-19 Thread Simon Riggs
On 12 June 2015 at 00:29, Tomas Vondra tomas.von...@2ndquadrant.com wrote: I see two ways to fix this: (1) enforce the 1GB limit (probably better for back-patching, if that's necessary) (2) make it work with hash tables over 1GB I'm in favor of (2) if there's a good way to do that.

Re: [HACKERS] More WITH

2015-08-19 Thread David Fetter
On Tue, Aug 18, 2015 at 11:23:32PM -0400, Tom Lane wrote: Craig Ringer cr...@2ndquadrant.com writes: On 18 August 2015 at 01:18, David Fetter da...@fetter.org wrote: FETCH [in WITH] I'd be a huge fan of this one. I'd love to see FETCH in subqueries, too. Currently doing anything like

Re: [HACKERS] [patch] psql tab completion for grant execute

2015-08-19 Thread Daniel Verite
Robert Haas wrote: A trivial patch is attached. It adds the condition that if EXECUTE is preceded by GRANT itself preceded by nothing, then that completion with PROCEDURE is skipped. Thanks, I committed this. I don't think we usually back-patch tab completion fixes, but I

Re: [HACKERS] allowing wal_level change at run time

2015-08-19 Thread Simon Riggs
On 18 August 2015 at 18:46, Andres Freund and...@anarazel.de wrote: ISTM that it's not too hard to a) make archive_mode PGC_SIGHUP b) make wal_level PGC_SIGHUP +1 c) automatically increase wal_level to logical whenever a logical replication slot is defined -1 It would be easier to

Re: [HACKERS] Declarative partitioning

2015-08-19 Thread Simon Riggs
On 18 August 2015 at 18:31, Josh Berkus j...@agliodbs.com wrote: 2. Creating a partition of a partitioned table CREATE TABLE table_name PARTITION OF partitioned_table_name FOR VALUES values_spec; Where values_spec is: listvalues: [IN] (val1, ...) rangevalues: START

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-19 Thread Kohei KaiGai
2015-08-19 21:29 GMT+09:00 Simon Riggs si...@2ndquadrant.com: On 19 August 2015 at 12:55, Kohei KaiGai kai...@kaigai.gr.jp wrote: 2015-08-19 20:12 GMT+09:00 Simon Riggs si...@2ndquadrant.com: On 12 June 2015 at 00:29, Tomas Vondra tomas.von...@2ndquadrant.com wrote: I see two ways to

Re: [HACKERS] Make HeapTupleSatisfiesMVCC more concurrent

2015-08-19 Thread Andres Freund
On 2015-08-18 20:36:13 -0400, Tom Lane wrote: I wrote: Just thinking about this ... I wonder why we need to call TransactionIdIsInProgress() at all rather than believing the answer from the snapshot? Under what circumstances could TransactionIdIsInProgress() return true where

Re: [HACKERS] Declarative partitioning

2015-08-19 Thread Simon Riggs
On 18 August 2015 at 11:30, Amit Langote langote_amit...@lab.ntt.co.jp wrote: There is no need to define tuple routing triggers. CopyFrom() and ExecInsert() determine target partition just before performing heap_insert() and ExecInsertIndexTuples(). IOW, any BR triggers and constraints (on

Re: [HACKERS] Bug? ExecChooseHashTableSize() got assertion failed with crazy number of rows

2015-08-19 Thread Kevin Grittner
Kouhei Kaigai kai...@ak.jp.nec.com wrote: we may need a couple of overhaul around HashJoin to support large size of data, not only nbuckets around 0x8000. Perhaps, but this is a clear bug, introduced to the 9.5 code, with an obvious fix; so I've pushed the change from 1 to 1L on that left

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread Tom Lane
Albe Laurenz laurenz.a...@wien.gv.at writes: Victor Wagner wrote: It would just take a bit more time for client and a bit more load for server - to make sure that this connection is read-write by issuing show transaction_read_only statement before considering connection useful. That's not

Re: [HACKERS] Make HeapTupleSatisfiesMVCC more concurrent

2015-08-19 Thread Tom Lane
Andres Freund and...@anarazel.de writes: On 2015-08-18 20:36:13 -0400, Tom Lane wrote: I experimented with the attached patch, which replaces HeapTupleSatisfiesMVCC's calls of TransactionIdIsInProgress with XidInMVCCSnapshot, and then as a cross-check has all the return false exits from

Re: [HACKERS] allowing wal_level change at run time

2015-08-19 Thread Andres Freund
On 2015-08-19 17:51:47 +0200, Magnus Hagander wrote: That's doable - but what about manual base backups? And if they don't go away, what about the ones that are generated by the nightly/weekly/hourly/whatever pg_basebackup -x ones? Good questions. I guess we could just make

Re: [HACKERS] Make HeapTupleSatisfiesMVCC more concurrent

2015-08-19 Thread Tom Lane
I wrote: Andres Freund and...@anarazel.de writes: I'm not sure about it, but it might be worthwhile to add a TransactionIdIsKnownCompleted() check before the more expensive parts of XidInMVCCSnapshot(). Neither the array search nor, much more so, the subtrans lookups are free. Hmmm... the

[HACKERS] how to write/setup a C trigger function in a background worker

2015-08-19 Thread jacques klein
I would like to execute a trigger function (written in C) in one of my background workers. Didn't figure out how to do that not even if it's possible. Currently my trigger function is called, but in the server-process connected to the client who does the insert into ... . Any hint/help is

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread Simon Riggs
On 19 August 2015 at 14:46, Andres Freund and...@anarazel.de wrote: On 2015-08-19 09:41:32 -0400, Tom Lane wrote: In fact, they'd still need to use DNS balancing for Postgres, because not everything connects with libpq (think JDBC for instance). It already does support this though.

Re: [HACKERS] Bug? ExecChooseHashTableSize() got assertion failed with crazy number of rows

2015-08-19 Thread Tom Lane
Kevin Grittner kgri...@ymail.com writes: Kouhei Kaigai kai...@ak.jp.nec.com wrote: we may need a couple of overhaul around HashJoin to support large size of data, not only nbuckets around 0x8000. Perhaps, but this is a clear bug, introduced to the 9.5 code, with an obvious fix; so I've

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread ''Victor Wagner *EXTERN*' *EXTERN*' *EXTERN*
On 2015.08.19 at 15:35:17 +0100, Simon Riggs wrote: I think we do need some way of saying that a readonly connection is OK. So I had such thing in my propsal (boolean parameter readonly). But haven't yet checked if it is compatible with jdbc syntax. the default would be to connect to each

Re: [HACKERS] Bug? ExecChooseHashTableSize() got assertion failed with crazy number of rows

2015-08-19 Thread Tom Lane
I wrote: I don't think it's anywhere near as clear as you think. Ah, scratch that --- I was looking at the wrong my_log2() call. -ENOCAFFEINE. I'm still doubtful that this is the only overflow risk in that new ExecChooseHashTableSize code, though. For instance, the only reason the line

Re: [HACKERS] Make HeapTupleSatisfiesMVCC more concurrent

2015-08-19 Thread Simon Riggs
On 19 August 2015 at 16:21, Tom Lane t...@sss.pgh.pa.us wrote: I wrote: Andres Freund and...@anarazel.de writes: I'm not sure about it, but it might be worthwhile to add a TransactionIdIsKnownCompleted() check before the more expensive parts of XidInMVCCSnapshot(). Neither the array

Re: [HACKERS] allowing wal_level change at run time

2015-08-19 Thread Magnus Hagander
On Wed, Aug 19, 2015 at 3:34 PM, Andres Freund and...@anarazel.de wrote: On 2015-08-19 10:49:46 +0200, Magnus Hagander wrote: What happens the first time? Meaning I'm on wal_level=minimal and take a base backup. Then when the replica first connects 10 minutes later, it needs WAL back in

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-19 Thread Simon Riggs
On 19 August 2015 at 14:53, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: On 19 August 2015 at 12:55, Kohei KaiGai kai...@kaigai.gr.jp wrote: Please don't be rush. :-) Please explain what rush you see? Yours. You appear to be in a hurry to apply patches

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread Tatsuo Ishii
Here we are discussing load-balancing on the client level, not on the statement level. I see. Suppose that we have 100 readonly clients and 3 standby servers + master. If all clients specify all four servers in the their connect strings, and connect randomly to them, each server would have

Re: [HACKERS] Bug? ExecChooseHashTableSize() got assertion failed with crazy number of rows

2015-08-19 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: I'm still doubtful that this is the only overflow risk in that new ExecChooseHashTableSize code, though. KaiGai already pointed that out on this thread and I completely agree; but I figured that I might as well fix the clear bug with an obvious fix that was

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread Shulgin, Oleksandr
On Wed, Aug 19, 2015 at 4:45 PM, ''Victor Wagner *EXTERN*' *EXTERN*' *EXTERN* vi...@wagner.pp.ru wrote: On 2015.08.19 at 15:35:17 +0100, Simon Riggs wrote: I think we do need some way of saying that a readonly connection is OK. So I had such thing in my propsal (boolean parameter

Re: [HACKERS] Reduce ProcArrayLock contention

2015-08-19 Thread Andres Freund
Hi, On Wed, Aug 5, 2015 at 10:59 AM, Amit Kapila amit.kapil...@gmail.com wrote: OK, committed. I spent some time today reviewing the commited patch. So far my only major complaint is that I think the comments are only insufficiently documenting the approach taken: Stuff like avoiding ABA type

Re: [HACKERS] Bug? ExecChooseHashTableSize() got assertion failed with crazy number of rows

2015-08-19 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Kevin Grittner kgri...@ymail.com writes: Kouhei Kaigai kai...@ak.jp.nec.com wrote: we may need a couple of overhaul around HashJoin to support large size of data, not only nbuckets around 0x8000. Perhaps, but this is a clear bug, introduced to the 9.5

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread David Fetter
On Wed, Aug 19, 2015 at 07:15:30AM +, Laurenz Albe wrote: Victor Wagner wrote: I wonder how useful this is at the present time. Maybe a better idea would be: host=db1.myorg.com,db2.myorg.com port=5432,2345 I think if we're going to provide multiple sets of connection info, we should

Re: [HACKERS] Make HeapTupleSatisfiesMVCC more concurrent

2015-08-19 Thread Andres Freund
On 2015-08-19 10:55:00 -0400, Tom Lane wrote: My own thought about reducing the cost of XidInMVCCSnapshot, if that proves necessary, is that maybe it would be worth the trouble to sort the arrays so we could use binary search. That would increase the cost of snapshot acquisition noticeably

Re: [HACKERS] how to write/setup a C trigger function in a background worker

2015-08-19 Thread David Fetter
On Wed, Aug 19, 2015 at 05:37:31PM +0200, jacques klein wrote: I would like to execute a trigger function (written in C) in one of my background workers. Didn't figure out how to do that not even if it's possible. You can write your trigger function in such a way as not to do the usual check

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread Victor Wagner
On 2015.08.20 at 00:17:35 +0900, Tatsuo Ishii wrote: But once connection is established, each client works with one server (at least until communication failure occurs and it would call PQreset. In this case it has to reprepare statements anyway). One downside of this is, if one of the

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread Tom Lane
Victor Wagner vi...@wagner.pp.ru writes: On 2015.08.20 at 00:17:35 +0900, Tatsuo Ishii wrote: One downside of this is, if one of the standby servers is not responding, every time clients will be blocked by the server before giving up the connection trial. This could last for hours (for This

[HACKERS] Badly broken logic in plpython composite-type handling

2015-08-19 Thread Tom Lane
I looked into the crash reported in bug #13579. The proximate cause of the crash is that PLyString_ToComposite does this: PLy_output_datum_func2(info-out.d, typeTup, exec_ctx-curr_proc-langid, exec_ctx-curr_proc-trftypes); without any

Re: [HACKERS] Our trial to TPC-DS but optimizer made unreasonable plan

2015-08-19 Thread Qingqing Zhou
On Tue, Aug 18, 2015 at 5:59 PM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: BTW, did you register the patch on the upcoming commit-fest? Not yet, it is in WIP status. I think it may be a helpful feature, if we can add alternative subquery-path towards cte-scan on set_cte_pathlist() and choose

Re: [HACKERS] how to write/setup a C trigger function in a background worker

2015-08-19 Thread jacques klein
Well, sorry David, I don't understand what you mean, let me explain what I want to do: in short, IPC between background workers. I am trying to transform my app. from a multi-threaded C SQL-client into some background workers, execution speed beeing the goal (avoid network io). Worker

Re: [HACKERS] proposal: function parse_ident

2015-08-19 Thread Pavel Stehule
Hi 2015-08-19 21:33 GMT+02:00 Tom Lane t...@sss.pgh.pa.us: Pavel Stehule pavel.steh...@gmail.com writes: I miss a functionality that helps with parsing any identifier to basic three parts - database, schema, objectname. We have this function internally, but it is not available for SQL

Re: [HACKERS] Declarative partitioning

2015-08-19 Thread Marc Mamin
On 2015-08-19 AM 02:57, Marc Mamin wrote: 2. Creating a partition of a partitioned table CREATE TABLE table_name PARTITION OF partitioned_table_name FOR VALUES values_spec; Where values_spec is: listvalues: [IN] (val1, ...) Would it make sense to allow one complementary partition to

Re: [HACKERS] proposal: function parse_ident

2015-08-19 Thread Tom Lane
Pavel Stehule pavel.steh...@gmail.com writes: I miss a functionality that helps with parsing any identifier to basic three parts - database, schema, objectname. We have this function internally, but it is not available for SQL layer. FUNCTION parse_ident(IN ident text, OUT dbname text, OUT

Re: [HACKERS] Declarative partitioning

2015-08-19 Thread Josh Berkus
On 08/19/2015 04:59 AM, Simon Riggs wrote: I like the idea of a regular partitioning step because it is how you design such tables - lets use monthly partitions. This gives sanely terse syntax, rather than specifying pages and pages of exact values in DDL PARTITION BY RANGE ON

Re: [HACKERS] Declarative partitioning

2015-08-19 Thread Thom Brown
On 19 August 2015 at 21:10, Josh Berkus j...@agliodbs.com wrote: On 08/19/2015 04:59 AM, Simon Riggs wrote: I like the idea of a regular partitioning step because it is how you design such tables - lets use monthly partitions. This gives sanely terse syntax, rather than specifying pages

Re: [HACKERS] Make HeapTupleSatisfiesMVCC more concurrent

2015-08-19 Thread Jeff Janes
On Tue, Aug 18, 2015 at 5:36 PM, Tom Lane t...@sss.pgh.pa.us wrote: I wrote: Just thinking about this ... I wonder why we need to call TransactionIdIsInProgress() at all rather than believing the answer from the snapshot? Under what circumstances could TransactionIdIsInProgress() return

Re: [HACKERS] Our trial to TPC-DS but optimizer made unreasonable plan

2015-08-19 Thread Josh Berkus
On 08/18/2015 04:40 PM, Qingqing Zhou wrote: Attached please find the WIP patch and also the ANALYZE results. Notes: the patch may not directly apply to head as some network issue here so my Linux box can't talk to git server. So, one of the things we previously mentioned is that currently

Re: [HACKERS] Our trial to TPC-DS but optimizer made unreasonable plan

2015-08-19 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: On 08/18/2015 04:40 PM, Qingqing Zhou wrote: Attached please find the WIP patch and also the ANALYZE results. Notes: the patch may not directly apply to head as some network issue here so my Linux box can't talk to git server. So, one of the things we

Re: [HACKERS] Mention column name in error messages

2015-08-19 Thread Tom Lane
Jeff Janes jeff.ja...@gmail.com writes: The hints you add end in a new line, which then gives two new lines once they are emitted. This is contrary to how other HINTs are formatted. Other HINTs are complete sentences (start with a capital letter, end with a period). But I think these belong

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-19 Thread Jim Nasby
On 8/19/15 2:56 AM, Masahiko Sawada wrote: The currently regression test for VM is that we just compare between the total number of all-visible and all-frozen in VM before and after VACUUM, and don't check particular a bit in VM. we could substitute it to the ANALYZE command with enough sampling

Re: [HACKERS] proposal: function parse_ident

2015-08-19 Thread Jim Nasby
On 8/19/15 2:44 PM, Pavel Stehule wrote: Don't say parse names for things other than tables. Only a minority of the types of objects used in the database have names that meet this specification. Really? My impression is that almost everything that's not a shared object allows for

Re: [HACKERS] how to write/setup a C trigger function in a background worker

2015-08-19 Thread Alvaro Herrera
Bill Moran wrote: On Wed, 19 Aug 2015 19:45:47 +0200 jacques klein jacques.k...@googlemail.com wrote: Well, sorry David, I don't understand what you mean, let me explain what I want to do: in short, IPC between background workers. I am trying to transform my app. from a

Re: [HACKERS] how to write/setup a C trigger function in a background worker

2015-08-19 Thread Bill Moran
On Wed, 19 Aug 2015 19:45:47 +0200 jacques klein jacques.k...@googlemail.com wrote: Well, sorry David, I don't understand what you mean, let me explain what I want to do: in short, IPC between background workers. I am trying to transform my app. from a multi-threaded C SQL-client into

Re: [HACKERS] Mention column name in error messages

2015-08-19 Thread Jeff Janes
On Sun, Aug 9, 2015 at 8:44 AM, Franck Verrot fra...@verrot.fr wrote: On Wed, Jul 1, 2015 at 12:30 AM, Tom Lane t...@sss.pgh.pa.us wrote: What seems more likely to lead to a usable patch is to arrange for the extra information you want to be emitted as error context, via an error context

[HACKERS] Missing equivalent xpath_table in xml2

2015-08-19 Thread Michal Šalko
Hello, I miss the newer features xpath_table equivalent in xml2. Are you planning to replace it with something? Thank's Michal Ing. Michal Šalko SQL specialista, vývojář +420 549 494 572 mailto:sa...@iba.muni.cz sa...@iba.muni.cz Masarykova univerzita, Institut biostatistiky

Re: [HACKERS] Our trial to TPC-DS but optimizer made unreasonable plan

2015-08-19 Thread Josh Berkus
On 08/19/2015 01:32 PM, Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: On 08/18/2015 04:40 PM, Qingqing Zhou wrote: Attached please find the WIP patch and also the ANALYZE results. Notes: the patch may not directly apply to head as some network issue here so my Linux box can't talk to

Re: [HACKERS] Declarative partitioning

2015-08-19 Thread Josh Berkus
On 08/19/2015 01:18 PM, Thom Brown wrote: On 19 August 2015 at 21:10, Josh Berkus j...@agliodbs.com mailto:j...@agliodbs.com wrote: On 08/19/2015 04:59 AM, Simon Riggs wrote: I like the idea of a regular partitioning step because it is how you design such tables - lets use

Re: [HACKERS] how to write/setup a C trigger function in a background worker

2015-08-19 Thread jacques klein
Ok, think I got it, I can use LISTEN and NOTIFY to do my IPC stuf, will just have to see if it's possible to listen in a worker, with a permanent server connection, I guess. (as I remember, I already did a listener 10 years ago in a C client app.) Thanks, Jaquest K. On Wed, 2015-08-19 at 16:54

Re: [HACKERS] Supporting fallback RADIUS server(s)

2015-08-19 Thread Tom Lane
Marko Tiikkaja ma...@joh.to writes: So I'm developing a patch to fix this issue, but I'm not exactly sure what the configuration should look like. I see multiple options, but the one I like the best is the following: Add two new HBA configuration options: radiusfallbackservers and

Re: [HACKERS] Our trial to TPC-DS but optimizer made unreasonable plan

2015-08-19 Thread Kouhei Kaigai
On Mon, Aug 17, 2015 at 9:40 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: I think SortSupport logic provides a reasonable way to solve this kind of problem. For example, btint4sortsupport() informs a function pointer of the fast version of comparator (btint4fastcmp) which takes two Datum

Re: [HACKERS] Our trial to TPC-DS but optimizer made unreasonable plan

2015-08-19 Thread Peter Geoghegan
On Wed, Aug 19, 2015 at 6:08 PM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Indeed, 6 of 8 grouping keys in this query uses bpchar() data type, so it is natural comparison function consumed larger portion of CPU cycles. Do we have any idea to assist these queries by the backend? With

Re: [HACKERS] PATCH: use foreign keys to improve join estimates v1

2015-08-19 Thread Tomas Vondra
Hi, attached is a significantly reworked patch for using the foreign keys in selectivity estimation. The previous patch only really worked if the clauses matched the foreign key perfectly (i.e. no additional join clauses) - this patch attempts to relax those restrictions a bit. This patch

Re: [HACKERS] PATCH: use foreign keys to improve join estimates v1

2015-08-19 Thread Tomas Vondra
On 08/20/2015 03:49 AM, Tomas Vondra wrote: Then on current master, I get these estimates (showing just rows, because that's what matter): while with the patch I get this: And of course I forgot to include the plans from master, so here we go: select * from f join a on (f1 = a1 and f2 =

Re: [HACKERS] Badly broken logic in plpython composite-type handling

2015-08-19 Thread Tom Lane
I wrote: I looked into the crash reported in bug #13579. The proximate cause of the crash is that PLyString_ToComposite does this: ... I'm inclined to think that maybe PLyString_ToComposite needs to be doing something more like what PLyObject_ToComposite does, ie doing its own lookup in a

[HACKERS] Supporting fallback RADIUS server(s)

2015-08-19 Thread Marko Tiikkaja
Hi, We use RADIUS authentication at $WORK, and it has one major flaw (well, two, but I already fixed the other one this week): it only supports specifying a single server, which as we might know, is bad for high availability. So I'm developing a patch to fix this issue, but I'm not exactly

Re: [HACKERS] Supporting fallback RADIUS server(s)

2015-08-19 Thread Marko Tiikkaja
On 2015-08-20 02:29, Tom Lane wrote: Marko Tiikkaja ma...@joh.to writes: So I'm developing a patch to fix this issue, but I'm not exactly sure what the configuration should look like. I see multiple options, but the one I like the best is the following: Add two new HBA configuration

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-19 Thread Tomas Vondra
Hi, On 08/19/2015 01:55 PM, Kohei KaiGai wrote: Merge Join (cost=25374644.08..1160509591.61 rows=60521928028 width=24) (actual time=138347.979..491889.343 rows=776157676 loops=1) Merge Cond: (ws1.ws_order_number = ws2.ws_order_number) Join Filter: (ws1.ws_warehouse_sk

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-19 Thread Tomas Vondra
Hello KaiGain-san, On 08/19/2015 03:19 PM, Kohei KaiGai wrote: Unless we have no fail-safe mechanism when planner estimated too large number of tuples than actual needs, a strange estimation will consume massive amount of RAMs. It's a bad side effect. My previous patch didn't pay attention to

Re: [HACKERS] Declarative partitioning

2015-08-19 Thread Amit Langote
On 2015-08-19 PM 09:23, Simon Riggs wrote: On 18 August 2015 at 11:30, Amit Langote langote_amit...@lab.ntt.co.jp wrote: There is no need to define tuple routing triggers. CopyFrom() and ExecInsert() determine target partition just before performing heap_insert() and

[HACKERS] Using quicksort for every external sort run

2015-08-19 Thread Peter Geoghegan
I'll start a new thread for this, since my external sorting patch has now evolved well past the original quicksort with spillover idea...although not quite how I anticipated it would. It seems like I've reached a good point to get some feedback. I attach a patch series featuring a new, more

Re: [HACKERS] Badly broken logic in plpython composite-type handling

2015-08-19 Thread Joshua D. Drake
On 08/19/2015 05:05 PM, Tom Lane wrote: Barring objections, I propose to commit and back-patch this. The crash can be demonstrated back to 9.1. regards, tom lane +1 -- Command Prompt, Inc. - http://www.commandprompt.com/ 503-667-4564 PostgreSQL Centered full

Re: [HACKERS] proposal: function parse_ident

2015-08-19 Thread Tom Lane
Jim Nasby jim.na...@bluetreble.com writes: Don't say parse names for things other than tables. Only a minority of the types of objects used in the database have names that meet this specification. Really? My impression is that almost everything that's not a shared object allows for a

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-19 Thread David Rowley
On 12 June 2015 at 02:40, Kohei KaiGai kai...@kaigai.gr.jp wrote: 2015-06-11 23:28 GMT+09:00 Robert Haas robertmh...@gmail.com: On Wed, Jun 10, 2015 at 10:57 PM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: The attached patch replaces this palloc0() by MemoryContextAllocHuge() + memset().

Re: [HACKERS] Make HeapTupleSatisfiesMVCC more concurrent

2015-08-19 Thread Amit Kapila
On Wed, Aug 19, 2015 at 6:06 AM, Tom Lane t...@sss.pgh.pa.us wrote: I wrote: Just thinking about this ... I wonder why we need to call TransactionIdIsInProgress() at all rather than believing the answer from the snapshot? Under what circumstances could TransactionIdIsInProgress() return

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread 'Victor Wagner *EXTERN*'
On 2015.08.18 at 08:32:28 +, Albe Laurenz wrote: I wonder how useful this is at the present time. If the primary goes down and the client gets connected to the standby, it would have read-only access there. Most applications wouldn't cope well with that. It is supposed that somebody

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread Amit Kapila
On Wed, Aug 19, 2015 at 12:21 PM, Victor Wagner *EXTERN* vi...@wagner.pp.ru wrote: On 2015.08.18 at 08:32:28 +, Albe Laurenz wrote: I wonder how useful this is at the present time. If the primary goes down and the client gets connected to the standby, it would have read-only access

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread Victor Wagner
On 2015.08.19 at 08:28:32 +0530, Amit Kapila wrote: On Tue, Aug 18, 2015 at 9:48 AM, Victor Wagner vi...@wagner.pp.ru wrote: Behavoir If PQconnectdb encounters connect string with multiple hosts specified, it attempts to establish connection with all these hosts

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread Victor Wagner *EXTERN*
On 2015.08.19 at 12:29:51 +0530, Amit Kapila wrote: It seems that most people discussing in this thread think in millisecond time intervals (failure and immediate reconnect). Why not have this as a separate parameter (*_timeout or something like that)? Because it is not in the software

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread Amit Kapila
On Wed, Aug 19, 2015 at 12:35 PM, Victor Wagner vi...@wagner.pp.ru wrote: On 2015.08.19 at 08:28:32 +0530, Amit Kapila wrote: On Tue, Aug 18, 2015 at 9:48 AM, Victor Wagner vi...@wagner.pp.ru wrote: Behavoir If PQconnectdb encounters connect string with multiple

Re: [HACKERS] checkpointer continuous flushing

2015-08-19 Thread Fabien COELHO
Sure, I think what can help here is a testcase/'s (in form of script file or some other form, to test this behaviour of patch) which you can write and post here, so that others can use that to get the data and share it. Sure... note that I already did that on this thread, without any echo...

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread Victor Wagner
On 2015.08.19 at 12:42:45 +0900, Tatsuo Ishii wrote: I wonder how extended protocol is handled by this proposal. Suppose load balacing mode is enabled. PQprepare is executed on standby1. Then PQexecPrepared gets called. This may be executed on standby2, which will fail because there's no

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread Albe Laurenz
Victor Wagner wrote: I wonder how useful this is at the present time. If the primary goes down and the client gets connected to the standby, it would have read-only access there. Most applications wouldn't cope well with that. It is supposed that somebody (either system administrator or

Re: [HACKERS] Declarative partitioning

2015-08-19 Thread Amit Langote
On 2015-08-18 PM 10:43, David Fetter wrote: After the first command is done, the second command would take exclusive lock on table_name, scan the table to check if it contains any values outside the boundaries defined by FOR VALUES clause defined previously, throw error if so, mark as valid

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread ''Victor Wagner *EXTERN*' *EXTERN*'
On 2015.08.19 at 07:15:30 +, Albe Laurenz wrote: Idea is that we don't need any extra administration actions such as IP migration to do it. Clients have list of alternate servers and discover which one to work with by trial and error. Yes, but that will only work reliably if the

Re: [HACKERS] Make HeapTupleSatisfiesMVCC more concurrent

2015-08-19 Thread Simon Riggs
On 19 August 2015 at 00:49, Tom Lane t...@sss.pgh.pa.us wrote: Jeff Janes jeff.ja...@gmail.com writes: When we check a tuple for MVCC, it has to pass checks that the inserting transaction has committed, and that it committed before our snapshot began. And similarly that the deleting

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-19 Thread Masahiko Sawada
On Wed, Aug 19, 2015 at 1:28 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, Aug 18, 2015 at 7:27 AM, Masahiko Sawada sawada.m...@gmail.com wrote: I have encountered the much cases where pg_stat_statement, pgstattuples are required in production, so I basically agree with moving such

Re: [HACKERS] Declarative partitioning

2015-08-19 Thread Amit Langote
On 2015-08-19 AM 02:57, Marc Mamin wrote: 2. Creating a partition of a partitioned table CREATE TABLE table_name PARTITION OF partitioned_table_name FOR VALUES values_spec; Where values_spec is: listvalues: [IN] (val1, ...) Would it make sense to allow one complementary partition to

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread Albe Laurenz
Victor Wagner wrote: Idea is that we don't need any extra administration actions such as IP migration to do it. Clients have list of alternate servers and discover which one to work with by trial and error. Yes, but that will only work reliably if the (read-only) standby does not

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread Victor Wagner
On 2015.08.19 at 12:55:15 +0530, Amit Kapila wrote: I think that failover procedure should begin before first connection is ever established. As far as I understand, failover gets initiated once the master server goes down or is not accessible due to some reason, so for such cases if

Re: [HACKERS] allowing wal_level change at run time

2015-08-19 Thread Magnus Hagander
On Tue, Aug 18, 2015 at 7:46 PM, Andres Freund and...@anarazel.de wrote: On 2015-08-18 13:24:54 -0400, Peter Eisentraut wrote: On 8/18/15 12:35 PM, Robert Haas wrote: If archive_mode=on or max_wal_senders0, then we need at least wal_level=archive. Otherwise wal_level=minimal is enough.

Re: [HACKERS] Proposal: Implement failover on libpq connect level.

2015-08-19 Thread Amit Kapila
On Wed, Aug 19, 2015 at 1:23 PM, Victor Wagner vi...@wagner.pp.ru wrote: On 2015.08.19 at 12:55:15 +0530, Amit Kapila wrote: I think that failover procedure should begin before first connection is ever established. As far as I understand, failover gets initiated once the master

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-19 Thread Tomas Vondra
Hi, On 08/20/2015 04:15 AM, Tomas Vondra wrote: Hello KaiGain-san, On 08/19/2015 03:19 PM, Kohei KaiGai wrote: Unless we have no fail-safe mechanism when planner estimated too large number of tuples than actual needs, a strange estimation will consume massive amount of RAMs. It's a bad side