Re: [HACKERS] Relation extension scalability

2015-03-29 Thread Amit Kapila
On Mon, Mar 30, 2015 at 12:26 AM, Andres Freund and...@2ndquadrant.com wrote: Hello, Currently bigger shared_buffers settings don't combine well with relations being extended frequently. Especially if many/most pages have a high usagecount and/or are dirty and the system is IO constrained.

Re: [HACKERS] Relation extension scalability

2015-03-29 Thread Andres Freund
On 2015-03-29 20:02:06 -0400, Robert Haas wrote: On Sun, Mar 29, 2015 at 2:56 PM, Andres Freund and...@2ndquadrant.com As a quick recap, relation extension basically works like: 1) We lock the relation for extension 2) ReadBuffer*(P_NEW) is being called, to extend the relation 3)

Re: [HACKERS] Rounding to even for numeric data type

2015-03-29 Thread Michael Paquier
On Mon, Mar 30, 2015 at 4:51 AM, James Cloos cl...@jhcloos.com wrote: MP == Michael Paquier michael.paqu...@gmail.com writes: MP So, attached is a patch that does 1) and 2) to make clear to the MP user how numeric and double precision behave regarding rounding. MP I am adding it to CF 2015-06

Re: [HACKERS] Relation extension scalability

2015-03-29 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: I thought the primary reason we did this is because we wanted to write-and-fsync the block so that, if we're out of disk space, any attendant failure will happen before we put data into the block. Once we've initialized the block, a subsequent failure

Re: [HACKERS] PATCH: pgbench - merging transaction logs

2015-03-29 Thread Fabien COELHO
Hello Tomas, The results are as follow: * 1 thread 33 runs median tps (average is consistent): - no logging:22062 - separate logging: 19360 (-12.2%) - merged logging:19326 (-12.4%, not significant from previous) Interesting. What hardware is this? Dell PowerEdge R720

Re: [HACKERS] PATCH: pgbench - merging transaction logs

2015-03-29 Thread Fabien COELHO
I also implemented a quick and dirty version for a merge log based on sharing a file handle (append mode + sprintf + fputs). I tried the append + per-thread 2KB buffered sprintf + fputs when full, with the same number of runs. The logs are out of order by chunks, the overhead seems higher

Re: [HACKERS] TRAP: BadArgument - mcxt.c, Line: 813

2015-03-29 Thread Erik Rijkers
On Sun, March 29, 2015 17:40, Erik Rijkers wrote: With the latest server testdb=# select version(); version --

Re: [HACKERS] getting rid of thread fork emulation in pgbench?

2015-03-29 Thread Tom Lane
Fabien COELHO coe...@cri.ensmp.fr writes: There is a thread fork emulation hack which allows pgbench to be compiled without threads by emulating them with forks, obviously with limited capabilities. This feature is triggered by configuring postgresql with --disable-thread-safety. I'm not

[HACKERS] TRAP: BadArgument - mcxt.c, Line: 813

2015-03-29 Thread Erik Rijkers
With the latest server testdb=# select version(); version -- PostgreSQL 9.5devel_HEAD_20150329_0154_2c33e0fbceb0 on

Re: [HACKERS] getting rid of thread fork emulation in pgbench?

2015-03-29 Thread Robert Haas
On Sun, Mar 29, 2015 at 1:49 PM, Fabien COELHO coe...@cri.ensmp.fr wrote: Now if we have threads, it is simpler (if the overhead is reasonable) that threads share a file handle and just generate one file, so there is no merging. I really, really wish you'd stop arguing against the patch to

Re: [HACKERS] Relation extension scalability

2015-03-29 Thread Robert Haas
On Sun, Mar 29, 2015 at 2:56 PM, Andres Freund and...@2ndquadrant.com As a quick recap, relation extension basically works like: 1) We lock the relation for extension 2) ReadBuffer*(P_NEW) is being called, to extend the relation 3) smgrnblocks() is used to find the new target block 4) We

[HACKERS] cache lookup error for shell type creation with incompatible output function (DDL deparsing bug)

2015-03-29 Thread Michael Paquier
Hi all, I just bumped into the following problem in HEAD (1c41e2a): =# create type my_array_float (INPUT = array_in, OUTPUT = array_out, ELEMENT = float4, INTERNALLENGTH = 32); ERROR: XX000: cache lookup failed for type 0 LOCATION: format_type_internal, format_type.c:135 First note that in

Re: [HACKERS] Combining Aggregates

2015-03-29 Thread David Rowley
On 18 December 2014 at 02:48, Simon Riggs si...@2ndquadrant.com wrote: David, if you can update your patch with some docs to explain the behaviour, it looks complete enough to think about committing it in early January, to allow other patches that depend upon it to stand a chance of getting

Re: [HACKERS] Combining Aggregates

2015-03-29 Thread Michael Paquier
On Mon, Mar 30, 2015 at 2:08 PM, David Rowley dgrowle...@gmail.com wrote: On 18 December 2014 at 02:48, Simon Riggs si...@2ndquadrant.com wrote: David, if you can update your patch with some docs to explain the behaviour, it looks complete enough to think about committing it in early

Re: [HACKERS] TRAP: BadArgument - mcxt.c, Line: 813

2015-03-29 Thread Tom Lane
Erik Rijkers e...@xs4all.nl writes: I get a crash while restoring a trgm GIN index. Hm, interesting: this seems to be the first negative fallout from commit eaa5808e8ec4e82ce1a87103a6b6f687666e4e4c, which made MemoryContextReset() delete not reset child contexts. ginbuild() is creating a

Re: [HACKERS] Providing catalog view to pg_hba.conf file - Patch submission

2015-03-29 Thread Pavel Stehule
Hi I checked this patch. I like the functionality and behave. There is minor issue with outdated regress test test rules... FAILED I have no objections. Regards Pavel 2015-03-27 9:23 GMT+01:00 Haribabu Kommi kommi.harib...@gmail.com: On Fri, Mar 13, 2015 at 1:33 PM,

Re: [HACKERS] improve pgbench syntax error messages

2015-03-29 Thread Fabien COELHO
Here is a v5. Here is v6, just a rebase. -- Fabien.diff --git a/contrib/pgbench/exprparse.y b/contrib/pgbench/exprparse.y index e68631e..68c85c9 100644 --- a/contrib/pgbench/exprparse.y +++ b/contrib/pgbench/exprparse.y @@ -26,6 +26,13 @@ static PgBenchExpr *make_op(char operator,

Re: [HACKERS] extend pgbench expressions with functions

2015-03-29 Thread Fabien COELHO
Here is a small v2 update: v3, just a rebase. -- Fabien.diff --git a/contrib/pgbench/exprparse.y b/contrib/pgbench/exprparse.y index e68631e..049937f 100644 --- a/contrib/pgbench/exprparse.y +++ b/contrib/pgbench/exprparse.y @@ -20,6 +20,7 @@ static PgBenchExpr *make_integer_constant(int64

Re: [HACKERS] getting rid of thread fork emulation in pgbench?

2015-03-29 Thread Fabien COELHO
Hello Tom, My question is: would someone still object strongly to the removal of this thread fork emulation hack in pgbench? By my count, the pthread-emulation code amounts to about 175 lines out of the nearly 4000 in pgbench.c. That's not an undue burden IMO (it's not, for comparison's

Re: [HACKERS] proposal: row_to_array function

2015-03-29 Thread Tom Lane
Pavel Stehule pavel.steh...@gmail.com writes: here is rebased patch. It contains both patches - row_to_array function and foreach array support. While I don't have a problem with hstore_to_array, I don't think that row_to_array is a very good idea; it's basically encouraging people to throw

Re: [HACKERS] proposal: row_to_array function

2015-03-29 Thread Pavel Stehule
Hi here is rebased patch. It contains both patches - row_to_array function and foreach array support. This design is in conformity with hstore functions. There can be good synergy. Regards Pavel 2015-03-28 23:53 GMT+01:00 Jeff Janes jeff.ja...@gmail.com: On Tue, Jan 27, 2015 at 10:58 AM,

Re: [HACKERS] getting rid of thread fork emulation in pgbench?

2015-03-29 Thread Tom Lane
Fabien COELHO coe...@cri.ensmp.fr writes: And what will you do instead? It would be fine I think for pgbench to not allow --jobs different from 1 on a threadless platform, but not for it to fail to work altogether. Sure. No thread really means working with only one thread. It looks to me

[HACKERS] Relation extension scalability

2015-03-29 Thread Andres Freund
Hello, Currently bigger shared_buffers settings don't combine well with relations being extended frequently. Especially if many/most pages have a high usagecount and/or are dirty and the system is IO constrained. As a quick recap, relation extension basically works like: 1) We lock the relation

Re: [HACKERS] How about to have relnamespace and relrole?

2015-03-29 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: I have just claimed this as committer in the CF, but on reviewing the emails it looks like there is disagreement about the need for it at all, especially from Tom and Robert. I confess I have often wanted regnamespace, particularly, and

Re: [HACKERS] getting rid of thread fork emulation in pgbench?

2015-03-29 Thread Fabien COELHO
As I said, I wouldn't have a problem with restricting the --jobs setting on threadless platforms, which seems like it would fix your problem since you wouldn't need to have more than one process involved. Indeed! So I undestand that thread fork emulation could eventually go from your point

Re: [HACKERS] Rounding to even for numeric data type

2015-03-29 Thread Michael Paquier
On Sun, Mar 29, 2015 at 9:21 AM, Gavin Flower gavinflo...@archidevsys.co.nz wrote: On 29/03/15 13:07, David G. Johnston wrote: On Sat, Mar 28, 2015 at 3:59 PM, Michael Paquier michael.paqu...@gmail.com mailto:michael.paqu...@gmail.comwrote: On Sun, Mar 29, 2015 at 5:34 AM, Gavin Flower

Re: [HACKERS] Rounding to even for numeric data type

2015-03-29 Thread Michael Paquier
On Sun, Mar 29, 2015 at 7:59 AM, Michael Paquier michael.paqu...@gmail.com wrote: On Sun, Mar 29, 2015 at 5:34 AM, Gavin Flower gavinflo...@archidevsys.co.nz wrote: On 28/03/15 21:58, Dean Rasheed wrote: [...] Andrew mentioned that there have been complaints from people doing calculations

[HACKERS] getting rid of thread fork emulation in pgbench?

2015-03-29 Thread Fabien COELHO
Hello hackers, This is take 2, I already suggested this 2 years ago... There is a thread fork emulation hack which allows pgbench to be compiled without threads by emulating them with forks, obviously with limited capabilities. This feature is triggered by configuring postgresql with

Re: [HACKERS] Exposing PG_VERSION_NUM in pg_config

2015-03-29 Thread Michael Paquier
On Wed, Mar 25, 2015 at 2:32 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Wed, Mar 25, 2015 at 8:26 AM, Tom Lane t...@sss.pgh.pa.us wrote: Hm. We're all agreed that there's a use case for exposing PG_VERSION_NUM to the makefiles, but I did not hear one for adding it to pg_config;

Re: [HACKERS] Relation extension scalability

2015-03-29 Thread Andres Freund
On 2015-03-29 15:21:44 -0400, Tom Lane wrote: There's two things that seem to make sense to me: First, decouple relation extension from ReadBuffer*, i.e. remove P_NEW and introduce a bufmgr function specifically for extension. I think that removing P_NEW is likely to require a fair amount

Re: [HACKERS] Rounding to even for numeric data type

2015-03-29 Thread James Cloos
MP == Michael Paquier michael.paqu...@gmail.com writes: MP So, attached is a patch that does 1) and 2) to make clear to the MP user how numeric and double precision behave regarding rounding. MP I am adding it to CF 2015-06 to keep track of it... Given that the examples show -2.5 rounds to -3,

Re: [HACKERS] Relation extension scalability

2015-03-29 Thread Andres Freund
On 2015-03-29 16:07:49 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2015-03-29 15:21:44 -0400, Tom Lane wrote: One idea that might help is to change smgrextend's API so that it doesn't need a buffer to write from, but just has an API of add a prezeroed block

Re: [HACKERS] compute_index_stats is missing a CHECK_FOR_INTERRUPTS

2015-03-29 Thread Tom Lane
Jeff Janes jeff.ja...@gmail.com writes: On Sat, Mar 28, 2015 at 3:37 PM, Tom Lane t...@sss.pgh.pa.us wrote: Hm. The other per-sample-row loops in analyze.c use vacuum_delay_point() rather than CHECK_FOR_INTERRUPTS() directly. Ordinarily that wouldn't make much difference here, but maybe a

Re: [HACKERS] Relation extension scalability

2015-03-29 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: As a quick recap, relation extension basically works like: 1) We lock the relation for extension 2) ReadBuffer*(P_NEW) is being called, to extend the relation 3) smgrnblocks() is used to find the new target block 4) We search for a victim buffer

Re: [HACKERS] proposal: row_to_array function

2015-03-29 Thread Pavel Stehule
2015-03-29 20:27 GMT+02:00 Tom Lane t...@sss.pgh.pa.us: Pavel Stehule pavel.steh...@gmail.com writes: here is rebased patch. It contains both patches - row_to_array function and foreach array support. While I don't have a problem with hstore_to_array, I don't think that row_to_array is a

Re: [HACKERS] Relation extension scalability

2015-03-29 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2015-03-29 15:21:44 -0400, Tom Lane wrote: One idea that might help is to change smgrextend's API so that it doesn't need a buffer to write from, but just has an API of add a prezeroed block on-disk and tell me the number of the block you added.

Re: [HACKERS] Removing INNER JOINs

2015-03-29 Thread Tom Lane
David Rowley dgrowle...@gmail.com writes: I'm not worried about the cost of the decision at plan init time. I was just confused about what Tom was recommending. I couldn't quite decide from his email if he meant to keep the alternative plan node around so that the executor must transition