Re: [HACKERS] delta relations in AFTER triggers

2014-08-27 Thread Heikki Linnakangas
On 08/27/2014 02:26 AM, Kevin Grittner wrote: spi-tuplestore-registry allows tuplestores, with associated name and TupleDesc, to be registered with the current SPI connection. Queries planned or executed on that connection will recognize the name as a tuplestore relation. It doesn't care who is

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-27 Thread Fabien COELHO
Hello Andres, [...] I think you're misunderstanding how spread checkpoints work. Yep, definitely:-) On the other hand I though I was seeking something "simple", namely correct latency under small load, that I would expect out of the box. What you describe is reasonable, and is more or les

Re: [HACKERS] Allow multi-byte characters as escape in SIMILAR TO and SUBSTRING

2014-08-27 Thread Heikki Linnakangas
On 08/27/2014 08:13 AM, Jeff Davis wrote: On Mon, 2014-08-25 at 17:41 +0300, Heikki Linnakangas wrote: your patch seems to be about 2x-3x as slow as unpatched master. So this needs some optimization. A couple of ideas: I didn't see anywhere near that kind of regression. On unpatched master, wi

Re: [HACKERS] pgbench throttling latency limit

2014-08-27 Thread Heikki Linnakangas
On 08/27/2014 03:47 AM, Rukh Meski wrote: Hi Fabien, On Tue, Aug 26, 2014 at 04:07 AM, Fabien COELHO wrote: Please find attached a new version which fixes these two points. Indeed it does. Marking the patch ready for a committer. I find the definition of the latency limit a bit strange.

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-27 Thread Andres Freund
On 2014-08-27 09:32:16 +0200, Fabien COELHO wrote: > > Hello Andres, > > >[...] > >I think you're misunderstanding how spread checkpoints work. > > Yep, definitely:-) On the other hand I though I was seeking something > "simple", namely correct latency under small load, that I would expect out >

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-27 Thread Fabien COELHO
[...] What's your evidence the pacing doesn't work? Afaik it's the fsync that causes the problem, not the the writes themselves. Hmmm. My (poor) understanding is that fsync would work fine if everything was already written beforehand:-) that is it has nothing to do but assess that all is alr

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-27 Thread Andres Freund
On 2014-08-27 11:05:52 +0200, Fabien COELHO wrote: > I can test a couple of patches. I already did one on someone advice (make > bgwriter round all stuff in 1s instead of 120s, without positive effect. I've quickly cobbled together the attached patch (which at least doesn't seem to crash & burn).

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-27 Thread David Rowley
On Wed, Aug 27, 2014 at 1:40 AM, Heikki Linnakangas wrote: > On 08/26/2014 03:28 PM, David Rowley wrote: > >> Any ideas or feedback on this would be welcome >> > > Before someone spends time reviewing this patch, are you sure this is > worth the effort? It seems like very narrow use case to me. I

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-27 Thread Andres Freund
On 2014-08-27 11:14:46 +0200, Andres Freund wrote: > On 2014-08-27 11:05:52 +0200, Fabien COELHO wrote: > > I can test a couple of patches. I already did one on someone advice (make > > bgwriter round all stuff in 1s instead of 120s, without positive effect. > > I've quickly cobbled together the a

Re: [HACKERS] pgbench throttling latency limit

2014-08-27 Thread Fabien COELHO
Hello Heikki, I find the definition of the latency limit a bit strange. It's a limit on how late a transaction can *start* compared to it's scheduled starting time, not how long a query is allowed to last. Yes. This is what can be done easily with pgbench under throttling. Note that if tran

[HACKERS] re-reading SSL certificates during server reload

2014-08-27 Thread Alexey Klyukin
Greetings, Is there a strong reason to disallow reloading server key and cert files during the PostgreSQL reload? Basically, once you run multiple databases in a cluster and use different DNS names to connect to different databases (in order for those databases to be moved somewhere without chang

Re: [HACKERS] pgbench throttling latency limit

2014-08-27 Thread Heikki Linnakangas
On 08/27/2014 12:41 PM, Fabien COELHO wrote: Hello Heikki, I find the definition of the latency limit a bit strange. It's a limit on how late a transaction can *start* compared to it's scheduled starting time, not how long a query is allowed to last. Yes. This is what can be done easily with

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-27 Thread Andres Freund
On 2014-08-27 11:19:22 +0200, Andres Freund wrote: > On 2014-08-27 11:14:46 +0200, Andres Freund wrote: > > On 2014-08-27 11:05:52 +0200, Fabien COELHO wrote: > > > I can test a couple of patches. I already did one on someone advice (make > > > bgwriter round all stuff in 1s instead of 120s, withou

Re: [HACKERS] re-reading SSL certificates during server reload

2014-08-27 Thread Magnus Hagander
On Wed, Aug 27, 2014 at 11:56 AM, Alexey Klyukin wrote: > Greetings, > > Is there a strong reason to disallow reloading server key and cert files > during the PostgreSQL reload? Key and cert files are loaded in the postmaster. We'd need to change that. I'm not saying that's not a good idea, but i

Re: [HACKERS] Sample LDIF for pg_service.conf no longer works

2014-08-27 Thread Magnus Hagander
On Sat, Aug 16, 2014 at 5:31 AM, Noah Misch wrote: > When using pg_service.conf with LDAP, we document[1] the following sample LDIF > for populating the LDAP server: > > version:1 > dn:cn=mydatabase,dc=mycompany,dc=com > changetype:add > objectclass:top > objectclass:groupOfUniqueNames > cn:mydata

Re: [HACKERS] Missing comment block at the top of streamutil.h and receivelog.h

2014-08-27 Thread Fujii Masao
On Wed, Aug 27, 2014 at 2:07 PM, Michael Paquier wrote: > On Wed, Aug 27, 2014 at 12:31 PM, Fujii Masao wrote: >> On Wed, Aug 27, 2014 at 10:34 AM, Michael Paquier >> wrote: >> I think that it's better to add an include guard like >> >> #ifndef RECEIVELOG_H >> #define RECEIVELOG_H >> ... >> #end

Re: [HACKERS] pgbench throttling latency limit

2014-08-27 Thread Fabien COELHO
Hello Heikki, [...] With a latency limit on when the query should finish, as opposed to how late it can start, it's a lot easier to give a number. For example, your requirements might state that a user must always get a response to a click on a web page in 200 ms, so you set the limit to 200

Re: [HACKERS] Audit of logout

2014-08-27 Thread Fujii Masao
On Sat, Aug 23, 2014 at 3:44 PM, Amit Kapila wrote: > On Tue, Aug 5, 2014 at 8:04 PM, Fujii Masao wrote: >> >> Yep, the attached patch introduces PGC_SU_BACKEND and >> changes the contexts of log_connections and log_disconnections >> to PGC_SU_BACKEND. Review? >> Thanks for reviewing the patch!

Re: [HACKERS] Specifying the unit in storage parameter

2014-08-27 Thread Fujii Masao
On Tue, Aug 26, 2014 at 3:27 AM, Alvaro Herrera wrote: > Fujii Masao wrote: >> On Thu, Aug 21, 2014 at 4:20 PM, Michael Paquier >> wrote: > >> > Looking at the patch, the parameter "fillfactor" in the category >> > RELOPT_KIND_HEAP (the first element in intRelOpts of reloptions.c) is >> > not upd

[HACKERS] Simplify calls of pg_class_aclcheck when multiple modes are used

2014-08-27 Thread Michael Paquier
Hi all, In a couple of code paths we do the following to check permissions on an object: if (pg_class_aclcheck(relid, userid, ACL_USAGE) != ACLCHECK_OK && pg_class_aclcheck(relid, userid, ACL_UPDATE) != ACLCHECK_OK) ereport(ERROR, blah); Wouldn't it be better to simplify that with a singl

Re: [HACKERS] After switching primary server while using replication slot.

2014-08-27 Thread Fujii Masao
On Fri, Aug 22, 2014 at 11:29 PM, Andres Freund wrote: > Hi, > > On 2014-08-20 13:14:30 -0400, Robert Haas wrote: >> On Tue, Aug 19, 2014 at 6:25 AM, Fujii Masao wrote: >> > On Mon, Aug 18, 2014 at 11:16 PM, Sawada Masahiko >> > wrote: >> >> Hi all, >> >> After switching primary serer while usi

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-27 Thread Fabien COELHO
Hello Amit, I see there is some merit in your point which is to make bgwriter more useful than its current form. I could see 3 top level points to think about whether improvement in any of those can improve the current situation: a. Scanning of buffer pool to find the dirty buffers that ca

Re: [HACKERS] Parallel Sequence Scan doubts

2014-08-27 Thread Robert Haas
On Thu, Aug 21, 2014 at 2:47 AM, Haribabu Kommi wrote: > Implementation of "Parallel Sequence Scan" > > Approach: > > 1."Parallel Sequence Scan" can achieved by using the background > workers doing the job of actual sequence scan including the > qualification check also. > > 2. Planner generates t

Re: [HACKERS] Function to know last log write timestamp

2014-08-27 Thread Fujii Masao
On Tue, Aug 19, 2014 at 1:07 AM, Robert Haas wrote: > On Fri, Aug 15, 2014 at 7:17 AM, Fujii Masao wrote: >> On Fri, Aug 15, 2014 at 3:40 AM, Andres Freund >> wrote: >>> On 2014-08-14 14:37:22 -0400, Robert Haas wrote: On Thu, Aug 14, 2014 at 2:21 PM, Andres Freund wrote: > On

Re: [HACKERS] re-reading SSL certificates during server reload

2014-08-27 Thread Stephen Frost
* Magnus Hagander (mag...@hagander.net) wrote: > That's certainly an issue. Potentially bigger ones are that you cannot > replace an expired certificate or CRL without a restart. +100. I had forgotten about that issue- but it definitely sucks. :( > Some of this is going to have to be at least pa

Re: [HACKERS] Code bug or doc bug?

2014-08-27 Thread Robert Haas
On Sun, Aug 24, 2014 at 6:26 PM, Josh Berkus wrote: > Quoth our docs > (http://www.postgresql.org/docs/9.3/static/sql-alterdatabase.html): > > "The fourth form changes the default tablespace of the database. Only > the database owner or a superuser can do this; you must also have create > privileg

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-27 Thread Claudio Freire
On Wed, Aug 27, 2014 at 6:05 AM, Fabien COELHO wrote: >> [...] What's your evidence the pacing doesn't work? Afaik it's the fsync >> that causes the problem, not the the writes themselves. > > > Hmmm. My (poor) understanding is that fsync would work fine if everything > was already written beforeh

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-27 Thread Claudio Freire
On Wed, Aug 27, 2014 at 10:10 AM, Claudio Freire wrote: > On Wed, Aug 27, 2014 at 6:05 AM, Fabien COELHO wrote: >>> [...] What's your evidence the pacing doesn't work? Afaik it's the fsync >>> that causes the problem, not the the writes themselves. >> >> >> Hmmm. My (poor) understanding is that f

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-27 Thread Andres Freund
On 2014-08-27 10:10:49 -0300, Claudio Freire wrote: > On Wed, Aug 27, 2014 at 6:05 AM, Fabien COELHO wrote: > >> [...] What's your evidence the pacing doesn't work? Afaik it's the fsync > >> that causes the problem, not the the writes themselves. > > > > > > Hmmm. My (poor) understanding is that f

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-27 Thread Claudio Freire
On Wed, Aug 27, 2014 at 10:15 AM, Andres Freund wrote: > On 2014-08-27 10:10:49 -0300, Claudio Freire wrote: >> On Wed, Aug 27, 2014 at 6:05 AM, Fabien COELHO wrote: >> >> [...] What's your evidence the pacing doesn't work? Afaik it's the fsync >> >> that causes the problem, not the the writes th

Re: [HACKERS] WIP Patch for GROUPING SETS phase 1

2014-08-27 Thread Robert Haas
On Mon, Aug 25, 2014 at 1:35 AM, Andrew Gierth wrote: > If you look at the latest patch post, there's a small patch in it that > does nothing but unreserve the keywords and fix ruleutils to make > deparse/parse work. The required fix to ruleutils is an example of > violating your "four kinds of ke

Re: [HACKERS] Code bug or doc bug?

2014-08-27 Thread Bruce Momjian
On Wed, Aug 27, 2014 at 09:05:41AM -0400, Robert Haas wrote: > Another idea is to have a command that you can run, while connected to > a particular database, that updates the default tablespace for that > database without actually moving any data on disk - i.e. it sets > pg_database.dattablespace,

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-27 Thread Andres Freund
On 2014-08-27 10:17:06 -0300, Claudio Freire wrote: > > I think a somewhat smarter version of the explicit flushes in the > > hack^Wpatch I posted nearby is going to more likely to be successful. > > > That path is "dangerous" (as in, may not work as intended) if the > filesystem doesn't properly

Re: [HACKERS] Set new system identifier using pg_resetxlog

2014-08-27 Thread Robert Haas
On Mon, Aug 25, 2014 at 4:06 PM, Heikki Linnakangas wrote: > I didn't understand this one. But it seems like the obvious solution is to > not use the consumer's system identifier as the slot name. Or rename it > afterwards. You can't use the consumer's system identifier as the slot name, because

Re: [HACKERS] Code bug or doc bug?

2014-08-27 Thread David G Johnston
Bruce Momjian wrote > On Wed, Aug 27, 2014 at 09:05:41AM -0400, Robert Haas wrote: >> Another idea is to have a command that you can run, while connected to >> a particular database, that updates the default tablespace for that >> database without actually moving any data on disk - i.e. it sets >>

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-27 Thread Claudio Freire
On Wed, Aug 27, 2014 at 10:20 AM, Andres Freund wrote: > On 2014-08-27 10:17:06 -0300, Claudio Freire wrote: >> > I think a somewhat smarter version of the explicit flushes in the >> > hack^Wpatch I posted nearby is going to more likely to be successful. >> >> >> That path is "dangerous" (as in, m

Re: [HACKERS] ALTER SYSTEM RESET?

2014-08-27 Thread Fujii Masao
On Mon, Aug 25, 2014 at 1:34 PM, Amit Kapila wrote: > On Wed, Jul 30, 2014 at 9:11 AM, Amit Kapila > wrote: >> I have verified the patch and found that it works well for >> all scenario's. Few minor suggestions: >> >> 1. >> !values to the postgresql.auto.conf file. >> !Setting the parame

Re: [HACKERS] Hardening pg_upgrade

2014-08-27 Thread Robert Haas
On Mon, Aug 25, 2014 at 3:04 PM, Tom Lane wrote: > Bruce Momjian writes: >> OK, I will move in the direction of removing 8.3 support and use a >> single query to pull schema information. I was hesistant to remove 8.3 >> support as I know we have kept pg_dump support all the way back to 7.0, >>

Re: [HACKERS] Optimization for updating foreign tables in Postgres FDW

2014-08-27 Thread Robert Haas
On Mon, Aug 25, 2014 at 8:58 AM, Albe Laurenz wrote: > Reading the code, I noticed that the pushed down UPDATE or DELETE statement > is executed > during postgresBeginForeignScan rather than during postgresIterateForeignScan. > It probably does not matter, but is there a reason to do it different

Re: [HACKERS] Specifying the unit in storage parameter

2014-08-27 Thread Alvaro Herrera
Fujii Masao wrote: > On Tue, Aug 26, 2014 at 3:27 AM, Alvaro Herrera > wrote: > > Fujii Masao wrote: > >> On Thu, Aug 21, 2014 at 4:20 PM, Michael Paquier > >> wrote: > > > >> > Looking at the patch, the parameter "fillfactor" in the category > >> > RELOPT_KIND_HEAP (the first element in intRelOp

Re: [HACKERS] Optimization for updating foreign tables in Postgres FDW

2014-08-27 Thread Tom Lane
Robert Haas writes: > On Mon, Aug 25, 2014 at 8:58 AM, Albe Laurenz wrote: >> Reading the code, I noticed that the pushed down UPDATE or DELETE statement >> is executed >> during postgresBeginForeignScan rather than during >> postgresIterateForeignScan. >> It probably does not matter, but is th

Re: [HACKERS] replication commands and log_statements

2014-08-27 Thread Robert Haas
On Tue, Aug 26, 2014 at 7:17 AM, Fujii Masao wrote: > On Wed, Aug 20, 2014 at 1:14 PM, Michael Paquier > wrote: >> On Wed, Aug 20, 2014 at 2:06 AM, Robert Haas wrote: >>> >>> On Sat, Aug 16, 2014 at 10:27 AM, Amit Kapila >>> wrote: >>> > I think ideally it would have been better if we could hav

Re: [HACKERS] Optimization for updating foreign tables in Postgres FDW

2014-08-27 Thread Tom Lane
I wrote: > Robert Haas writes: >> Hmm, I'm worried that may be an API contract violation. > Indeed it is. You could get away with it if you check the > EXEC_FLAG_EXPLAIN_ONLY flag before doing anything with visible > side-effects, but it's still pretty ugly. Actually, there's another problem th

Re: [HACKERS] [RFC, POC] Don't require a NBuffer sized PrivateRefCount array of local buffer pins

2014-08-27 Thread Alvaro Herrera
Andres Freund wrote: > On 2014-08-26 22:19:47 -0400, Tom Lane wrote: > > Andres Freund writes: > > I would say that the issue most deserving of performance testing is your > > sizing of the linear-search array --- it's not obvious that 8 is a good > > size. > > It's about the size of a cacheline

Re: [HACKERS] Hardening pg_upgrade

2014-08-27 Thread Bruce Momjian
On Wed, Aug 27, 2014 at 09:54:11AM -0400, Robert Haas wrote: > Having said that, there are obviously advantages for our users if we > don't get too crazy about requiring that. I've used products in the > past where to get from version 3 to version 11 you have to upgrade > from 3 to 5, then 5 to 7,

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-27 Thread Aidan Van Dyk
On Wed, Aug 27, 2014 at 3:32 AM, Fabien COELHO wrote: > > Hello Andres, > > [...] >> >> I think you're misunderstanding how spread checkpoints work. >> > > Yep, definitely:-) On the other hand I though I was seeking something > "simple", namely correct latency under small load, that I would expe

Re: [HACKERS] pgbench throttling latency limit

2014-08-27 Thread Heikki Linnakangas
On 08/27/2014 02:37 PM, Fabien COELHO wrote: As for an actual "latency limit" under throttling, this is significantly more tricky and invasive to implement... ISTM that it would mean: - if the tx is not stated an the latency is already consummed, SKIP++. - if the tx is after its schedule

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-27 Thread Andres Freund
On 2014-08-27 10:32:19 -0400, Aidan Van Dyk wrote: > On Wed, Aug 27, 2014 at 3:32 AM, Fabien COELHO wrote: > > > > > Hello Andres, > > > > [...] > >> > >> I think you're misunderstanding how spread checkpoints work. > >> > > > > Yep, definitely:-) On the other hand I though I was seeking somethi

[HACKERS] Re: [BUGS] Re: BUG #9555: pg_dump for tables with inheritance recreates the table with the wrong order of columns

2014-08-27 Thread Bruce Momjian
On Mon, Mar 17, 2014 at 07:12:12PM -0400, Noah Misch wrote: > On Fri, Mar 14, 2014 at 12:33:04PM -0300, Alvaro Herrera wrote: > > Tom Lane wrote: > > > Alvaro Herrera writes: > > > > I wonder if the real fix here is to have ALTER / INHERIT error out of > > > > the columns in B are not a prefix of

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-27 Thread Fabien COELHO
Hello, If all you want is to avoid the write storms when fsyncs start happening on slow storage, can you not just adjust the kernel vm.dirty* tunables to start making the kernel write out dirty buffers much sooner instead of letting them accumulate until fsyncs force them out all at once? I c

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-08-27 Thread Robert Haas
On Tue, Aug 26, 2014 at 8:14 AM, Fujii Masao wrote: > On Tue, Aug 19, 2014 at 6:37 PM, Rahila Syed wrote: >> Hello, >> Thank you for comments. >> >>>Could you tell me where the patch for "single block in one run" is? >> Please find attached patch for single block compression in one run. > > Thank

Re: [HACKERS] Hardening pg_upgrade

2014-08-27 Thread Robert Haas
On Wed, Aug 27, 2014 at 10:13 AM, Bruce Momjian wrote: > On Wed, Aug 27, 2014 at 09:54:11AM -0400, Robert Haas wrote: >> Having said that, there are obviously advantages for our users if we >> don't get too crazy about requiring that. I've used products in the >> past where to get from version 3

Re: [HACKERS] Scaling shared buffer eviction

2014-08-27 Thread Robert Haas
On Tue, Aug 26, 2014 at 11:10 AM, Tom Lane wrote: > Amit Kapila writes: >> On Tue, Aug 5, 2014 at 9:21 PM, Robert Haas wrote: I think you should get rid of BufFreelistLock completely and just decide that freelist_lck will protect everything the freeNext links, plus everything in S

Re: [HACKERS] Scaling shared buffer eviction

2014-08-27 Thread Robert Haas
On Tue, Aug 26, 2014 at 10:53 AM, Amit Kapila wrote: > Today, while working on updating the patch to improve locking > I found that as now we are going to have a new process, we need > a separate latch in StrategyControl to wakeup that process. > Another point is I think it will be better to prote

Re: [HACKERS] pgbench throttling latency limit

2014-08-27 Thread Fabien COELHO
As for an actual "latency limit" under throttling, this is significantly more tricky and invasive to implement... ISTM that it would mean: [...] Yeah, something like that. I don't think it would be necessary to set statement_timeout, you can inject that in your script or postgresql.conf if

Re: [HACKERS] pgbench throttling latency limit

2014-08-27 Thread Heikki Linnakangas
On 08/27/2014 06:08 PM, Fabien COELHO wrote: I've submitted this "simple" lag limit version because being able to measure quickly and simply (un)responsiveness seems like a good idea, especially given the current state of things. Ok, fair enough. I don't think doing a "latency limit" would be s

Re: [HACKERS] [BUGS] Re: BUG #9555: pg_dump for tables with inheritance recreates the table with the wrong order of columns

2014-08-27 Thread Tom Lane
Bruce Momjian writes: > I looked at this issue from March and I think we need to do something. > In summary, the problem is that tables using inheritance can be dumped > and reloaded with columns in a different order from the original > cluster. Yeah ... this has been a well-understood issue for

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-08-27 Thread Arthur Silva
Em 26/08/2014 09:16, "Fujii Masao" escreveu: > > On Tue, Aug 19, 2014 at 6:37 PM, Rahila Syed wrote: > > Hello, > > Thank you for comments. > > > >>Could you tell me where the patch for "single block in one run" is? > > Please find attached patch for single block compression in one run. > > Thank

[HACKERS] Missing plpgsql.o Symbols on OS X

2014-08-27 Thread David E . Wheeler
Hackers, I’m trying to build Pavel’s plpgsql_check against the 9.4 beta on OS X 10.9, but get these errors: make gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -I/usr/lo

Re: [HACKERS] Similar to csvlog but not really, json logs?

2014-08-27 Thread Greg Sabino Mullane
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Stephen Frost wrote: > To try to clarify that a bit, as it comes across as rather opaque even > on my re-reading, consider a case where you can't have the > "credit_card_number" field ever exported to an audit or log file, but > you're required

Re: [HACKERS] SKIP LOCKED DATA (work in progress)

2014-08-27 Thread Alvaro Herrera
Thomas Munro wrote: > On 25 August 2014 02:57, Alvaro Herrera wrote: > > Thomas Munro wrote: > >> The difficulty of course will be testing all these racy cases > >> reproducibly... > > > > Does this help? > > http://www.postgresql.org/message-id/51fb4305.3070...@2ndquadrant.com > > The useful tri

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-27 Thread Heikki Linnakangas
On 08/27/2014 04:20 PM, Andres Freund wrote: On 2014-08-27 10:17:06 -0300, Claudio Freire wrote: I think a somewhat smarter version of the explicit flushes in the hack^Wpatch I posted nearby is going to more likely to be successful. That path is "dangerous" (as in, may not work as intended) i

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-27 Thread Andres Freund
On 2014-08-27 19:23:04 +0300, Heikki Linnakangas wrote: > On 08/27/2014 04:20 PM, Andres Freund wrote: > >On 2014-08-27 10:17:06 -0300, Claudio Freire wrote: > >>>I think a somewhat smarter version of the explicit flushes in the > >>>hack^Wpatch I posted nearby is going to more likely to be success

Re: [HACKERS] Similar to csvlog but not really, json logs?

2014-08-27 Thread Jim Nasby
On 8/26/14, 8:45 PM, Michael Paquier wrote: Hi all, As mentioned here, we support multiple logging format: http://www.postgresql.org/docs/devel/static/runtime-config-logging.html Now what about a json format logging with one json object per log entry? A single json entry would need more space t

Re: [HACKERS] delta relations in AFTER triggers

2014-08-27 Thread Jim Nasby
On 8/27/14, 2:23 AM, Heikki Linnakangas wrote: Does this make sense? In essence, make the relations work like PL/pgSQL variables do. If you squint a little, the new/old relation is a variable from the function's point of view, and a parameter from the planner/executor's point of view. It's jus

Re: [HACKERS] Similar to csvlog but not really, json logs?

2014-08-27 Thread Andres Freund
On 2014-08-26 23:04:48 -0500, Jim Nasby wrote: > On 8/26/14, 8:45 PM, Michael Paquier wrote: > >Hi all, > > > >As mentioned here, we support multiple logging format: > >http://www.postgresql.org/docs/devel/static/runtime-config-logging.html > >Now what about a json format logging with one json obje

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-27 Thread Fabien COELHO
off: $ pgbench -p 5440 -h /tmp postgres -M prepared -c 16 -j16 -T 120 -R 180 -L 200 number of skipped transactions: 1345 (6.246 %) on: $ pgbench -p 5440 -h /tmp postgres -M prepared -c 16 -j16 -T 120 -R 180 -L 200 number of skipped transactions: 1 (0.005 %) That machine is far from idle ri

Re: [HACKERS] SKIP LOCKED DATA (work in progress)

2014-08-27 Thread Thomas Munro
On 27 August 2014 17:18, Alvaro Herrera wrote: > Thomas Munro wrote: >> On 25 August 2014 02:57, Alvaro Herrera wrote: >> > Thomas Munro wrote: >> >> The difficulty of course will be testing all these racy cases >> >> reproducibly... >> > >> > Does this help? >> > http://www.postgresql.org/messa

Re: [HACKERS] pgbench throttling latency limit

2014-08-27 Thread Fabien COELHO
[...] Yeah, something like that. I don't think it would be necessary to set statement_timeout, you can inject that in your script or postgresql.conf if you want. I don't think aborting a transaction that's already started is necessary either. You could count it as LATE, but let it finish fi

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-27 Thread Andres Freund
On 2014-08-27 19:00:12 +0200, Fabien COELHO wrote: > > >off: > > > >$ pgbench -p 5440 -h /tmp postgres -M prepared -c 16 -j16 -T 120 -R 180 -L > >200 > >number of skipped transactions: 1345 (6.246 %) > > > >on: > > > >$ pgbench -p 5440 -h /tmp postgres -M prepared -c 16 -j16 -T 120 -R 180 -L > >

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-27 Thread Jim Nasby
On 8/26/14, 8:40 AM, Heikki Linnakangas wrote: Just so everyone is on the same page on what kind of queries this helps with, here are some examples from the added regression tests: -- Test join removals for semi and anti joins CREATE TEMP TABLE b (id INT NOT NULL PRIMARY KEY, val INT); CREATE

Re: [HACKERS] Similar to csvlog but not really, json logs?

2014-08-27 Thread Petr Jelinek
On 27/08/14 18:53, Andres Freund wrote: On 2014-08-26 23:04:48 -0500, Jim Nasby wrote: On 8/26/14, 8:45 PM, Michael Paquier wrote: Hi all, As mentioned here, we support multiple logging format: http://www.postgresql.org/docs/devel/static/runtime-config-logging.html Now what about a json format

Re: [HACKERS] delta relations in AFTER triggers

2014-08-27 Thread David Fetter
On Wed, Aug 27, 2014 at 11:51:40AM -0500, Jim Nasby wrote: > On 8/27/14, 2:23 AM, Heikki Linnakangas wrote: > >Does this make sense? In essence, make the relations work like > >PL/pgSQL variables do. If you squint a little, the new/old relation > >is a variable from the function's point of view, an

Re: [HACKERS] Parallel Sequence Scan doubts

2014-08-27 Thread Jim Nasby
On 8/24/14, 6:22 AM, Haribabu Kommi wrote: Yes, we are mainly targeting CPU-limited sequential scans, Because of this reason only I want the worker to handle the predicates also not just reading the tuples from disk. In that case, I would suggest focusing on parallel execution of conditions re

Re: [HACKERS] Function to know last log write timestamp

2014-08-27 Thread Jim Nasby
On 8/27/14, 7:33 AM, Fujii Masao wrote: On Tue, Aug 19, 2014 at 1:07 AM, Robert Haas wrote: On Fri, Aug 15, 2014 at 7:17 AM, Fujii Masao wrote: On Fri, Aug 15, 2014 at 3:40 AM, Andres Freund wrote: On 2014-08-14 14:37:22 -0400, Robert Haas wrote: On Thu, Aug 14, 2014 at 2:21 PM, Andres Fre

Re: [HACKERS] [RFC, POC] Don't require a NBuffer sized PrivateRefCount array of local buffer pins

2014-08-27 Thread Jim Nasby
On 8/27/14, 1:38 AM, Andres Freund wrote: It occurs to me that it'd also be nice to have some >stats available on how this is performing; perhaps a dtrace probe for >whenever we overflow to the hash table, and one that shows maximum >usage for a statement? (Presumably that's not much extra code o

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-27 Thread Tom Lane
Jim Nasby writes: > On 8/26/14, 8:40 AM, Heikki Linnakangas wrote: >> Just so everyone is on the same page on what kind of queries this helps >> with, here are some examples from the added regression tests: >> > -- Test join removals for semi and anti joins > CREATE TEMP TABLE b (id INT NOT NULL

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-27 Thread Arthur Silva
On Wed, Aug 27, 2014 at 1:09 AM, Arthur Silva wrote: > It won't be faster by any means, but it should definitely be incorporated > if any format changes are made (like Tom already suggested). > > I think it's important we gather at least 2 more things before making any > calls: > * Josh tests w/

[HACKERS] possible optimization: push down aggregates

2014-08-27 Thread Pavel Stehule
Hi one user asked about using a partitioning for faster aggregates queries. I found so there is not any optimization. create table x1(a int, d date); create table x_1 ( check(d = '2014-01-01'::date)) inherits(x1); create table x_2 ( check(d = '2014-01-02'::date)) inherits(x1); create table x_3 (

Re: [HACKERS] possible optimization: push down aggregates

2014-08-27 Thread Merlin Moncure
On Wed, Aug 27, 2014 at 2:07 PM, Pavel Stehule wrote: > Hi > > one user asked about using a partitioning for faster aggregates queries. > > I found so there is not any optimization. > > create table x1(a int, d date); > create table x_1 ( check(d = '2014-01-01'::date)) inherits(x1); > create table

Re: [HACKERS] possible optimization: push down aggregates

2014-08-27 Thread Claudio Freire
On Wed, Aug 27, 2014 at 4:41 PM, Merlin Moncure wrote: > On Wed, Aug 27, 2014 at 2:07 PM, Pavel Stehule > wrote: >> Hi >> >> one user asked about using a partitioning for faster aggregates queries. >> >> I found so there is not any optimization. >> >> create table x1(a int, d date); >> create ta

Re: [HACKERS] possible optimization: push down aggregates

2014-08-27 Thread Pavel Stehule
2014-08-27 21:41 GMT+02:00 Merlin Moncure : > On Wed, Aug 27, 2014 at 2:07 PM, Pavel Stehule > wrote: > > Hi > > > > one user asked about using a partitioning for faster aggregates queries. > > > > I found so there is not any optimization. > > > > create table x1(a int, d date); > > create table

Re: [HACKERS] possible optimization: push down aggregates

2014-08-27 Thread Pavel Stehule
2014-08-27 21:46 GMT+02:00 Claudio Freire : > On Wed, Aug 27, 2014 at 4:41 PM, Merlin Moncure > wrote: > > On Wed, Aug 27, 2014 at 2:07 PM, Pavel Stehule > wrote: > >> Hi > >> > >> one user asked about using a partitioning for faster aggregates queries. > >> > >> I found so there is not any opti

Re: [HACKERS] possible optimization: push down aggregates

2014-08-27 Thread Merlin Moncure
On Wed, Aug 27, 2014 at 2:46 PM, Claudio Freire wrote: > On Wed, Aug 27, 2014 at 4:41 PM, Merlin Moncure wrote: >> On Wed, Aug 27, 2014 at 2:07 PM, Pavel Stehule >> wrote: >>> Hi >>> >>> one user asked about using a partitioning for faster aggregates queries. >>> >>> I found so there is not any

Re: [HACKERS] SKIP LOCKED DATA (work in progress)

2014-08-27 Thread Alvaro Herrera
Thomas Munro wrote: > On 27 August 2014 17:18, Alvaro Herrera wrote: > > Thomas Munro wrote: > >> Yes it does, thanks Alvaro and Craig. I think the attached spec > >> reproduces the problem using that trick, ie shows NOWAIT blocking, > >> presumably in EvalPlanQualFetch (though I haven't stepped

Re: [HACKERS] possible optimization: push down aggregates

2014-08-27 Thread Tom Lane
Merlin Moncure writes: > associative bit just makes it easier (which is important of course!). > mean for example can be pushed down if the 'pushed down' aggregates > return to the count to the "reaggregator" so that you can weight the > final average. that's a lot more complicated though. The r

Re: [Fwd: Re: [HACKERS] proposal: new long psql parameter --on-error-stop]

2014-08-27 Thread Pavel Stehule
Hi I chose \? xxx, because it is related to psql features. I wrote commands: \? options \? variables comments? Regards Pavel 2014-08-26 13:48 GMT+02:00 Andres Freund : > On 2014-08-26 13:44:16 +0200, Pavel Stehule wrote: > > 2014-08-26 13:30 GMT+02:00 Petr Jelinek : > > > > > On 26/08/14 1

Re: [HACKERS] possible optimization: push down aggregates

2014-08-27 Thread Pavel Stehule
2014-08-27 22:27 GMT+02:00 Tom Lane : > Merlin Moncure writes: > > associative bit just makes it easier (which is important of course!). > > mean for example can be pushed down if the 'pushed down' aggregates > > return to the count to the "reaggregator" so that you can weight the > > final avera

Re: [HACKERS] delta relations in AFTER triggers

2014-08-27 Thread Kevin Grittner
Heikki Linnakangas wrote: > On 08/27/2014 02:26 AM, Kevin Grittner wrote: >> spi-tuplestore-registry allows tuplestores, with associated name >> and TupleDesc, to be registered with the current SPI connection. >> Queries planned or executed on that connection will recognize the >> name as a tuple

Re: [HACKERS] delta relations in AFTER triggers

2014-08-27 Thread Kevin Grittner
Jim Nasby wrote: > Something to keep in mind is that users will definitely think about NEW/OLD as > tables. I suspect that it won't be long after release before someone asks > why they can't create an index on it. :) I'm comfortable saying "No" to that.  But it's a good point -- I'll review err

Re: [HACKERS] delta relations in AFTER triggers

2014-08-27 Thread Stephen Frost
* Kevin Grittner (kgri...@ymail.com) wrote: > Heikki Linnakangas wrote: > > In essence, make the relations work like PL/pgSQL > > variables do. If you squint a little, the new/old relation is a variable > > from the function's point of view, and a parameter from the > > planner/executor's point of

Re: [HACKERS] delta relations in AFTER triggers

2014-08-27 Thread Heikki Linnakangas
On 08/28/2014 12:03 AM, Kevin Grittner wrote: Heikki Linnakangas wrote: I suggest adding a new hook to the ParseState struct, (p_rangevar_hook ?). The planner calls it whenever it sees a reference to a table, and the hook function returns back some sort of placeholder reference to the tuplestor

Re: [HACKERS] possible optimization: push down aggregates

2014-08-27 Thread Tomas Vondra
On 27 Srpen 2014, 21:41, Merlin Moncure wrote: > On Wed, Aug 27, 2014 at 2:07 PM, Pavel Stehule >> >> Are there some plans to use partitioning for aggregation? > > Besides min/max, what other aggregates (mean/stddev come to mind) > would you optimize and how would you determine which ones could be

Re: [HACKERS] possible optimization: push down aggregates

2014-08-27 Thread Merlin Moncure
On Wed, Aug 27, 2014 at 3:27 PM, Tom Lane wrote: > Merlin Moncure writes: >> associative bit just makes it easier (which is important of course!). >> mean for example can be pushed down if the 'pushed down' aggregates >> return to the count to the "reaggregator" so that you can weight the >> fina

Re: [HACKERS] possible optimization: push down aggregates

2014-08-27 Thread Claudio Freire
On Wed, Aug 27, 2014 at 6:46 PM, Merlin Moncure wrote: > > Yeah: I was overthinking it. My mind was on parallel processing of > the aggregate (which is not what Pavel was proposing) because that > just happens to be what I'm working on currently -- using dblink to > decompose various aggregates

Re: [HACKERS] [TODO] Track number of files ready to be archived in pg_stat_archiver

2014-08-27 Thread Julien Rouhaud
Le 25/08/2014 19:00, Gilles Darold a écrit : > Le 21/08/2014 10:17, Julien Rouhaud a écrit : >> Hello, >> >> Attached patch implements the following TODO item : >> >> Track number of WAL files ready to be archived in pg_stat_archiver >> >> However, it will track the total number of any file ready t

Re: [HACKERS] Missing plpgsql.o Symbols on OS X

2014-08-27 Thread Tom Lane
"David E. Wheeler" writes: > Hackers, > I’m trying to build Pavel’s plpgsql_check against the 9.4 beta on OS X 10.9, > but get these errors: > make > gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith > -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute > -Wformat-security

Re: [HACKERS] FOR [SHARE|UPDATE] NOWAIT may still block in EvalPlanQualFetch

2014-08-27 Thread Alvaro Herrera
Craig Ringer wrote: > FOR SHARE|UPDATE NOWAIT will still block if they have to follow a ctid > chain because the call to EvalPlanQualFetch doesn't take a param for > noWait, so it doesn't know not to block if the updated row can't be locked. Applied with some further editorialization. In another

Re: [HACKERS] SKIP LOCKED DATA (work in progress)

2014-08-27 Thread Alvaro Herrera
Thomas Munro wrote: > Thanks, I hadn't seen this, I should have checked the archives better. > I have actually already updated my patch to handle EvalPlanQualFetch > with NOWAIT and SKIP LOCKED with isolation specs, see attached. I > will compare with Craig's and see if I screwed anything up... o

Re: [HACKERS] re-reading SSL certificates during server reload

2014-08-27 Thread Robert Haas
On Wed, Aug 27, 2014 at 6:40 AM, Magnus Hagander wrote: > On Wed, Aug 27, 2014 at 11:56 AM, Alexey Klyukin wrote: >> Greetings, >> >> Is there a strong reason to disallow reloading server key and cert files >> during the PostgreSQL reload? > > Key and cert files are loaded in the postmaster. We'd

  1   2   >