Re: [HACKERS] Protect syscache from bloating with negative cache entries

2017-01-24 Thread Kyotaro HORIGUCHI
Hello, thank you for lookin this. At Mon, 23 Jan 2017 16:54:36 -0600, Jim Nasby wrote in <21803f50-a823-c444-ee2b-9a153114f...@bluetreble.com> > On 1/21/17 6:42 PM, Jim Nasby wrote: > > On 12/26/16 2:31 AM, Kyotaro HORIGUCHI wrote: > >> The points of discussion are the following, I think. > >> >

Re: [HACKERS] Failure in commit_ts tap tests

2017-01-24 Thread Andrew Dunstan
On 01/23/2017 09:22 AM, Tom Lane wrote: > Andrew Dunstan writes: >> On 01/23/2017 09:03 AM, Tom Lane wrote: >>> Andrew Dunstan writes: On 01/20/2017 01:22 PM, Tom Lane wrote: > It looks like at least part of the answer is that the buildfarm isn't > running this test. AFAICS, it ru

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-24 Thread Mithun Cy
On Tue, Jan 24, 2017 at 5:07 AM, Jim Nasby wrote: > I took a look at this again, and it doesn't appear to be working for me. The > library is being loaded during startup, but I don't see any further activity > in the log, and I don't see an autoprewarm file in $PGDATA. Hi Jim, Thanks for lookin

Re: [HACKERS] WIP: About CMake v2

2017-01-24 Thread Craig Ringer
On 24 Jan. 2017 18:00, "Andres Freund" wrote: On 2017-01-24 15:50:47 +0900, Michael Paquier wrote: > I am marking this patch as "returned with feedback". That's quite a > heavy change and it looks to be too late in the development cycle of > PG10 to consider it. Peter's commit bits, who is also t

Re: [HACKERS] Protect syscache from bloating with negative cache entries

2017-01-24 Thread Kyotaro HORIGUCHI
Hello, I have tried to cap the number of negative entries for myself (by removing negative entries in least recentrly created first order) but the ceils cannot be reasonably determined both absolutely or relatively to positive entries. Apparently it differs widely among caches and applications. A

Re: \if, \elseif, \else, \endif (was Re: [HACKERS] PSQL commands: \quit_if, \quit_unless)

2017-01-24 Thread Fabien COELHO
I would suggest to assume false on everything else, and/or maybe to ignore the whole if/endif section in such cases. +1, it also halves the number of values we have to support later. After giving it some thought, I revise a little bit my opinion: I think that if the value is evaluated to TR

Re: [HACKERS] [PROPOSAL] Temporal query processing with range types

2017-01-24 Thread Peter Moser
2017-01-18 3:57 GMT+01:00 Peter Eisentraut : > > On 1/13/17 9:22 AM, Peter Moser wrote: > > The goal of temporal aligners and normalizers is to split ranges to allow a > > reduction from temporal queries to their non-temporal counterparts. > > Splitting > > ranges is necessary for temporal query pr

Re: [HACKERS] Cache Hash Index meta page.

2017-01-24 Thread Amit Kapila
On Wed, Jan 18, 2017 at 11:51 AM, Mithun Cy wrote: > On Tue, Jan 17, 2017 at 10:07 AM, Amit Kapila wrote: > >> (b) Another somewhat bigger problem is that with this new change it >> won't retain the pin on meta page till the end which means we might >> need to perform an I/O again during operatio

Re: [HACKERS] parallelize queries containing subplans

2017-01-24 Thread Amit Kapila
On Tue, Jan 24, 2017 at 10:59 AM, Amit Kapila wrote: > On Thu, Jan 19, 2017 at 4:51 PM, Dilip Kumar wrote: >> On Thu, Jan 19, 2017 at 3:05 PM, Dilip Kumar wrote: >>> During debugging I found that subplan created for below part of the >>> query is parallel_unsafe, Is it a problem or there is some

Re: [HACKERS] Failure in commit_ts tap tests

2017-01-24 Thread Andrew Dunstan
On 01/24/2017 03:18 AM, Andrew Dunstan wrote: > > On 01/23/2017 09:22 AM, Tom Lane wrote: >> Andrew Dunstan writes: >>> On 01/23/2017 09:03 AM, Tom Lane wrote: Andrew Dunstan writes: > On 01/20/2017 01:22 PM, Tom Lane wrote: >> It looks like at least part of the answer is that the

Re: [HACKERS] Failure in commit_ts tap tests

2017-01-24 Thread Alvaro Herrera
Andrew Dunstan wrote: > Well, that crashed. The DDL deparse comment test should not assume that > contrib_regression exists. Possibly it should create it if it doesn't > exist. The buildfarm mostly runs with USE_DB_MODULE=1 so we don't > constantly overwrite the same db. Maybe we can drop that li

Re: [HACKERS] Floating point comparison inconsistencies of the geometric types

2017-01-24 Thread Kyotaro HORIGUCHI
Sorry, this is the continuation of the previous comment. At Wed, 18 Jan 2017 17:36:12 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170118.173612.13506164.horiguchi.kyot...@lab.ntt.co.jp> > Hello. > > (I apologize in advance for possible inaccurate wording on maths, > which might

Re: [HACKERS] [PATCH] Rename pg_switch_xlog to pg_switch_wal

2017-01-24 Thread Vladimir Rusinov
On Mon, Jan 23, 2017 at 6:59 PM, Stephen Frost wrote: > > For the record, I don't like the name "xlog" either. It would be nice > > if we could have more consistent and intuitive naming. > > > > But I don't see any proposals to actually change all uses of "xlog" to > > "wal". What about program

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-24 Thread Beena Emerson
On Tue, Jan 24, 2017 at 1:56 PM, Mithun Cy wrote: > On Tue, Jan 24, 2017 at 5:07 AM, Jim Nasby > wrote: > > I took a look at this again, and it doesn't appear to be working for me. > The library is being loaded during startup, but I don't see any further > activity in the log, and I don't see an

Re: [HACKERS] Speedup twophase transactions

2017-01-24 Thread Nikhil Sontakke
>> Speeding up recovery or failover activity via a faster promote is a >> desirable thing. So, maybe, we should look at teaching the relevant >> code about using "KnownPreparedList"? I know that would increase the >> size of this patch and would mean more testing, but this seems to be >> last remai

Re: [HACKERS] Review: GIN non-intrusive vacuum of posting tree

2017-01-24 Thread Andrew Borodin
Hello! I see your point on alternatives. I will do my best to generate more ideas on how vacuum can be done withing existing index structure, this will take a day or two. In this message I'll answer some questions. 2017-01-23 22:45 GMT+05:00 Jeff Davis : > 1. I haven't seen the approach before of

Re: [HACKERS] Add pgstathashindex() to get hash index table statistics.

2017-01-24 Thread Kuntal Ghosh
On Tue, Jan 24, 2017 at 10:09 AM, Ashutosh Sharma wrote: >> I've looked at the patch. It looks good. However, I was wondering why >> an exclusive lock for extension is necessary for reading the number >> blocks in this case. Please refer to the following code. >> >> + /* Get the current rela

Re: [HACKERS] Failure in commit_ts tap tests

2017-01-24 Thread Andrew Dunstan
On 01/24/2017 05:17 AM, Alvaro Herrera wrote: > Andrew Dunstan wrote: > >> Well, that crashed. The DDL deparse comment test should not assume that >> contrib_regression exists. Possibly it should create it if it doesn't >> exist. The buildfarm mostly runs with USE_DB_MODULE=1 so we don't >> const

[HACKERS] Superowners

2017-01-24 Thread Simon Riggs
So I was thinking about various annoying admin/security issues recently, so I came up with this: a new type of user called a “superowner”. It’s somewhere between a superuser and a normal user. Superowner would own all objects defined by users, so it would do useful things in contexts where superu

Re: [HACKERS] Speedup twophase transactions

2017-01-24 Thread Stas Kelvich
> On 24 Jan 2017, at 09:42, Michael Paquier wrote: > > On Mon, Jan 23, 2017 at 9:00 PM, Nikhil Sontakke > wrote: >> Speeding up recovery or failover activity via a faster promote is a >> desirable thing. So, maybe, we should look at teaching the relevant >> code about using "KnownPreparedList"?

Re: [HACKERS] patch proposal

2017-01-24 Thread David Steele
On 1/23/17 10:52 PM, Michael Paquier wrote: > On Tue, Jan 24, 2017 at 7:22 AM, David Steele wrote: >> 3) There are no tests. I believe that >> src/test/recovery/t/006_logical_decoding.pl would be the best place to >> insert your tests. > > 003_recovery_targets.pl is the test for recovery targets

Re: [HACKERS] Checksums by default?

2017-01-24 Thread Ants Aasma
On Tue, Jan 24, 2017 at 4:07 AM, Tom Lane wrote: > Peter Geoghegan writes: >> I thought that checksums went in in part because we thought that there >> was some chance that they'd find bugs in Postgres. > > Not really. AFAICS the only point is to catch storage-system malfeasance. This matches m

Re: [HACKERS] Failure in commit_ts tap tests

2017-01-24 Thread Tom Lane
Andrew Dunstan writes: > On 01/24/2017 05:17 AM, Alvaro Herrera wrote: >> Maybe we can drop that line and put it back once we get COMMENT ON >> CURRENT_DATABASE. > Works for me. If that's enough to get the "make check" cases passing in the buildfarm, then +1. regards, to

Re: [HACKERS] Superowners

2017-01-24 Thread Tom Lane
Simon Riggs writes: > So I was thinking about various annoying admin/security issues > recently, so I came up with this: a new type of user called a > “superowner”. It’s somewhere between a superuser and a normal user. > Superowner would own all objects defined by users, so it would do > useful

Re: [HACKERS] Checksums by default?

2017-01-24 Thread Stephen Frost
Greetings, * Ants Aasma (ants.aa...@eesti.ee) wrote: > On Tue, Jan 24, 2017 at 4:07 AM, Tom Lane wrote: > > Peter Geoghegan writes: > >> I thought that checksums went in in part because we thought that there > >> was some chance that they'd find bugs in Postgres. > > > > Not really. AFAICS the

Re: [HACKERS] Assignment of valid collation for SET operations on queries with UNKNOWN types.

2017-01-24 Thread Ashutosh Bapat
On Mon, Jan 23, 2017 at 9:56 PM, Tom Lane wrote: > I wrote: >> Ashutosh Bapat writes: >>> UNKNOWN is not exactly a pseudo-type. > >> Well, as I said to Michael just now, I think we should turn it into one >> now that we're disallowing it in tables, because "cannot be used as a >> table column" is

Re: [HACKERS] Online enabling of page level checksums

2017-01-24 Thread David Steele
On 1/23/17 9:45 PM, Jim Nasby wrote: > To put it another way, I think it's entirely reasonable *from a > technical standpoint* to enable by default in 10, with only the ability > to dynamically disable. Given the concerns that keep popping up about > dynamically enabling, I'm not at all sure that

Re: [HACKERS] WIP: About CMake v2

2017-01-24 Thread Tom Lane
Craig Ringer writes: > Personally I think we should aim to have this in as a non default build > mode in pg10 if it can be made ready, and aim to make it default in pg11 at > least for Windows. AFAIK we haven't committed to accepting this at all, let alone trying to do so on a tight schedule. An

Re: [HACKERS] Declarative partitioning vs. BulkInsertState

2017-01-24 Thread Robert Haas
On Mon, Jan 23, 2017 at 5:25 AM, Amit Langote wrote: > I tried implementing the second idea in the attached patch. It fixes the > bug (multiple reports as mentioned in the commit message) that tuples may > be inserted into the wrong partition. Looks good to me, thanks. Committed with a few twea

[HACKERS] PostgreSQL 8.2 installation error on Windows 2016 server

2017-01-24 Thread Shruti Rawal
Hi, We are doing an assessment for for migrating our Perl applications to Windows 2016 server. I am trying to install PostgreSQL 8.2 version on my Windows server 2016. But it is giving me following error: Malformed permissions property: 'langid' We could not find any relavant information on Pos

Re: [HACKERS] Valgrind-detected bug in partitioning code

2017-01-24 Thread Robert Haas
On Mon, Jan 23, 2017 at 12:45 AM, Amit Langote wrote: > Sorry for jumping in late. Attached patch replaces the call to > partitioning-specific comparison function by the call to datumIsEqual(). > I wonder if it is safe to assume that datumIsEqual() would return true for > a datum and copy of it m

Re: [HACKERS] [PATCH] Generic type subscription

2017-01-24 Thread Tom Lane
I wrote: > BTW, a different approach that might be worth considering is to say that > the nodetree representation of one of these things *is* a FuncExpr, and > the new magic thing is just that we invent a new CoercionForm value > which causes ruleutils.c to print the expression as a subscripting op

Re: [HACKERS] Superowners

2017-01-24 Thread Simon Riggs
On 24 January 2017 at 13:19, Tom Lane wrote: > Simon Riggs writes: >> So I was thinking about various annoying admin/security issues >> recently, so I came up with this: a new type of user called a >> “superowner”. It’s somewhere between a superuser and a normal user. >> Superowner would own al

Re: [HACKERS] pg_hba_file_settings view patch

2017-01-24 Thread Ashutosh Bapat
On Mon, Jan 23, 2017 at 1:43 PM, Haribabu Kommi wrote: > > > On Sat, Jan 21, 2017 at 8:01 AM, Tom Lane wrote: >> >> Haribabu Kommi writes: >> > [ pg_hba_rules_10.patch ] >> >> I took a quick look over this. > > > Thanks for the review. > >> >> * I'm not exactly convinced that the way you approac

Re: [HACKERS] Superowners

2017-01-24 Thread Stephen Frost
Simon, * Simon Riggs (si...@2ndquadrant.com) wrote: > So I was thinking about various annoying admin/security issues > recently, so I came up with this: a new type of user called a > “superowner”. It’s somewhere between a superuser and a normal user. I like the general idea, but I'm not really

Re: [HACKERS] Speedup twophase transactions

2017-01-24 Thread Nikhil Sontakke
> Thanks for review, Nikhil and Michael. > > I don’t follow here. We are moving data away from WAL to files on checkpoint > because after checkpoint > there is no guaranty that WAL segment with our prepared tx will be still > available. > We are talking about the recovery/promote code path. Spec

[HACKERS] Deadlock in XLogInsert at AIX

2017-01-24 Thread Konstantin Knizhnik
Hi Hackers, We are running Postgres at AIX and encoountered two strqange problems: active zombies process and deadlock in XLOG writer. First problem I will explain in separate mail, now I am mostly concerning about deadlock. It is irregularly reproduced with standard pgbench launched with 100

Re: [HACKERS] PostgreSQL 8.2 installation error on Windows 2016 server

2017-01-24 Thread Aaron W. Swenson
On 2017-01-24 09:20, Shruti Rawal wrote: > Hi, > > We are doing an assessment for for migrating our Perl applications to Windows > 2016 server. > I am trying to install PostgreSQL 8.2 version on my Windows server 2016. But > it is giving me following error: > Malformed permissions property: 'lan

Re: [HACKERS] Assignment of valid collation for SET operations on queries with UNKNOWN types.

2017-01-24 Thread Tom Lane
Michael Paquier writes: > As unknown is a pseudo type, I don't think you need > TYPCATEGORY_UNKNOWN in pg_type.h or even the mention to the unknown > type in catalogs.sgml as that becomes a pseudo-type. I wondered whether to remove TYPCATEGORY_UNKNOWN but thought it was an unnecessary change. "u

Re: [HACKERS] Assignment of valid collation for SET operations on queries with UNKNOWN types.

2017-01-24 Thread Tom Lane
Ashutosh Bapat writes: > On Mon, Jan 23, 2017 at 9:56 PM, Tom Lane wrote: >> I've grepped the code for references to UNKNOWNOID and TYPTYPE_PSEUDO, >> and I can't find any places where the behavior would change in a way >> that we don't want. Basically it looks like we'd disallow UNKNOWN as >> a

Re: [HACKERS] [COMMITTERS] pgsql: Add pg_sequence system catalog

2017-01-24 Thread Peter Eisentraut
On 1/19/17 11:03 AM, Stephen Frost wrote: > I'd suggest using our usual approach in pg_dump, which is matching based > on the OID, like so: > > WHERE c.oid = '%u'::oid > > The OID is in: tbinfo->dobj.catId.oid > > Also, you should move the selectSourceSchema() into the per-version > branches and

Re: [HACKERS] [PATCH] Change missleading comment for tm_mon field of pg_tm structure

2017-01-24 Thread Robert Haas
On Thu, Jan 19, 2017 at 2:52 PM, Dmitry Fedin wrote: > According to usages, tm_mon (month) field has origin 1, not 0. This is > difference from C-library version of struct tm, which has origin 0 for > real. Hmm, thanks for noticing. Looks like it has been wrong for a really long time. Your patc

[HACKERS] Active zombies at AIX

2017-01-24 Thread Konstantin Knizhnik
Hi hackers, Yet another story about AIX. For some reasons AIX very slowly cleaning zombie processes. If we launch pgbench with -C parameter then very soon limit for maximal number of connections is exhausted. If maximal number of connection is set to 1000, then after ten seconds of pgbench act

Re: [HACKERS] [COMMITTERS] pgsql: Add pg_sequence system catalog

2017-01-24 Thread Stephen Frost
Peter, * Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote: > On 1/19/17 11:03 AM, Stephen Frost wrote: > > I'd suggest using our usual approach in pg_dump, which is matching based > > on the OID, like so: > > > > WHERE c.oid = '%u'::oid > > > > The OID is in: tbinfo->dobj.catId.oid > >

Re: [HACKERS] Active zombies at AIX

2017-01-24 Thread Tom Lane
Konstantin Knizhnik writes: > But ps shows that status of process is > [14:46:02]root@postgres:~ # ps -elk | grep 25691556 > * A - 25691556 - - - - - As far as I could find by googling, this means that the process is not actually a zombie yet, so it's not the postmaster's fault. Apparently i

Re: [HACKERS] Active zombies at AIX

2017-01-24 Thread Konstantin Knizhnik
On 24.01.2017 18:26, Tom Lane wrote: Konstantin Knizhnik writes: But ps shows that status of process is [14:46:02]root@postgres:~ # ps -elk | grep 25691556 * A - 25691556 - - - - - As far as I could find by googling, this means that the process is not actually a zombie yet, so it's not

Re: [HACKERS] Assignment of valid collation for SET operations on queries with UNKNOWN types.

2017-01-24 Thread Tom Lane
I wrote: > Michael Paquier writes: >> The table of Pseudo-Types needs to be updated as well with unknown in >> datatype.sgml. > Check. Here's an updated patch with doc changes. Aside from that one, I tried to spell "pseudo-type" consistently, and I changed one place where we were calling someth

Re: [HACKERS] [PATCH] Transaction traceability - txid_status(bigint)

2017-01-24 Thread Robert Haas
On Mon, Jan 23, 2017 at 1:32 AM, Craig Ringer wrote: > Rebased patch attached. I've split the clog changes out from > txid_status() its self. I think it's fairly surprising that TruncateCLOG() has responsibility for writing the xlog record that protects advancing ShmemVariableCache->oldestXid, bu

Re: [HACKERS] [COMMITTERS] pgsql: Reindent table partitioning code.

2017-01-24 Thread Robert Haas
On Tue, Jan 24, 2017 at 10:28 AM, Tom Lane wrote: > Robert Haas writes: >> Reindent table partitioning code. > > Oh, thank you, I was starting to get annoyed with that too. Glad you like. The pgindent damage introduced by the logical replication commits is even more extensive, but I didn't want

Re: [HACKERS] Checksums by default?

2017-01-24 Thread Torsten Zuehlsdorff
On 21.01.2017 19:37, Stephen Frost wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: Stephen Frost writes: Because I see having checksums as, frankly, something we always should have had (as most other databases do, for good reason...) and because they will hopefully prevent data loss. I'm will

Re: [HACKERS] Checksums by default?

2017-01-24 Thread Torsten Zuehlsdorff
On 21.01.2017 19:35, Tom Lane wrote: Andres Freund writes: Sure, it might be easy, but we don't have it. Personally I think checksums just aren't even ready for prime time. If we had: - ability to switch on/off at runtime (early patches for that have IIRC been posted) - *builtin* tooling to

Re: [HACKERS] [ patch ] pg_dump: new --custom-fetch-table and --custom-fetch-value parameters

2017-01-24 Thread Robert Haas
On Fri, Jan 20, 2017 at 12:52 AM, Andrea Urbani wrote: > I have used "custom" parameters because I want to decrease the fetch size > only on the tables with big bloab fields. If we remove the > "custom-fetch-table" parameter and we provide only the "fetch-size" parameter > all the tables will u

Re: [HACKERS] Deadlock in XLogInsert at AIX

2017-01-24 Thread Konstantin Knizhnik
More information about the problem - Postgres log contains several records: 2017-01-24 19:15:20.272 MSK [19270462] LOG: request to flush past end of generated WAL; request 6/AAEBE000, currpos 6/AAEBC2B0 and them correspond to the time when deadlock happen. There is the following comment in xl

Re: [HACKERS] logical-replication.sgml improvements

2017-01-24 Thread Robert Haas
On Fri, Jan 20, 2017 at 11:00 AM, Erik Rijkers wrote: > logical-replication.sgml.diff changes I had a look at this but these are not all open-and-shut cases. In many cases I think your proposed wording is better, but it's probably a good idea for the people who were involved in the development o

Re: [HACKERS] Checksums by default?

2017-01-24 Thread Joshua D. Drake
On 01/21/2017 09:09 AM, Tom Lane wrote: Stephen Frost writes: As for checksums, I do see value in them and I'm pretty sure that the author of that particular feature did as well, or we wouldn't even have it as an option. You seem to be of the opinion that we might as well just rip all of that

Re: [HACKERS] pgbench more operators & functions

2017-01-24 Thread Robert Haas
On Fri, Dec 2, 2016 at 1:28 AM, Fabien COELHO wrote: >> Closed in 2016-11 commitfest with "returned with feedback" status. >> Please feel free to update the status once you submit the updated patch. > > Given the thread discussions, I do not understand why this "ready for > committer" patch is swi

Re: [HACKERS] pgbench more operators & functions

2017-01-24 Thread Robert Haas
On Tue, Jan 24, 2017 at 11:32 AM, Robert Haas wrote: > On Fri, Dec 2, 2016 at 1:28 AM, Fabien COELHO wrote: >>> Closed in 2016-11 commitfest with "returned with feedback" status. >>> Please feel free to update the status once you submit the updated patch. >> >> Given the thread discussions, I do

Re: [HACKERS] logical-replication.sgml improvements

2017-01-24 Thread Erik Rijkers
On 2017-01-24 17:29, Robert Haas wrote: On Fri, Jan 20, 2017 at 11:00 AM, Erik Rijkers wrote: logical-replication.sgml.diff changes I had a look at this but these are not all open-and-shut cases. In many cases I think your proposed wording is better, but it's probably a good idea for the peo

Re: [HACKERS] Improvements in psql hooks for variables

2017-01-24 Thread Daniel Verite
Tom Lane wrote: > However, it only really works if psql never overwrites the values > after startup, whereas I believe all of these are overwritten by > a \c command. Yes, there are reset to reflect the properties of the new connection. > So maybe it's more user-friendly to make these va

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-24 Thread Mithun Cy
On Fri, Oct 28, 2016 at 6:36 AM, Tsunakawa, Takayuki wrote: > I welcome this feature! I remember pg_hibernate did this. I wonder what > happened to pg_hibernate. Did you check it? Thanks, when I checked with pg_hibernate there were two things people complained about it. Buffer loading will st

Re: [HACKERS] Improvements in psql hooks for variables

2017-01-24 Thread Daniel Verite
Tom Lane wrote: > I took a quick look through this. It seems to be going in generally > the right direction, but here's a couple of thoughts: Here's an update with these changes: per Tom's suggestions upthread: - change ParseVariableBool() signature to return validity as bool. - remov

Re: [HACKERS] Improvements in psql hooks for variables

2017-01-24 Thread Daniel Verite
I just wrote: > - add enum-style suggestions on invalid input for \pset x, \pset pager, > and \set of ECHO, ECHO_HIDDEN, ON_ERROR_ROLLBACK, COMP_KEYWORD_CASE, > HISTCONTROL, VERBOSITY, SHOW_CONTEXT, \x, \pager There's no such thing as \pager, I meant to write: \pset x, \pset pager, and \

[HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-24 Thread Tobias Oberstein
Hi guys, pls bare with me, this is my first post here. Pls also excuse the length .. I was trying to do all my homework before posting here;) The overhead of lseek/read/write vs pread/pwrite (or even pvread/pvwrite) was previously discussed here https://www.postgresql.org/message-id/flat/CA

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-24 Thread Andres Freund
Hi, On 2017-01-24 18:11:09 +0100, Tobias Oberstein wrote: > I have done lots of benchmarking over the last days on a massive box, and I > can provide numbers that I think show that the impact can be significant. > Above number was using psync FIO engine .. with libaio, it's at 9.7 mio with > muc

Re: [HACKERS] Performance improvement for joins where outer side is unique

2017-01-24 Thread Tom Lane
[ getting back to this at long last ] David Rowley writes: > However, having said that, I'm not sure why we'd need outer_unique > available so we'd know that we could skip mark/restore. I think > inner_unique is enough for this purpose. Take the comment from > nodeMergejoin.c: > * outer: (0 ^1 1

Re: [HACKERS] Review: GIN non-intrusive vacuum of posting tree

2017-01-24 Thread Jeff Davis
On Tue, Jan 24, 2017 at 3:18 AM, Andrew Borodin wrote: > Technically, approach of locking a subtree is not novel. Lehman and > Yao focused on "that any process for manipulating the tree uses only a > small (constant) number of locks at any time." We are locking unknown > and possibly large amount

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-24 Thread Tobias Oberstein
Hi, Switching to sync engine, it drops to 9.1 mio - but the system load then is also much higher! I doubt those have very much to do with postgres - I'd quite strongly In the machine in production, we see 8kB reads in the 300k-650k/s range. In spikes, because, yes, due to the 3TB RAM, we ha

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-24 Thread Andres Freund
Hi, On 2017-01-24 18:37:14 +0100, Tobias Oberstein wrote: > > assume that it'd get more than swamped with doing actualy work, and with > > buffering the frequently accessed stuff in memory. > > > > > > > What I am trying to say is: the syscall overhead of doing lseek/read/write > > > instead of

Re: [HACKERS] ICU integration

2017-01-24 Thread Peter Eisentraut
On 1/9/17 3:45 PM, Peter Geoghegan wrote: > * I think it's worth looking into ucol_nextSortKeyPart(), and using > that as an alternative to ucol_getSortKey(). It doesn't seem any > harder, and when I tested it it was clearly faster. (I think that > ucol_nextSortKeyPart() is more or less intended to

Re: [HACKERS] Declarative partitioning - another take

2017-01-24 Thread Robert Haas
On Thu, Jan 19, 2017 at 9:58 PM, Amit Langote wrote: >> But I wonder why we don't instead just change this function to >> consider tdhasoid rather than tdtypeid. I mean, if the only point of >> comparing the type OIDs is to find out whether the table-has-OIDs >> setting matches, we could instead

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-24 Thread Tobias Oberstein
Hi, Am 24.01.2017 um 18:41 schrieb Andres Freund: Hi, On 2017-01-24 18:37:14 +0100, Tobias Oberstein wrote: assume that it'd get more than swamped with doing actualy work, and with buffering the frequently accessed stuff in memory. What I am trying to say is: the syscall overhead of doing l

Re: [HACKERS] pgbench more operators & functions

2017-01-24 Thread Tom Lane
Robert Haas writes: >> On Fri, Dec 2, 2016 at 1:28 AM, Fabien COELHO wrote: >>> This decision is both illogical and arbitrary. >> I disagree. I think his decision was probably based on this email from me: > (argh, sent too soon) > https://www.postgresql.org/message-id/ca+tgmoa0zp4a+s+kosav4qfd

Re: \if, \elseif, \else, \endif (was Re: [HACKERS] PSQL commands: \quit_if, \quit_unless)

2017-01-24 Thread Daniel Verite
Corey Huinker wrote: > > > > 1: unrecognized value "whatever" for "\if"; assuming "on" > > > > I do not think that the script should continue with such an assumption. > > > > I agree, and this means we can't use ParseVariableBool() as-is The patch at https://commitfest.postgresql.org/12/

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-24 Thread Andres Freund
Hi, On 2017-01-24 18:57:47 +0100, Tobias Oberstein wrote: > Am 24.01.2017 um 18:41 schrieb Andres Freund: > > On 2017-01-24 18:37:14 +0100, Tobias Oberstein wrote: > > > The syscall overhead is visible in production too .. I watched PG using > > > perf > > > live, and lseeks regularily appear at

Re: \if, \elseif, \else, \endif (was Re: [HACKERS] PSQL commands: \quit_if, \quit_unless)

2017-01-24 Thread Corey Huinker
> > ISTM that it's important that eventually ParseVariableBool() > and \if agree on what evaluates to true and false (and the > more straightforward way to achieve that is by \if calling > directly ParseVariableBool), but that it's not productive that we > discuss /if issues relatively to the behav

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-24 Thread Tobias Oberstein
Hi, pid |syscall| cnt | cnt_per_sec -+---+-+- | syscalls:sys_enter_lseek | 4091584 | 136386 | syscalls:sys_enter_newfstat | 2054988 | 68500 | syscalls

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-24 Thread Alvaro Herrera
Tobias Oberstein wrote: > I am benchmarking IOPS, and while doing so, it becomes apparent that at > these scales it does matter _how_ IO is done. > > The most efficient way is libaio. I get 9.7 million/sec IOPS with low CPU > load. Using any synchronous IO engine is slower and produces higher loa

Re: \if, \elseif, \else, \endif (was Re: [HACKERS] PSQL commands: \quit_if, \quit_unless)

2017-01-24 Thread Corey Huinker
On Tue, Jan 24, 2017 at 1:25 PM, Corey Huinker wrote: > This might be asking a lot, but could we make a "strict" mode for > ParseVariableBool() that returns a success boolean, and have the existing > ParseVariableBool() signature call that new function, and issue the > "assuming " warning if the

Re: [HACKERS] pgbench more operators & functions

2017-01-24 Thread David G. Johnston
On Wed, Oct 5, 2016 at 2:03 AM, Fabien COELHO wrote: > > I've got no objection to a more-nearly-TPC-B script as an option. >> > > Good, because adding a "per-spec" tpc-b as an additional builtin option is > one of my intentions, once pgbench is capable of it. ​Trying to scan the thread as an in

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-24 Thread Andres Freund
On 2017-01-24 15:36:13 -0300, Alvaro Herrera wrote: > Tobias Oberstein wrote: > > > I am benchmarking IOPS, and while doing so, it becomes apparent that at > > these scales it does matter _how_ IO is done. > > > > The most efficient way is libaio. I get 9.7 million/sec IOPS with low CPU > > load.

Re: [HACKERS] [PATCH] Rename pg_switch_xlog to pg_switch_wal

2017-01-24 Thread Stephen Frost
* Vladimir Rusinov (vrusi...@google.com) wrote: > On Mon, Jan 23, 2017 at 6:59 PM, Stephen Frost wrote: > > I don't have any problem with asking for a summary of the exact set of > > changes that he's planning to make though. My understanding is that it > > includes changing program names, comman

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-24 Thread Andres Freund
On 2017-01-24 19:25:52 +0100, Tobias Oberstein wrote: > Hi, > > > > pid |syscall| cnt | cnt_per_sec > > > -+---+-+- > > > | syscalls:sys_enter_lseek | 4091584 | 136386 > > >

Re: [HACKERS] pgbench more operators & functions

2017-01-24 Thread Tom Lane
"David G. Johnston" writes: > Maybe consider writing a full patch series that culminates with this > additional builtin option being added as the final patch - breaking out > this (and probably other) "requirements" patches separately to aid in > review. The presentation of just "add new stuff th

Re: \if, \elseif, \else, \endif (was Re: [HACKERS] PSQL commands: \quit_if, \quit_unless)

2017-01-24 Thread Tom Lane
"Daniel Verite" writes: > ISTM that it's important that eventually ParseVariableBool() > and \if agree on what evaluates to true and false (and the > more straightforward way to achieve that is by \if calling > directly ParseVariableBool), but that it's not productive that we > discuss /if issues

Re: \if, \elseif, \else, \endif (was Re: [HACKERS] PSQL commands: \quit_if, \quit_unless)

2017-01-24 Thread Corey Huinker
Revised patch, with one caveat: It contains copy/pasted code from variable.c intended to bridge the gap until https://commitfest.postgresql. org/12/799/ (changing ParseVariableBool to detect invalid boolean-ish strings) is merged. We may want to pause full-review of this patch pending resolution o

Re: [HACKERS] Improvements in psql hooks for variables

2017-01-24 Thread Tom Lane
"Daniel Verite" writes: > Here's an update with these changes: I scanned this patch very quickly and think it addresses my previous stylistic objections. Rahila is the reviewer of record though, so I'll wait for his comments before going further. regards, tom lane --

Re: [HACKERS] PoC plpgsql - possibility to force custom or generic plan

2017-01-24 Thread Pavel Stehule
2017-01-24 6:38 GMT+01:00 Pavel Stehule : > Hi > > 2017-01-23 21:59 GMT+01:00 Jim Nasby : > >> On 1/23/17 2:10 PM, Pavel Stehule wrote: >> >>> Comments, notes? >>> >> >> +1 on the idea. It'd also be nice if we could expose control of plans for >> dynamic SQL, though I suspect that's not terribly u

Re: [HACKERS] Declarative partitioning - another take

2017-01-24 Thread Robert Haas
On Thu, Jan 19, 2017 at 9:58 PM, Amit Langote wrote: > [ new patches ] Committed 0001 and 0002. See my earlier email for comments on 0003. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgre

Re: [HACKERS] pgbench more operators & functions

2017-01-24 Thread Robert Haas
On Tue, Jan 24, 2017 at 12:58 PM, Tom Lane wrote: > I'd be okay with the parts of this that duplicate existing backend syntax > and semantics, but I don't especially want to invent further than that. I agree, and I think that's pretty much what we all said back in October, and the patch hasn't be

Re: [HACKERS] [PATCH] Rename pg_switch_xlog to pg_switch_wal

2017-01-24 Thread Robert Haas
On Mon, Jan 23, 2017 at 1:55 PM, Peter Eisentraut wrote: > For the record, I don't like the name "xlog" either. It would be nice > if we could have more consistent and intuitive naming. Great! > But I don't see any proposals to actually change all uses of "xlog" to > "wal". What about program

Re: [HACKERS] Parallel Index Scans

2017-01-24 Thread Robert Haas
On Mon, Jan 23, 2017 at 1:03 AM, Amit Kapila wrote: > In spite of being careful, I missed reorganizing the functions in > genam.h which I have done in attached patch. Cool. Committed parallel-generic-index-scan.2.patch. Thanks. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Ente

Re: [HACKERS] increasing the default WAL segment size

2017-01-24 Thread Robert Haas
On Fri, Jan 20, 2017 at 7:00 PM, Michael Paquier wrote: >> No, because the output of SHOW is always of type text, regardless of >> the type of the GUC. > > Thinking about that over night, that looks pretty nice. What would be > nicer though would be to add INT8OID and BYTEAOID in the list, and > c

Re: [HACKERS] Re: [bug fix] Cascading standby cannot catch up and get stuck emitting the same message repeatedly

2017-01-24 Thread Fujii Masao
On Thu, Nov 24, 2016 at 4:24 PM, Amit Kapila wrote: > On Thu, Nov 24, 2016 at 10:29 AM, Tsunakawa, Takayuki > wrote: >> From: pgsql-hackers-ow...@postgresql.org >>> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Amit Kapila >>> Thanks for the clarification, I could reproduce the issue a

Re: [HACKERS] [PATCH] pgpassfile connection option

2017-01-24 Thread Tom Lane
Fabien COELHO writes: > I've switch in the CF to "ready for committer", and we'll see what the > next level thinks about it:-) Pushed with a few adjustments: * It seemed to me that the appropriate parameter name is "passfile" not "pgpassfile". In all but a couple of legacy cases, the environme

Re: [HACKERS] [COMMITTERS] pgsql: Add pg_sequence system catalog

2017-01-24 Thread Peter Eisentraut
On 1/24/17 10:23 AM, Stephen Frost wrote: > It wouldn't hurt to add a comment as to why things are so different in > PG10 than other versions, ie: > > /* > * In PG10, sequence meta-data was moved into pg_sequence, so switch to > * the pg_catalog schema instead of operating in a user schema and p

Re: [HACKERS] PATCH: recursive json_populate_record()

2017-01-24 Thread Nikita Glukhov
On 22.01.2017 21:58, Tom Lane wrote: In the meantime, we are *not* going to have attndims be semantically significant in just this one function. Therefore, please remove this patch's dependence on attndims. Ok, I've removed patch's dependence on attndims. But I still believe that someday Post

Re: [HACKERS] patch: function xmltable

2017-01-24 Thread Andres Freund
Hi, On 2017-01-24 17:38:49 -0300, Alvaro Herrera wrote: > +static Datum ExecEvalTableExpr(TableExprState *tstate, ExprContext *econtext, > + bool *isnull); > +static Datum ExecEvalTableExprFast(TableExprState *exprstate, ExprContext > *econtext, > +

Re: [HACKERS] patch: function xmltable

2017-01-24 Thread Alvaro Herrera
Andres Freund wrote: > Hi, > > On 2017-01-24 17:38:49 -0300, Alvaro Herrera wrote: > > +static Datum ExecEvalTableExpr(TableExprState *tstate, ExprContext > > *econtext, > > + bool *isnull); > > +static Datum ExecEvalTableExprFast(TableExprState *exprstate, ExprContext

Re: [HACKERS] patch: function xmltable

2017-01-24 Thread Andres Freund
On 2017-01-24 21:32:56 -0300, Alvaro Herrera wrote: > Andres Freund wrote: > > Hi, > > > > On 2017-01-24 17:38:49 -0300, Alvaro Herrera wrote: > > > +static Datum ExecEvalTableExpr(TableExprState *tstate, ExprContext > > > *econtext, > > > + bool *isnull); > > > +static

Re: [HACKERS] Declarative partitioning - another take

2017-01-24 Thread Amit Langote
Hi Keith, On 2017/01/20 12:40, Keith Fiske wrote: > So testing things out in pg_partman for native sub-partitioning and ran > into what is a bug for me that I know I have to fix, but I'm curious if > this can be prevented in the first place within the native partitioning > code itself. The below s

  1   2   >