Re: [HACKERS] logical decoding of two-phase transactions

2017-01-25 Thread Stas Kelvich
>> >> Yes, that’s also possible but seems to be less flexible restricting us to >> some >> specific GID format. >> >> Anyway, I can measure WAL space overhead introduced by the GID’s inside >> commit records >> to know exactly what will be the cost of such approach. > > Stas, > > Have you had

Re: [HACKERS] Speedup twophase transactions

2017-01-25 Thread Michael Paquier
On Thu, Jan 26, 2017 at 4:09 PM, Nikhil Sontakke wrote: >>I look at this patch from you and that's present for me: >>https://www.postgresql.org/message-id/CAMGcDxf8Bn9ZPBBJZba9wiyQq->Qk5uqq=vjomnrnw5s+fks...@mail.gmail.com > >> --- a/src/backend/access/transam/xlog.c >> +++ b/src/backend/access/tr

Re: [HACKERS] Radix tree for character conversion

2017-01-25 Thread Michael Paquier
On Wed, Jan 25, 2017 at 7:18 PM, Ishii Ayumi wrote: > I patched 4 patchset and run "make", but I got failed. > Is this a bug or my mistake ? > I'm sorry if I'm wrong. > > [$(TOP)]$ patch -p1 < ../0001-Add-missing-semicolon.patch > [$(TOP)]$ patch -p1 < ../0002-Correct-reference-resolution-syntax.p

Re: [HACKERS] Radix tree for character conversion

2017-01-25 Thread Michael Paquier
On Tue, Jan 10, 2017 at 8:22 PM, Kyotaro HORIGUCHI wrote: > [...patch...] Nobody has showed up yet to review this patch, so I am giving it a shot. The patch file sizes are scary at first sight, but after having a look: 36 files changed, 1411 insertions(+), 54398 deletions(-) Yes that's a surpri

Re: [HACKERS] Speedup twophase transactions

2017-01-25 Thread Nikhil Sontakke
>I look at this patch from you and that's present for me: >https://www.postgresql.org/message-id/CAMGcDxf8Bn9ZPBBJZba9wiyQq->Qk5uqq=vjomnrnw5s+fks...@mail.gmail.com > --- a/src/backend/access/transam/xlog.c > +++ b/src/backend/access/transam/xlog.c > @@ -9573,6 +9573,7 @@ xlog_redo(XLogReaderState

Re: [HACKERS] Speedup twophase transactions

2017-01-25 Thread Michael Paquier
On Thu, Jan 26, 2017 at 1:38 PM, Nikhil Sontakke wrote: >> We should really try to do things right now, or we'll never come back >> to it. 9.3 (if my memory does not fail me?) has reduced the time to do >> promotion by removing the need of the end-of-recovery checkpoint, >> while I agree that ther

Re: [HACKERS] sequence data type

2017-01-25 Thread Michael Paquier
On Wed, Jan 25, 2017 at 11:53 PM, Peter Eisentraut wrote: > Here is an updated patch that allows changing the sequence type. This > was clearly a concern for reviewers, and the presented use cases seemed > convincing. I have been torturing this patch and it looks rather solid to me. Here are a c

Re: [HACKERS] Speedup twophase transactions

2017-01-25 Thread Nikhil Sontakke
>> The question remains whether saving off a few fsyncs/reads for these >> long-lived prepared transactions is worth the additional code churn. >> Even if we add code to go through the KnownPreparedList, we still will >> have to go through the other on-disk 2PC transactions anyways. So, >> maybe no

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Jan 25, 2017 at 7:37 PM, Andres Freund wrote: > > On 2017-01-25 19:30:08 -0500, Stephen Frost wrote: > >> * Peter Geoghegan (p...@heroku.com) wrote: > >> > On Wed, Jan 25, 2017 at 3:30 PM, Stephen Frost > >> > wrote: > >> > > As it is, there

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > Your proposed policy is essentially that functions should have > built-in superuser checks if having access to that function is > sufficient to escalate your account to full superuser privileges. Yes, I do. > 1. There's no consensus on any s

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Greg Stark
On 26 January 2017 at 01:58, Thomas Munro wrote: > > I don't know how comparable it is to our checksum technology, but > MySQL seems to have some kind of checksums on table data, and you can > find public emails, blogs etc lamenting corrupted databases by > searching Google for the string "InnoDB:

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 7:37 PM, Andres Freund wrote: > On 2017-01-25 19:30:08 -0500, Stephen Frost wrote: >> * Peter Geoghegan (p...@heroku.com) wrote: >> > On Wed, Jan 25, 2017 at 3:30 PM, Stephen Frost wrote: >> > > As it is, there are backup solutions which *do* check the checksum when >> > >

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 8:22 PM, Stephen Frost wrote: > Apparently we disagree about what is a 'reasonable manner'. Yes. I think that a "reasonable manner" should mean "what the DBA thinks is reasonable", whereas you apparently think it should mean "what the DBA thinks is reasonable, but only if

Re: [HACKERS] BUG: pg_stat_statements query normalization issues with combined queries

2017-01-25 Thread Craig Ringer
On 15 January 2017 at 05:18, Tom Lane wrote: > Fabien COELHO writes: >>> It ends up being about 30 fewer lines of code overall, despite there >>> being four places that have to make ad-hoc RawStmt nodes. On the whole >>> I feel like this is cleaner, > >> I agree: Better typing, more homogeneous

Re: [HACKERS] pg_hba_file_settings view patch

2017-01-25 Thread Michael Paquier
On Thu, Jan 26, 2017 at 2:32 AM, Tom Lane wrote: > The way I'd be inclined to make the individual reporting changes is like > > if (!EnableSSL) > +{ > - ereport(LOG, > + ereport(elevel, > (errcode(ERRCODE_CONFIG_FILE_ERR

Re: [HACKERS] logical decoding of two-phase transactions

2017-01-25 Thread Craig Ringer
On 5 January 2017 at 20:43, Stas Kelvich wrote: > >> On 5 Jan 2017, at 13:49, Simon Riggs wrote: >> >> Surely in this case the master server is acting as the Transaction >> Manager, and it knows the mapping, so we are good? >> >> I guess if you are using >2 nodes then you need to use full 2PC on

Re: [HACKERS] Speedup twophase transactions

2017-01-25 Thread Michael Paquier
On Wed, Jan 25, 2017 at 11:55 PM, Nikhil Sontakke wrote: >> We are talking about the recovery/promote code path. Specifically this >> call to KnownPreparedRecreateFiles() in PrescanPreparedTransactions(). >> >> We write the files to disk and they get immediately read up in the >> following code. W

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Thomas Munro
On Thu, Jan 26, 2017 at 2:28 PM, Stephen Frost wrote: > Sadly, without having them enabled by default, there's not a huge corpus > of example cases to draw from. > > There have been a few examples already posted about corruption failures > with PG, but one can't say with certainty that they would

Re: [HACKERS] COPY as a set returning function

2017-01-25 Thread Corey Huinker
> > >> I don't understand why do we have all these checks. Can't we just pass >> the values to COPY and let it apply the checks? That way, when COPY is >> updated to support multibyte escape chars (for example) we don't need to >> touch this code. Together with removing the unneeded braces that

Re: [HACKERS] pgbench more operators & functions

2017-01-25 Thread Stephen Frost
Fabien, * Fabien COELHO (coe...@cri.ensmp.fr) wrote: > I think that there is a misunderstanding, most of which being my fault. No worries, it happens. :) > I have really tried to do everything that was required from > committers, including revising the patch to match all previous > feedback. Th

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Stephen Frost
Peter, * Peter Geoghegan (p...@heroku.com) wrote: > On Wed, Jan 25, 2017 at 1:22 PM, Peter Geoghegan wrote: > > I understand that my experience with storage devices is unusually > > narrow compared to everyone else here. That's why I remain neutral on > > the high level question of whether or not

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Peter Geoghegan
On Wed, Jan 25, 2017 at 1:22 PM, Peter Geoghegan wrote: > I understand that my experience with storage devices is unusually > narrow compared to everyone else here. That's why I remain neutral on > the high level question of whether or not we ought to enable checksums > by default. I'll ask other

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Stephen Frost
Andres, * Andres Freund (and...@anarazel.de) wrote: > On 2017-01-25 18:04:09 -0500, Stephen Frost wrote: > > Robert's made it clear that he'd like to have a blanket rule that we > > don't have superuser checks in these code paths if they can be GRANT'd > > at the database level, which goes beyond

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Jan 25, 2017 at 6:30 PM, Stephen Frost wrote: > > I hope to discuss it further after we have the ability to turn it off > > easily. > > I think we should have the ability to flip it in BOTH directions easily. Presumably you imply this to mea

Re: [HACKERS] patch: function xmltable

2017-01-25 Thread Pavel Stehule
2017-01-25 15:07 GMT+01:00 Alvaro Herrera : > Tom Lane wrote: > > Andres Freund writes: > > > On 2017-01-24 21:32:56 -0300, Alvaro Herrera wrote: > > >> XMLTABLE is specified by the standard to return multiple rows ... but > > >> then as far as my reading goes, it is only supposed to be supported

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Stephen Frost
Michael, * Michael Paquier (michael.paqu...@gmail.com) wrote: > That would be enough. It should also be rare enough that there would > not be that many pages to track when looking at records from the > backup start position to minimum recovery point. It could be also > simpler, though more time-co

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Stephen Frost
* Andres Freund (and...@anarazel.de) wrote: > On 2017-01-25 19:30:08 -0500, Stephen Frost wrote: > > * Peter Geoghegan (p...@heroku.com) wrote: > > > On Wed, Jan 25, 2017 at 3:30 PM, Stephen Frost wrote: > > > > As it is, there are backup solutions which *do* check the checksum when > > > > backin

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Michael Paquier
On Thu, Jan 26, 2017 at 9:32 AM, Stephen Frost wrote: > * Robert Haas (robertmh...@gmail.com) wrote: >> On Wed, Jan 25, 2017 at 7:19 PM, Michael Paquier >> wrote: >> > On Thu, Jan 26, 2017 at 9:14 AM, Peter Geoghegan wrote: >> >> On Wed, Jan 25, 2017 at 3:30 PM, Stephen Frost wrote: >> >>> As i

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Andres Freund
On 2017-01-25 19:30:08 -0500, Stephen Frost wrote: > * Peter Geoghegan (p...@heroku.com) wrote: > > On Wed, Jan 25, 2017 at 3:30 PM, Stephen Frost wrote: > > > As it is, there are backup solutions which *do* check the checksum when > > > backing up PG. This is no longer, thankfully, some hypothet

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Jan 25, 2017 at 7:19 PM, Michael Paquier > wrote: > > On Thu, Jan 26, 2017 at 9:14 AM, Peter Geoghegan wrote: > >> On Wed, Jan 25, 2017 at 3:30 PM, Stephen Frost wrote: > >>> As it is, there are backup solutions which *do* check the checksum

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Stephen Frost
* Michael Paquier (michael.paqu...@gmail.com) wrote: > On Thu, Jan 26, 2017 at 9:14 AM, Peter Geoghegan wrote: > > On Wed, Jan 25, 2017 at 3:30 PM, Stephen Frost wrote: > >> As it is, there are backup solutions which *do* check the checksum when > >> backing up PG. This is no longer, thankfully,

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Stephen Frost
* Peter Geoghegan (p...@heroku.com) wrote: > On Wed, Jan 25, 2017 at 3:30 PM, Stephen Frost wrote: > > As it is, there are backup solutions which *do* check the checksum when > > backing up PG. This is no longer, thankfully, some hypothetical thing, > > but something which really exists and will

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Michael Paquier
On Thu, Jan 26, 2017 at 9:22 AM, Andres Freund wrote: > On 2017-01-26 09:19:28 +0900, Michael Paquier wrote: >> On Thu, Jan 26, 2017 at 9:14 AM, Peter Geoghegan wrote: >> > On Wed, Jan 25, 2017 at 3:30 PM, Stephen Frost wrote: >> >> As it is, there are backup solutions which *do* check the check

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 7:19 PM, Michael Paquier wrote: > On Thu, Jan 26, 2017 at 9:14 AM, Peter Geoghegan wrote: >> On Wed, Jan 25, 2017 at 3:30 PM, Stephen Frost wrote: >>> As it is, there are backup solutions which *do* check the checksum when >>> backing up PG. This is no longer, thankfully

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 6:30 PM, Stephen Frost wrote: > I hope to discuss it further after we have the ability to turn it off > easily. I think we should have the ability to flip it in BOTH directions easily. >> Second, really hard to enable is a relative term. I accept that >> enabling checksu

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Andres Freund
On 2017-01-26 09:19:28 +0900, Michael Paquier wrote: > On Thu, Jan 26, 2017 at 9:14 AM, Peter Geoghegan wrote: > > On Wed, Jan 25, 2017 at 3:30 PM, Stephen Frost wrote: > >> As it is, there are backup solutions which *do* check the checksum when > >> backing up PG. This is no longer, thankfully,

Re: [HACKERS] safer node casting

2017-01-25 Thread Tom Lane
Andres Freund writes: > On 2016-12-31 12:08:22 -0500, Peter Eisentraut wrote: >> RestrictInfo *rinfo = castNode(RestrictInfo, lfirst(lc)); > Are you planning to add this / update this patch? Because I really would > have liked this a number of times already... I can update it according > to my s

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Michael Paquier
On Thu, Jan 26, 2017 at 9:14 AM, Peter Geoghegan wrote: > On Wed, Jan 25, 2017 at 3:30 PM, Stephen Frost wrote: >> As it is, there are backup solutions which *do* check the checksum when >> backing up PG. This is no longer, thankfully, some hypothetical thing, >> but something which really exist

Re: [HACKERS] safer node casting

2017-01-25 Thread Andres Freund
Hi Peter, On 2016-12-31 12:08:22 -0500, Peter Eisentraut wrote: > RestrictInfo *rinfo = castNode(RestrictInfo, lfirst(lc)); Are you planning to add this / update this patch? Because I really would have liked this a number of times already... I can update it according to my suggestions (to a

Re: [HACKERS] Should buffer of initialization fork have a BM_PERMANENT flag

2017-01-25 Thread Michael Paquier
(Adding Robert in CC.) On Thu, Jan 26, 2017 at 4:34 AM, Wang Hao wrote: > An unlogged table has an initialization fork. The initialization fork does > not have an BM_PERMANENT flag when get a buffer. > In checkpoint (not shutdown or end of recovery), it will not write to disk. > after a crash rec

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Peter Geoghegan
On Wed, Jan 25, 2017 at 3:30 PM, Stephen Frost wrote: > As it is, there are backup solutions which *do* check the checksum when > backing up PG. This is no longer, thankfully, some hypothetical thing, > but something which really exists and will hopefully keep users from > losing data. Wouldn't

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Jan 25, 2017 at 2:23 PM, Stephen Frost wrote: > > Yet, our default is to have them disabled and *really* hard to enable. > > First of all, that could be fixed by further development. I'm certainly all for doing so, but I don't agree

Re: [HACKERS] tuplesort_gettuple_common() and *should_free argument

2017-01-25 Thread Tom Lane
[ in the service of closing out this thread... ] Peter Geoghegan writes: > Finally, 0003-* is a Valgrind suppression borrowed from my parallel > CREATE INDEX patch. It's self-explanatory. Um, I didn't find it all that self-explanatory. Why wouldn't we want to avoid writing undefined data? I th

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Andres Freund
On 2017-01-25 18:04:09 -0500, Stephen Frost wrote: > Andres, > > * Andres Freund (and...@anarazel.de) wrote: > > On 2017-01-25 16:52:38 -0500, Stephen Frost wrote: > > > * Robert Haas (robertmh...@gmail.com) wrote: > > > > Preventing people from calling functions by denying the ability to > > > >

Re: [HACKERS] tuplesort_gettuple_common() and *should_free argument

2017-01-25 Thread Peter Geoghegan
On Wed, Jan 25, 2017 at 3:11 PM, Tom Lane wrote: > Please. You might want to hit the existing ones with a separate patch, > but it doesn't much matter; I'd be just as happy with a patch that did > both things. Got it. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] tuplesort_gettuple_common() and *should_free argument

2017-01-25 Thread Tom Lane
Peter Geoghegan writes: > It means "another call to tuplesort_gettupleslot", but I believe that > it would be safer (more future-proof) to actually specify "the slot > contents may be invalidated by any subsequent manipulation of the > tuplesort's state" instead. WFM. >> There are several other

Re: [HACKERS] tuplesort_gettuple_common() and *should_free argument

2017-01-25 Thread Peter Geoghegan
On Wed, Jan 25, 2017 at 2:49 PM, Tom Lane wrote: > I looked at the 0002 patch, and while the code is probably OK, I am > dissatisfied with this API spec: > > + * If copy is TRUE, the slot receives a copied tuple that will stay valid > + * regardless of future manipulations of the tuplesort's state

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Stephen Frost
Andres, * Andres Freund (and...@anarazel.de) wrote: > On 2017-01-25 16:52:38 -0500, Stephen Frost wrote: > > * Robert Haas (robertmh...@gmail.com) wrote: > > > Preventing people from calling functions by denying the ability to > > > meaningfully GRANT EXECUTE on those functions doesn't actually st

Re: [HACKERS] tuplesort_gettuple_common() and *should_free argument

2017-01-25 Thread Tom Lane
Peter Geoghegan writes: > I should have already specifically pointed out that the original > discussion on what became 0002-* is here: > postgr.es/m/7256.1476711...@sss.pgh.pa.us > As I said already, the general idea seems uncontroversial. I looked at the 0002 patch, and while the code is probabl

Re: [HACKERS] patch: function xmltable

2017-01-25 Thread Pavel Stehule
2017-01-25 23:33 GMT+01:00 Andres Freund : > On 2017-01-25 22:51:37 +0100, Pavel Stehule wrote: > > 2017-01-25 22:40 GMT+01:00 Andres Freund : > > > > I afraid when I cannot to reuse a SRF infrastructure, I have to > > > reimplement > > > > it partially :( - mainly for usage in "ROWS FROM ()" > >

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Andres Freund
Hi, On 2017-01-25 16:52:38 -0500, Stephen Frost wrote: > * Robert Haas (robertmh...@gmail.com) wrote: > > Preventing people from calling functions by denying the ability to > > meaningfully GRANT EXECUTE on those functions doesn't actually stop > > them from delegating those functions to non-super

Re: [HACKERS] patch: function xmltable

2017-01-25 Thread Andres Freund
On 2017-01-25 22:51:37 +0100, Pavel Stehule wrote: > 2017-01-25 22:40 GMT+01:00 Andres Freund : > > > I afraid when I cannot to reuse a SRF infrastructure, I have to > > reimplement > > > it partially :( - mainly for usage in "ROWS FROM ()" > > > > The TableExpr implementation is based on SRF now.

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Jan 25, 2017 at 3:17 PM, Stephen Frost wrote: > > Your email is 'pg_ls_dir & friends', which I took to imply at *least* > > pg_read_file() and pg_read_binary_file(), and it's not unreasonable to > > think you may have meant everything

Re: [HACKERS] patch: function xmltable

2017-01-25 Thread Pavel Stehule
2017-01-25 22:40 GMT+01:00 Andres Freund : > Hi, > > > > > I'll try to explain my motivation. Please, check it and correct me > if I > > > am > > > > wrong. I don't keep on my implementation - just try to implement > XMLTABLE > > > > be consistent with another behave and be used all time without a

Re: [HACKERS] PATCH: recursive json_populate_record()

2017-01-25 Thread Tom Lane
Nikita Glukhov writes: > On 25.01.2017 23:58, Tom Lane wrote: >> I think you need to take a second look at the code you're producing >> and realize that it's not so clean either. This extract from >> populate_record_field, for example, is pretty horrid: > But what if we introduce some helper mac

Re: [HACKERS] multivariate statistics (v19)

2017-01-25 Thread Michael Paquier
On Wed, Jan 25, 2017 at 9:56 PM, Alvaro Herrera wrote: > Michael Paquier wrote: >> And nothing has happened since. Are there people willing to review >> this patch and help it proceed? > > I am going to grab this patch as committer. Thanks, that's good to know. -- Michael -- Sent via pgsql-ha

Re: [HACKERS] patch: function xmltable

2017-01-25 Thread Andres Freund
Hi, > > > I'll try to explain my motivation. Please, check it and correct me if I > > am > > > wrong. I don't keep on my implementation - just try to implement XMLTABLE > > > be consistent with another behave and be used all time without any > > > surprise. > > > > > > 1. Any function that produce

Re: [HACKERS] PATCH: recursive json_populate_record()

2017-01-25 Thread Nikita Glukhov
On 25.01.2017 23:58, Tom Lane wrote: I think you need to take a second look at the code you're producing and realize that it's not so clean either. This extract from populate_record_field, for example, is pretty horrid: But what if we introduce some helper macros like this: #define JsValueIsN

Re: [HACKERS] patch: function xmltable

2017-01-25 Thread Pavel Stehule
> > > > > > > If you plan to hold support SRFin target list, then nothing is different. > > In last patch is executed under nodeProjectSet. > > It is, because we suddenly need to call different functions - and I'm > revamping most of execQual to have an opcode dispatch based execution > model (whic

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-25 Thread Tobias Oberstein
Hi, Synthetic PG workload or real world production workload? Both might work, production-like has bigger pull, but I'd guess synthetic is good enough. Thanks! The box should get PostgreSQL in the not too distant future. It'll get a backup from prod, but will act as new prod, so it might tak

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Peter Geoghegan
On Wed, Jan 25, 2017 at 12:23 PM, Robert Haas wrote: > Also, I think that one of the big problems with the way checksums work > is that you don't find problems with your archived data until it's too > late. Suppose that in February bits get flipped in a block. You > don't access the data until J

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 2:46 PM, Jim Nasby wrote: > I think the two need to be integrated much better than they are right now. > They should certainly be in the same .so, and as others have mentioned the > docs need to be fixed. For consistency, I think the name should just be > pg_prewarm, as wel

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 3:17 PM, Stephen Frost wrote: > Your email is 'pg_ls_dir & friends', which I took to imply at *least* > pg_read_file() and pg_read_binary_file(), and it's not unreasonable to > think you may have meant everything in adminpack, which also includes > pg_file_write(), pg_file_

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-01-25 Thread Alvaro Herrera
Looking at your 0002 patch now. It no longer applies, but the conflicts are trivial to fix. Please rebase and resubmit. I think the way WARM works has been pretty well hammered by now, other than the CREATE INDEX CONCURRENTLY issues, so I'm looking at the code from a maintainability point of vie

Re: [HACKERS] PoC plpgsql - possibility to force custom or generic plan

2017-01-25 Thread Pavel Stehule
2017-01-25 21:06 GMT+01:00 Jim Nasby : > On 1/23/17 11:38 PM, Pavel Stehule wrote: > >> >> Instead of paralleling all the existing namespace stuff, I wonder if >> it'd be better to create explicit block infrastructure. AFAIK >> PRAGMAs are going to have a lot of the same requirements (

Re: [HACKERS] PATCH: recursive json_populate_record()

2017-01-25 Thread Tom Lane
Nikita Glukhov writes: > On 22.01.2017 21:58, Tom Lane wrote: >> 5. The business with having some arguments that are only for json and >> others that are only for jsonb, eg in populate_scalar(), also makes me >> itch. > I've refactored json/jsonb passing using new struct JsValue which contains >

Re: [HACKERS] patch: function xmltable

2017-01-25 Thread Andres Freund
Hi, On 2017-01-25 05:45:24 +0100, Pavel Stehule wrote: > 2017-01-25 1:35 GMT+01:00 Andres Freund : > > > On 2017-01-24 21:32:56 -0300, Alvaro Herrera wrote: > > > Andres Freund wrote: > > > > Hi, > > > > > > > > On 2017-01-24 17:38:49 -0300, Alvaro Herrera wrote: > > > > > +static Datum ExecEvalT

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 2:23 PM, Stephen Frost wrote: >> Sure. If the database runs fast enough with checksums enabled, >> there's basically no reason to have them turned off. The issue is >> when it doesn't. > > I don't believe we're talking about forcing every user to have checksums > enabled.

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Ants Aasma
On Wed, Jan 25, 2017 at 8:18 PM, Robert Haas wrote: > Also, it's not as if there are no other ways of checking whether your > disks are failing. SMART, for example, is supposed to tell you about > incipient hardware failures before PostgreSQL ever sees a bit flip. > Surely an average user would l

[HACKERS] Should buffer of initialization fork have a BM_PERMANENT flag

2017-01-25 Thread Wang Hao
An unlogged table has an initialization fork. The initialization fork does not have an BM_PERMANENT flag when get a buffer. In checkpoint (not shutdown or end of recovery), it will not write to disk. after a crash recovery, the page of initialization fork will not correctly, then make the main fork

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Jan 25, 2017 at 2:08 PM, Stephen Frost wrote: > > That isn't what you're doing with those functions though, you're giving > > the monitoring tool superuser-level rights but trying to pretend like > > you're not. > > Uh, how so? None

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Joshua D. Drake
On 01/25/2017 11:41 AM, Tom Lane wrote: Stephen Frost writes: Would you say that most user's databases run fast enough with checksums enabled? Or more than most, maybe 70%? 80%? In today's environment, I'd probably say that it's more like 90+%. It would be nice if there were some actual ev

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2017-01-25 Thread Claudio Freire
On Wed, Jan 25, 2017 at 1:54 PM, Masahiko Sawada wrote: > Thank you for updating the patch! > > + /* > +* Quickly rule out by lower bound (should happen a lot) Upper bound > was > +* already checked by segment search > +*/ > + if (vac_cmp_itemptr((void *) itemp

Re: [HACKERS] PoC plpgsql - possibility to force custom or generic plan

2017-01-25 Thread Jim Nasby
On 1/23/17 11:38 PM, Pavel Stehule wrote: Instead of paralleling all the existing namespace stuff, I wonder if it'd be better to create explicit block infrastructure. AFAIK PRAGMAs are going to have a lot of the same requirements (certainly the nesting is the same), and we might

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Jan 25, 2017 at 2:13 PM, Stephen Frost wrote: > > I went over *every* superuser check in the system when I did that work, > > wrote up a long email about why I made the decisions that I did, posted > > it here, had follow-on discussio

[HACKERS] Re: [BUGS] Problem in using pgbench's --connect(-C) and --rate=rate(-R rate) options together.

2017-01-25 Thread Fabien COELHO
Repost from bugs. -- Fabien. -- Forwarded message -- Date: Wed, 25 Jan 2017 18:59:45 +0100 (CET) From: Fabien COELHO To: nuko yokohama Cc: PostgreSQL Bugs List Subject: Re: [BUGS] Problem in using pgbench's --connect(-C) and --rate=rate(-R rate) options together. It ope

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-25 Thread Andres Freund
Hi, On 2017-01-25 10:16:32 +0100, Tobias Oberstein wrote: > > > Using pread instead of lseek+read halfes the syscalls. > > > > > > I really don't understand what you are fighting here .. > > > > Sure, there's some overhead. And as I said upthread, I'm much less > > against this change than Tom.

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-25 Thread Jim Nasby
On 1/25/17 1:46 PM, Jim Nasby wrote: Based on that and other feedback I'm going to mark this as returned with feedback, though if you're able to get a revised patch in the next few days please do. Actually, based on the message that popped up when I went to do that I guess it's better not to d

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-25 Thread Jim Nasby
On 1/24/17 11:13 PM, Beena Emerson wrote: On Wed, Jan 25, 2017 at 10:36 AM, Jim Nasby mailto:jim.na...@bluetreble.com>> wrote: On 1/24/17 2:26 AM, Mithun Cy wrote: Thanks for looking into this patch, I just downloaded the patch and applied same to the latest code, I can see

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Tom Lane
Stephen Frost writes: > Would you say that most user's databases run fast enough with checksums > enabled? Or more than most, maybe 70%? 80%? In today's environment, > I'd probably say that it's more like 90+%. It would be nice if there were some actual evidence about this, rather than numbers

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Stephen Frost
* Peter Geoghegan (p...@heroku.com) wrote: > On Wed, Jan 25, 2017 at 10:18 AM, Robert Haas wrote: > > Trying to force those people to use checksums is just masterminding; > > they've made their own decision that it's not worth bothering with. > > When something goes wrong, WE still care about dist

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Jan 25, 2017 at 12:02 AM, Jim Nasby wrote: > > I'm not completely grokking your second paragraph, but I would think that an > > average user would love got get a heads-up that their hardware is failing. > > Sure. If the database run

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 2:13 PM, Stephen Frost wrote: > I went over *every* superuser check in the system when I did that work, > wrote up a long email about why I made the decisions that I did, posted > it here, had follow-on discussions, all of which lead to the patch which > ended up going in.

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 2:08 PM, Stephen Frost wrote: > That isn't what you're doing with those functions though, you're giving > the monitoring tool superuser-level rights but trying to pretend like > you're not. Uh, how so? None of those functions can be used to escalate to superuser privilege

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > Also, the same argument could be made about removing the built-in > superuser check from ANY function, and we've already rejected that > argument for a bunch of other functions. If we say that argument is > valid for some functions but not ot

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Stephen Frost
Greg, * Greg Stark (st...@mit.edu) wrote: > I tend to agree. But in the past when this came up people pointed out > you could equally do things this way and still grant all the access > you wanted using SECURITY DEFINER. Arguably that's a better approach > because then instead of auditing the enti

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > The use case I have in mind is > a monitoring tool that needs access to one more of those functions -- > in keeping with the principle of least privilege, it's much better to > give the monitoring user only the privileges which it actually nee

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 1:37 PM, Peter Geoghegan wrote: > On Wed, Jan 25, 2017 at 10:18 AM, Robert Haas wrote: >> Trying to force those people to use checksums is just masterminding; >> they've made their own decision that it's not worth bothering with. >> When something goes wrong, WE still care

Re: [HACKERS] COPY as a set returning function

2017-01-25 Thread Corey Huinker
> > > + /* param 7: escape text default null, -- defaults to whatever > quote is */ > > + if (PG_ARGISNULL(7)) > > + { > > + copy_state.escape = copy_state.quote; > > + } > > + else > > + { > > + if (copy_state.csv_mode) > > + { > > +

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Peter Geoghegan
On Wed, Jan 25, 2017 at 10:18 AM, Robert Haas wrote: > Trying to force those people to use checksums is just masterminding; > they've made their own decision that it's not worth bothering with. > When something goes wrong, WE still care about distinguishing hardware > failure from PostgreSQL failu

Re: [HACKERS] PATCH: recursive json_populate_record()

2017-01-25 Thread Tom Lane
Nikita Glukhov writes: > On 22.01.2017 21:58, Tom Lane wrote: >> If you want such macros I think it would be better to submit a separate >> cosmetic patch that tries to hide such bit-tests behind macros throughout >> the jsonb code. > I've attached that patch, but not all the bit-tests were hidde

Re: [HACKERS] COPY as a set returning function

2017-01-25 Thread Corey Huinker
On Wed, Jan 25, 2017 at 1:10 PM, David Fetter wrote: > On Wed, Jan 25, 2017 at 12:23:23PM -0500, Corey Huinker wrote: > > > > Feel free to mark it returned as feedback. The concept didn't > > generate as much enthusiasm as I had hoped, so I think the right > > thing to do now is scale it back to

Re: [HACKERS] Declarative partitioning vs. information_schema

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 1:04 PM, Peter Eisentraut wrote: > On 1/18/17 2:32 PM, Robert Haas wrote: >> Unless we can find something official, I suppose we should just >> display BASE TABLE in that case as we do in other cases. I wonder if >> the schema needs some broader revision; for example, are

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 12:02 AM, Jim Nasby wrote: > I'm not completely grokking your second paragraph, but I would think that an > average user would love got get a heads-up that their hardware is failing. Sure. If the database runs fast enough with checksums enabled, there's basically no reaso

Re: [HACKERS] COPY as a set returning function

2017-01-25 Thread David Fetter
On Wed, Jan 25, 2017 at 12:23:23PM -0500, Corey Huinker wrote: > > Feel free to mark it returned as feedback. The concept didn't > generate as much enthusiasm as I had hoped, so I think the right > thing to do now is scale it back to a patch that makes > CopyFromRawFields() externally visible so t

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Robert Haas
On Sat, Jan 21, 2017 at 11:57 AM, Andres Freund wrote: > On 2017-01-21 11:39:18 +0100, Magnus Hagander wrote: >> Is it time to enable checksums by default, and give initdb a switch to turn >> it off instead? > > -1 - the WAL overhead is quite massive, and in contrast to the other > GUCs recently c

Re: [HACKERS] Declarative partitioning vs. information_schema

2017-01-25 Thread Peter Eisentraut
On 1/18/17 2:32 PM, Robert Haas wrote: > Unless we can find something official, I suppose we should just > display BASE TABLE in that case as we do in other cases. I wonder if > the schema needs some broader revision; for example, are there > information_schema elements intended to show informatio

Re: [HACKERS] pdf versus single-html

2017-01-25 Thread Peter Eisentraut
On 1/21/17 6:29 AM, Erik Rijkers wrote: > It might even be good to include such a single-file html in the Makefile > as an option. > > I'll give it a try but has anyone done this work already, perhaps? Already exists: doc/src/sgml$ make postgres.html -- Peter Eisentraut http://ww

Re: [HACKERS] logical-replication.sgml improvements

2017-01-25 Thread Peter Eisentraut
On 1/20/17 11:00 AM, Erik Rijkers wrote: > logical-replication.sgml.diff changes Committed, thanks! -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgres

Re: [HACKERS] [FEATURE PATCH] pg_stat_statements with plans

2017-01-25 Thread Simon Riggs
On 25 January 2017 at 17:34, Julian Markwort wrote: > Analogous to this, a bad_plan is saved, when the time has been exceeded by a > factor greater than 1.1 . ...and the plan differs? Probably best to use some stat math to calculate deviation, rather than fixed %. Sounds good. -- Simon Riggs

  1   2   >