Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2016-08-22 Thread Craig Ringer
On 23 August 2016 at 08:27, Craig Ringer wrote: > On 10 August 2016 at 14:44, Michael Paquier > wrote: > >> > I am looking a bit more seriously at this patch and assigned myself as >> a reviewer. >> > > Much appreciated. > > >> > macos complains here. You may want to replace %06lds by just %06

Re: [HACKERS] Write Ahead Logging for Hash Indexes

2016-08-22 Thread Ashutosh Sharma
Hi All, Following are the steps that i have followed to verify the WAL Logging of hash index, 1. I used Mithun's patch to improve coverage of hash index code [1] to verify the WAL Logging of hash index. Firstly i have confirmed if all the XLOG records associated with hash index are being covered

Re: [HACKERS] "Some tests to cover hash_index"

2016-08-22 Thread Ashutosh Sharma
Hi, I missed to attach the patch in my previous mail. Here i attach the patch. With Regards, Ashutosh Sharma EnterpriseDB:http://www.enterprisedb.com On Tue, Aug 23, 2016 at 11:47 AM, Ashutosh Sharma wrote: > Hi All, > > I have reverified the code coverage for hash index code using the test >

Re: [HACKERS] "Some tests to cover hash_index"

2016-08-22 Thread Ashutosh Sharma
Hi All, I have reverified the code coverage for hash index code using the test file (commit-hash_coverage_test) attached with this mailing list and have found that some of the code in _hash_squeezebucket() function flow is not being covered. For this i have added a small testcase on top of 'commi

Re: [HACKERS] Patch: initdb: "'" for QUOTE_PATH (non-windows)

2016-08-22 Thread Ryan Murphy
Thanks for committing it Tom! Thank you all for your help. I really like the Postgres project! If there's anything that especially needs worked on let me know, I'd love to help. Best, Ryan

Re: [HACKERS] Tracking wait event for latches

2016-08-22 Thread Michael Paquier
On Tue, Aug 23, 2016 at 12:44 AM, Robert Haas wrote: > On Mon, Aug 22, 2016 at 9:49 AM, Michael Paquier > wrote: >> The reason why I chose this way is that there are a lot of them. It is >> painful to maintain the order of the array elements in perfect mapping >> with the list of IDs... > > You c

Re: [HACKERS] WAL consistency check facility

2016-08-22 Thread Amit Kapila
On Tue, Aug 23, 2016 at 10:57 AM, Michael Paquier wrote: > > Also, what's the use case of allowing only a certain set of rmgrs to > be checked. Wouldn't a simple on/off switch be simpler? > I think there should be a way to test WAL for one particular resource manager. For example, if someone dev

Re: [HACKERS] Bug in to_timestamp().

2016-08-22 Thread amul sul
Hi Artur Zakirov, Please see following review comment for "0001-to-timestamp-format-checking-v2.patch" & share your thought: #1. 15 + to_timestamp('2000JUN', ' MON') Documented as working case, but unfortunatly it does not : postgres=# SELECT to_timestamp('2000JUN', '

Re: [HACKERS] WAL consistency check facility

2016-08-22 Thread Michael Paquier
On Tue, Aug 23, 2016 at 1:32 PM, Amit Kapila wrote: > On Mon, Aug 22, 2016 at 9:16 PM, Robert Haas wrote: >> On Mon, Aug 22, 2016 at 9:25 AM, Michael Paquier >> wrote: >>> Another pin-point is: given a certain page, how do we identify of >>> which type it is? One possibility would be again to ex

Re: [HACKERS] WAL consistency check facility

2016-08-22 Thread Kuntal Ghosh
Yes, I've verified the outputs and log contents after running gmake installcheck and gmake installcheck-world. The status of the test was marked as pass for all the testcases. On Mon, Aug 22, 2016 at 9:26 PM, Simon Riggs wrote: > On 22 August 2016 at 13:44, Kuntal Ghosh wrote: > >> Please let m

Re: [HACKERS] WAL consistency check facility

2016-08-22 Thread Amit Kapila
On Mon, Aug 22, 2016 at 9:16 PM, Robert Haas wrote: > On Mon, Aug 22, 2016 at 9:25 AM, Michael Paquier > wrote: >> Another pin-point is: given a certain page, how do we identify of >> which type it is? One possibility would be again to extend the AM >> handler with some kind of is_self function w

Re: [HACKERS] Write Ahead Logging for Hash Indexes

2016-08-22 Thread Amit Kapila
On Tue, Aug 23, 2016 at 8:54 AM, Amit Kapila wrote: > $SUBJECT will make hash indexes reliable and usable on standby. > AFAIU, currently hash indexes are not recommended to be used in > production mainly because they are not crash-safe and with this patch, > I hope we can address that limitation a

Re: [HACKERS] Forbid use of LF and CR characters in database and role names

2016-08-22 Thread Tom Lane
Peter Geoghegan writes: > On Mon, Aug 22, 2016 at 6:28 PM, Michael Paquier > wrote: >> There is no need to put restrictions on those I think, and they are >> actually supported. > Bi-directional text support (i.e., the use of right-to-left control > characters) is known to have security implicat

Re: [HACKERS] Forbid use of LF and CR characters in database and role names

2016-08-22 Thread Peter Geoghegan
On Mon, Aug 22, 2016 at 6:28 PM, Michael Paquier wrote: > There is no need to put restrictions on those I think, and they are > actually supported. Bi-directional text support (i.e., the use of right-to-left control characters) is known to have security implications, FWIW. There is an interesting

Re: [HACKERS] Exporting more function in libpq

2016-08-22 Thread Tom Lane
Alvaro Herrera writes: > Craig Ringer wrote: >> Shouldn't that generally be done by extending libpq to add the required >> functionality? > The thought that came to me was that maybe we need a separate library > that handles the lower level operations (a "fe/be" library, if you will) > which can

Re: [HACKERS] Forbid use of LF and CR characters in database and role names

2016-08-22 Thread Michael Paquier
On Tue, Aug 23, 2016 at 10:19 AM, Peter Geoghegan wrote: > I haven't looked at the patch, but offhand I wonder if it's worth > considering control characters added by unicode, if you haven't already. There is no need to put restrictions on those I think, and they are actually supported. Look for

Re: [HACKERS] Forbid use of LF and CR characters in database and role names

2016-08-22 Thread Peter Geoghegan
I haven't looked at the patch, but offhand I wonder if it's worth considering control characters added by unicode, if you haven't already. -- Peter Geoghegan

Re: [HACKERS] Forbid use of LF and CR characters in database and role names

2016-08-22 Thread Michael Paquier
On Fri, Aug 12, 2016 at 10:12 AM, Michael Paquier wrote: > Note that pg_dump[all] and pg_upgrade already have safeguards against > those things per the same routines putting quotes for execution as > commands into psql and shell. So attached is a patch to implement this > restriction in the backen

[HACKERS] Better locale-specific-character-class handling for regexps

2016-08-22 Thread Tom Lane
I got tired of hearing complaints about the issue described in this thread: https://www.postgresql.org/message-id/flat/24241.1329347196%40sss.pgh.pa.us Here's a proposed fix. I've not done extensive performance testing, but it seems to be as fast or faster than the old code in cases where there a

Re: [HACKERS] [PATCH] Transaction traceability - txid_status(bigint)

2016-08-22 Thread Craig Ringer
On 23 August 2016 at 01:03, Robert Haas wrote: > > I think you should use underscores to separate all of the words > instead of only some of them. > > ifassigned => if_assigned ifrecent=> if_recent Updated patch series attached. As before, 0-4 intended for commit, 5 just because it'll be handy

[HACKERS] Re: [BUGS] Re: Missing rows with index scan when collation is not "C" (PostgreSQL 9.5)

2016-08-22 Thread Peter Geoghegan
On Wed, Mar 23, 2016 at 10:46 AM, Tom Lane wrote: > Robert Haas writes: >> Are you still in information-gathering more, or are you going to issue >> a recommendation on how we should proceed here, or what? > > If I had to make a recommendation right now, I would go for your > option #4, ie shut '

Re: [HACKERS] [PATCH] Transaction traceability - txid_status(bigint)

2016-08-22 Thread Craig Ringer
On 23 August 2016 at 01:03, Robert Haas wrote: > I think you should use underscores to separate all of the words > instead of only some of them. > Right. Will fix. Thanks for taking a look. > Also, note that the corresponding internal function is > GetTopTransactionIdIfAny(), which might sug

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2016-08-22 Thread Craig Ringer
On 10 August 2016 at 14:44, Michael Paquier wrote: > On Fri, Jun 3, 2016 at 8:51 PM, Dmitry Igrishin wrote: > >> BTW, I've publushed the HTML-ified SGML docs to > >> http://2ndquadrant.github.io/postgres/libpq-batch-mode.html as a > preview. > > Typo detected: "Returns 1 if the batch curently be

Re: [HACKERS] Why --backup-and-modify-in-place in perltidy config?

2016-08-22 Thread Bruce Momjian
On Mon, Aug 15, 2016 at 10:19:12AM -0400, Tom Lane wrote: > Andrew Dunstan writes: > > On 08/14/2016 04:38 PM, Tom Lane wrote: > >> I did a trial run following the current pgindent README procedure, and > >> noticed that the perltidy step left me with a pile of '.bak' files > >> littering the enti

Re: [HACKERS] Server crash due to SIGBUS(Bus Error) when trying to access the memory created using dsm_create().

2016-08-22 Thread Thomas Munro
On Tue, Aug 23, 2016 at 8:41 AM, Robert Haas wrote: > We could test to see how much it slows things down. But it > may be worth paying the cost even if it ends up being kinda expensive. Here are some numbers from a Xeon E7-8830 @ 2.13GHz running Linux 3.10 running the attached program. It's fai

Re: [HACKERS] Logical decoding of sequence advances, part II

2016-08-22 Thread Craig Ringer
On 23 Aug 2016 05:43, "Kevin Grittner" wrote: > > On Mon, Aug 22, 2016 at 3:29 PM, Robert Haas wrote: > > > it seems to me that > > this is just one facet of a much more general problem: given two > > transactions T1 and T2, the order of replay must match the order of > > commit unless you can pr

Re: [HACKERS] UTF-8 docs?

2016-08-22 Thread Tatsuo Ishii
> On 8/22/16 9:32 AM, Tatsuo Ishii wrote: >> I don't know what kind of problem you are seeing with encoding >> handling, but at least UTF-8 is working for Japanese, French and >> Russian. > > Those translations are using DocBook XML. But in the mean time I can create UTF-8 HTML files like this:

Re: [HACKERS] distinct estimate of a hard-coded VALUES list

2016-08-22 Thread Tomas Vondra
On 08/22/2016 07:42 PM, Alvaro Herrera wrote: Robert Haas wrote: On Sat, Aug 20, 2016 at 4:58 PM, Tom Lane wrote: Jeff Janes writes: On Thu, Aug 18, 2016 at 2:25 PM, Tom Lane wrote: It does know it, what it doesn't know is how many duplicates there are. Does it know whether the count

[HACKERS] pg_receivexlog does not report flush position with --synchronous

2016-08-22 Thread Gabriele Bartolini
Hi guys, while adding synchronous WAL streaming to Barman, I noticed that pg_receivexlog - unless a replication slot is specified and --synchronous is passed - does not become a synchronous receiver (if the application_name is in the synchronous_standby_names value). I was a bit surprised by thi

Re: [HACKERS] Server crash due to SIGBUS(Bus Error) when trying to access the memory created using dsm_create().

2016-08-22 Thread Thomas Munro
On Tue, Aug 23, 2016 at 8:41 AM, Robert Haas wrote: > On Tue, Aug 16, 2016 at 7:41 PM, Thomas Munro > wrote: >> I still think it's worth thinking about something along these lines on >> Linux only, where holey Swiss tmpfs files can bite you. Otherwise >> disabling overcommit on your OS isn't eno

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2016-08-22 Thread Gavin Flower
On 23/08/16 09:40, Andres Freund wrote: Hi, as noted in [1] I started hacking on removing the current implementation of SRFs in the targetlist (tSRFs henceforth). IM discussion brought the need for a description of the problem, need and approach to light. There are several reasons for wanting t

Re: [HACKERS] Changed SRF in targetlist handling

2016-08-22 Thread Andres Freund
Hi, On 2016-08-22 16:20:58 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2016-08-17 17:41:28 -0700, Andres Freund wrote: > >> Tom, do you think this is roughly going in the right direction? > > I've not had time to look at this patch, I'm afraid. If you still > want me to, I can make ti

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-08-22 Thread Peter Geoghegan
On Mon, Aug 1, 2016 at 3:18 PM, Peter Geoghegan wrote: > Attached WIP patch series: This has bitrot, since commit da1c9163 changed the interface for checking parallel safety. I'll have to fix that, and will probably take the opportunity to change how workers have maintenance_work_mem apportioned

Re: [HACKERS] Logical decoding of sequence advances, part II

2016-08-22 Thread Kevin Grittner
On Mon, Aug 22, 2016 at 3:29 PM, Robert Haas wrote: > it seems to me that > this is just one facet of a much more general problem: given two > transactions T1 and T2, the order of replay must match the order of > commit unless you can prove that there are no dependencies between > them. I don't

[HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2016-08-22 Thread Andres Freund
Hi, as noted in [1] I started hacking on removing the current implementation of SRFs in the targetlist (tSRFs henceforth). IM discussion brought the need for a description of the problem, need and approach to light. There are several reasons for wanting to get rid of tSRFs. The primary ones in my

Re: [HACKERS] pg_bsd_indent - improvements around offsetof and sizeof

2016-08-22 Thread Alvaro Herrera
Bruce Momjian wrote: > On Tue, Aug 16, 2016 at 11:47:09AM -0700, Andres Freund wrote: > > On 2016-08-15 18:09:02 +, Piotr Stefaniak wrote: > > > There are more fixes I intend to do, of which the most relevant for > > > Postgres are: > > > 1) fixing "function pointer typedef formatting" > > >

Re: [HACKERS] New SQL counter statistics view (pg_stat_sql)

2016-08-22 Thread Gavin Flower
On 23/08/16 08:27, Tom Lane wrote: Andres Freund writes: On 2016-08-22 13:54:43 -0400, Robert Haas wrote: On Sat, Aug 20, 2016 at 11:17 AM, Tom Lane wrote: I'm inclined to suggest you forget this approach and propose a single counter for "SQL commands executed", which avoids all of the above

Re: [HACKERS] sslmode=require fallback

2016-08-22 Thread Bruce Momjian
On Fri, Aug 19, 2016 at 09:22:32AM -0700, Jeff Janes wrote: > On Sat, Jul 30, 2016 at 11:18 AM, Bruce Momjian wrote: > > On Fri, Jul 29, 2016 at 11:27:06AM -0400, Peter Eisentraut wrote: > > On 7/29/16 11:13 AM, Bruce Momjian wrote: > > > Yes, I am thinking of a case where Postgres is

Re: [HACKERS] pg_bsd_indent - improvements around offsetof and sizeof

2016-08-22 Thread Bruce Momjian
On Tue, Aug 16, 2016 at 11:47:09AM -0700, Andres Freund wrote: > On 2016-08-15 18:09:02 +, Piotr Stefaniak wrote: > > There are more fixes I intend to do, of which the most relevant for > > Postgres are: > > 1) fixing "function pointer typedef formatting" > > This alone would warrant a bottle

Re: [HACKERS] [Patch] Temporary tables that do not bloat pg_catalog (a.k.a fast temp tables)

2016-08-22 Thread Andres Freund
On 2016-08-22 17:50:11 -0300, Alvaro Herrera wrote: > > 2. You can't write to unlogged tables on standby servers, so this > > doesn't help solve the problem of wanting to use temporary tables on > > standbys. > > Check. We could think about relaxing this restriction, which would > enable the feat

Re: [HACKERS] Add -c to rsync commands on SR tutorial wiki page

2016-08-22 Thread Robert Haas
On Thu, Aug 18, 2016 at 2:27 PM, Jim Nasby wrote: > I don't think it's any great leap for someone to think they can use those > commands incrementally. It's certainly one of the first things you think of > when using rsync. AFAIK there's no downside at all to using -c when it is a > brand new copy

Re: [HACKERS] [Patch] Temporary tables that do not bloat pg_catalog (a.k.a fast temp tables)

2016-08-22 Thread Alvaro Herrera
Robert Haas wrote: > However: > > 1. The number of tables for which we would need to add a duplicate, > unlogged table is formidable. You need pg_attribute, pg_attrdef, > pg_constraint, pg_description, pg_type, pg_trigger, pg_rewrite, etc. > And the backend changes needed so that we used the unl

Re: [HACKERS] Server crash due to SIGBUS(Bus Error) when trying to access the memory created using dsm_create().

2016-08-22 Thread Robert Haas
On Tue, Aug 16, 2016 at 7:41 PM, Thomas Munro wrote: > I still think it's worth thinking about something along these lines on > Linux only, where holey Swiss tmpfs files can bite you. Otherwise > disabling overcommit on your OS isn't enough to prevent something > which is really a kind of deferre

Re: [HACKERS] [Patch] Temporary tables that do not bloat pg_catalog (a.k.a fast temp tables)

2016-08-22 Thread Robert Haas
On Tue, Aug 16, 2016 at 8:03 PM, Jim Nasby wrote: > On 8/16/16 11:59 AM, Robert Haas wrote: > ... >> >> That doesn't really solve the problem, because OTHER backends won't be >> able to see them. So, if I create a fast temporary table in one >> session that depends on a permanent object, some oth

Re: [HACKERS] Logical decoding of sequence advances, part II

2016-08-22 Thread Andres Freund
On 2016-08-22 16:29:12 -0400, Robert Haas wrote: > So, I wish I could give you some better advice on this topic, but > sadly I am not an expert in this area. However, it seems to me that > this is just one facet of a much more general problem: given two > transactions T1 and T2, the order of repla

Re: [HACKERS] Logical decoding of sequence advances, part II

2016-08-22 Thread Robert Haas
On Sun, Aug 21, 2016 at 11:13 PM, Craig Ringer wrote: > If the sequence is created in the current xact (i.e. uncommitted) we have to > add the sequence updates to that xact to be replayed only if it commits. The > sequence is visible only to the toplevel xact that created the sequence so > advance

Re: [HACKERS] New SQL counter statistics view (pg_stat_sql)

2016-08-22 Thread Tom Lane
Andres Freund writes: > On 2016-08-22 13:54:43 -0400, Robert Haas wrote: >> On Sat, Aug 20, 2016 at 11:17 AM, Tom Lane wrote: >>> I'm inclined to suggest you forget this approach and propose a single >>> counter for "SQL commands executed", which avoids all of the above >>> definitional problems.

Re: [HACKERS] distinct estimate of a hard-coded VALUES list

2016-08-22 Thread Tom Lane
Robert Haas writes: > On Sat, Aug 20, 2016 at 4:58 PM, Tom Lane wrote: >> I am not exactly sold on this assumption that applications have >> de-duplicated the contents of a VALUES or IN list. They haven't been >> asked to do that in the past, so why do you think they are doing it? > It's hard t

Re: [HACKERS] Bug in abbreviated keys abort handling (found with amcheck)

2016-08-22 Thread Peter Geoghegan
On Mon, Aug 22, 2016 at 12:34 PM, Robert Haas wrote: > Ugh, that sucks. Thanks for the report and patch. Committed and > back-patched to 9.5. Thanks. Within Heroku, there is a lot of enthusiasm for the idea of sharing hard data about the prevalence of problems like this. I hope to be able to sh

Re: [HACKERS] Changed SRF in targetlist handling

2016-08-22 Thread Tom Lane
Andres Freund writes: > On 2016-08-17 17:41:28 -0700, Andres Freund wrote: >> Tom, do you think this is roughly going in the right direction? I've not had time to look at this patch, I'm afraid. If you still want me to, I can make time in a day or so. > I'm working on these. Atm ExecMakeTableFu

Re: [HACKERS] Bug in abbreviated keys abort handling (found with amcheck)

2016-08-22 Thread Robert Haas
On Fri, Aug 19, 2016 at 6:07 PM, Peter Geoghegan wrote: > I found another bug as a result of using amcheck on Heroku customer > databases. This time, the bug is in core Postgres. It's one of mine. > > There was a thinko in tuplesort's abbreviation abort logic, causing > certain SortTuples to be sp

Re: [HACKERS] Changed SRF in targetlist handling

2016-08-22 Thread Andres Freund
Hi, On 2016-05-23 09:26:03 +0800, Craig Ringer wrote: > SRFs-in-tlist are a lot faster for lockstep iteration etc. They're also > much simpler to write, though if the result result rowcount differs > unexpectedly between the functions you get exciting and unexpected > behaviour. > > WITH ORDINALI

Re: [HACKERS] Exporting more function in libpq

2016-08-22 Thread Alvaro Herrera
Craig Ringer wrote: > On 19 August 2016 at 14:17, Tatsuo Ishii wrote: > > > I would like to proppse to export these functions in libpq. > > > > pqPutMsgStart > > pqPutMsgEnd > > pqPutc > > pqPuts > > pqPutInt > > pqPutnchar > > pqFlush > > pqHandleSendFailure > > > > I think this would be useful

Re: [HACKERS] Changed SRF in targetlist handling

2016-08-22 Thread Andres Freund
On 2016-08-17 17:41:28 -0700, Andres Freund wrote: > Tom, do you think this is roughly going in the right direction? My plan > here is to develop two patches, to come before this: > > a) Allow to avoid using a tuplestore for SRF_PERCALL SRFs in ROWS FROM - >otherwise our performance would regr

Re: [HACKERS] Most efficient way for libPQ .. PGresult serialization

2016-08-22 Thread Joshua Bay
Oh I see. I just read more on use cases PgBouncer, but seems like it can't be used for my project. The reason is that I need to have my middleware to have full control over each transaction. That is it must be able to decide if it's going to commit or abort a single query (reason why libpq is used

Re: [HACKERS] Most efficient way for libPQ .. PGresult serialization

2016-08-22 Thread Joshua Bay
No, it can be anything else. Please correctly me if I'm wrong, but to me, PgPool-II looks like a proxy server that forwards all the data without interpretation. Can I intercept in the middle and control the flow between client and server? For e.g, I need control when the result of transaction is s

Re: [HACKERS] distinct estimate of a hard-coded VALUES list

2016-08-22 Thread Alvaro Herrera
Robert Haas wrote: > On Sat, Aug 20, 2016 at 4:58 PM, Tom Lane wrote: > > Jeff Janes writes: > >> On Thu, Aug 18, 2016 at 2:25 PM, Tom Lane wrote: > >>> It does know it, what it doesn't know is how many duplicates there are. > > > >> Does it know whether the count comes from a parsed query-strin

Re: [HACKERS] New SQL counter statistics view (pg_stat_sql)

2016-08-22 Thread Andres Freund
On 2016-08-22 13:54:43 -0400, Robert Haas wrote: > On Sat, Aug 20, 2016 at 11:17 AM, Tom Lane wrote: > > I'm inclined to suggest you forget this approach and propose a single > > counter for "SQL commands executed", which avoids all of the above > > definitional problems. People who need more det

Re: [HACKERS] New SQL counter statistics view (pg_stat_sql)

2016-08-22 Thread Robert Haas
On Sat, Aug 20, 2016 at 11:17 AM, Tom Lane wrote: > I'm inclined to suggest you forget this approach and propose a single > counter for "SQL commands executed", which avoids all of the above > definitional problems. People who need more detail than that are > probably best advised to look to cont

Re: [HACKERS] Showing parallel status in \df+

2016-08-22 Thread Pavel Stehule
2016-08-22 18:19 GMT+02:00 Robert Haas : > On Mon, Aug 22, 2016 at 4:49 AM, Pavel Stehule > wrote: > > This feature shows source code for PL function when \df statement was > used. > > I am not too sure, if this functionality is necessary - but I don't see > any > > argument against. Sometimes it

Re: [HACKERS] Re: PROPOSAL: make PostgreSQL sanitizers-friendly (and prevent information disclosure)

2016-08-22 Thread Andres Freund
On 2016-08-22 13:49:47 -0400, Robert Haas wrote: > On Mon, Aug 22, 2016 at 1:46 PM, Andres Freund wrote: > > I don't think the runtime overhead is likely to be all that high - if > > you look at valgrind.supp the peformancecritical parts basically are: > > - pgstat_send - the context switching is

Re: [HACKERS] Re: PROPOSAL: make PostgreSQL sanitizers-friendly (and prevent information disclosure)

2016-08-22 Thread Robert Haas
On Mon, Aug 22, 2016 at 1:46 PM, Andres Freund wrote: > I don't think the runtime overhead is likely to be all that high - if > you look at valgrind.supp the peformancecritical parts basically are: > - pgstat_send - the context switching is going to drown out some zeroing > - xlog insertions - mak

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-22 Thread Andres Freund
On 2016-08-22 13:41:57 -0400, Robert Haas wrote: > On Mon, Aug 22, 2016 at 1:32 PM, Heikki Linnakangas wrote: > >> But what about the best case? If we create a scenario where there are > >> no open read-write transactions at all and (somehow) lots and lots of > >> ProcArrayLock contention, how mu

Re: [HACKERS] Re: PROPOSAL: make PostgreSQL sanitizers-friendly (and prevent information disclosure)

2016-08-22 Thread Andres Freund
On 2016-08-22 13:16:34 -0400, Robert Haas wrote: > On Sat, Aug 20, 2016 at 3:46 PM, Tom Lane wrote: > > So to me, it seems like the core of this complaint boils down to "my > > sanitizer doesn't understand the valgrind exclusion patterns that have > > been created for Postgres". We can address th

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-22 Thread Robert Haas
On Mon, Aug 22, 2016 at 1:32 PM, Heikki Linnakangas wrote: >> But what about the best case? If we create a scenario where there are >> no open read-write transactions at all and (somehow) lots and lots of >> ProcArrayLock contention, how much does this help? > > I ran some quick pgbench tests on

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-22 Thread Andres Freund
Hi, On 2016-08-22 20:32:42 +0300, Heikki Linnakangas wrote: > I ran some quick pgbench tests on my laptop, but didn't see any meaningful > benefit. I think the best I could see is about 5% speedup, when running > "pgbench -S", with 900 idle connections sitting in the background. On the > positive

Re: [HACKERS] Re: PROPOSAL: make PostgreSQL sanitizers-friendly (and prevent information disclosure)

2016-08-22 Thread Robert Haas
On Sat, Aug 20, 2016 at 3:46 PM, Tom Lane wrote: > Noah Misch writes: >> On Fri, Aug 19, 2016 at 07:22:02PM -0400, Tom Lane wrote: >>> So maybe we ought to work towards taking those out? > >> Maybe. It's a policy question boiling down to our willingness to spend >> cycles >> zeroing memory in o

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-22 Thread Heikki Linnakangas
On 08/22/2016 07:49 PM, Robert Haas wrote: Nice to see you working on this again. On Mon, Aug 22, 2016 at 12:35 PM, Heikki Linnakangas wrote: A sequential scan of a table like that with 10 million rows took about 700 ms on my laptop, when the hint bits are set, without this patch. With this pa

Re: [HACKERS] distinct estimate of a hard-coded VALUES list

2016-08-22 Thread Robert Haas
On Sat, Aug 20, 2016 at 4:58 PM, Tom Lane wrote: > Jeff Janes writes: >> On Thu, Aug 18, 2016 at 2:25 PM, Tom Lane wrote: >>> It does know it, what it doesn't know is how many duplicates there are. > >> Does it know whether the count comes from a parsed query-string list/array, >> rather than be

Re: [HACKERS] [PATCH] Transaction traceability - txid_status(bigint)

2016-08-22 Thread Robert Haas
On Sat, Aug 20, 2016 at 9:46 AM, Craig Ringer wrote: > Ahem. Forgot to squash in a fixup commit. Updated patch of > txid_status(bigint) attachd. > > A related patch follows, adding a new txid_current_ifassigned(bigint) > function as suggested by Jim Nasby. It's usefully connected to txid_status()

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-22 Thread Robert Haas
Nice to see you working on this again. On Mon, Aug 22, 2016 at 12:35 PM, Heikki Linnakangas wrote: > A sequential scan of a table like that with 10 million rows took about 700 > ms on my laptop, when the hint bits are set, without this patch. With this > patch, if there's a snapshot holding back

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-22 Thread Heikki Linnakangas
On 08/22/2016 07:35 PM, Heikki Linnakangas wrote: And here's a new patch version... And here's the attachment I forgot, *sigh*.. - Heikki csn-4.patch.gz Description: application/gzip -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-22 Thread Heikki Linnakangas
And here's a new patch version. Still lots of work to do, especially in performance testing, and minimizing the worst-case performance hit. On 08/09/2016 03:16 PM, Heikki Linnakangas wrote: Next steps: * Hot standby feedback is broken, now that CSN != LSN again. Will have to switch this back t

Re: [HACKERS] Showing parallel status in \df+

2016-08-22 Thread Robert Haas
On Mon, Aug 22, 2016 at 4:49 AM, Pavel Stehule wrote: > This feature shows source code for PL function when \df statement was used. > I am not too sure, if this functionality is necessary - but I don't see any > argument against. Sometimes it can be useful, mainly when we work with > overloaded fu

Re: [HACKERS] Typo in comment to function LockHasWaitersRelation() [master branch]

2016-08-22 Thread Robert Haas
On Mon, Aug 22, 2016 at 9:01 AM, Dmitry Ivanov wrote: >> Hi hackers, >> >> I've found a typo in a comment to function LockHasWaitersRelation() [lmgr.c >> : >> 271, master branch]: >> >> This is a functiion to check > > Attached a patch. Thanks. Committed with a bit of additional wordsmithing. -

Re: [HACKERS] WAL consistency check facility

2016-08-22 Thread Simon Riggs
On 22 August 2016 at 13:44, Kuntal Ghosh wrote: > Please let me know your thoughts on this. Do the regression tests pass with this option enabled? -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsq

Re: [HACKERS] LSN as a recovery target

2016-08-22 Thread Robert Haas
On Mon, Aug 22, 2016 at 8:28 AM, Michael Paquier wrote: > On Mon, Aug 22, 2016 at 7:12 PM, Adrien Nayrat > wrote: >> As Julien said, there is nothing to notice that error comes from >> recovery.conf. >> My fear would be that an user encounters an error like this. Il will be >> difficult to link

Re: [HACKERS] WAL consistency check facility

2016-08-22 Thread Robert Haas
On Mon, Aug 22, 2016 at 9:25 AM, Michael Paquier wrote: > Another pin-point is: given a certain page, how do we identify of > which type it is? One possibility would be again to extend the AM > handler with some kind of is_self function with a prototype like that: > bool handler->is_self(Page); >

Re: [HACKERS] Tracking wait event for latches

2016-08-22 Thread Robert Haas
On Mon, Aug 22, 2016 at 9:49 AM, Michael Paquier wrote: > The reason why I chose this way is that there are a lot of them. It is > painful to maintain the order of the array elements in perfect mapping > with the list of IDs... You can use stupid macro tricks to help with that problem... -- Rob

Re: [HACKERS] UTF-8 docs?

2016-08-22 Thread Peter Eisentraut
On 8/22/16 9:32 AM, Tatsuo Ishii wrote: > I don't know what kind of problem you are seeing with encoding > handling, but at least UTF-8 is working for Japanese, French and > Russian. Those translations are using DocBook XML. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL

Re: [HACKERS] improved DefElem list processing

2016-08-22 Thread Alvaro Herrera
Peter Eisentraut wrote: > I'm not happy that utils/acl.h has prototypes for aclchk.c, because > acl.h is included all over the place. Perhaps I should make a > src/include/catalog/aclchk.c to clean that up. I've been bothered by that too in the past. +1 for the cleanup. -- Álvaro Herrera

Re: [HACKERS] improved DefElem list processing

2016-08-22 Thread Michael Paquier
On Mon, Aug 22, 2016 at 10:41 PM, Pavel Stehule wrote: > 1. This patch introduce location in DefElement node, and inject ParserState > to SQL commands, where ParserState was not used. It allows to show the > position of an error. This patch is not small, but almost changes are > trivial. > > 2. Th

Re: [HACKERS] dsm_unpin_segment

2016-08-22 Thread Amit Kapila
On Mon, Aug 22, 2016 at 6:06 PM, Thomas Munro wrote: > On Tue, Aug 23, 2016 at 12:07 AM, Amit Kapila wrote: >> + int control_slot = -1; >> ... >> + if (control_slot == -1) >> + elog(ERROR, "cannot unpin unknown segment handle"); >> >> Isn't it better to use INVALID_CONTROL_SLOT for control_slot a

Re: [HACKERS] Tracking wait event for latches

2016-08-22 Thread Michael Paquier
On Mon, Aug 22, 2016 at 6:09 PM, Alexander Korotkov wrote: > Hi, Michael! > > On Thu, Aug 4, 2016 at 8:26 AM, Michael Paquier > wrote: > I took a look at your patch. Couple of notes from me. Thanks! >> const char * >> GetEventIdentifier(uint16 eventId) >> { >> const char *res; >> switch (eventI

Re: [HACKERS] improved DefElem list processing

2016-08-22 Thread Pavel Stehule
Hi 2016-08-11 17:32 GMT+02:00 Peter Eisentraut < peter.eisentr...@2ndquadrant.com>: > On 8/5/16 11:25 AM, Peter Eisentraut wrote: > > On 8/4/16 2:21 PM, Tom Lane wrote: > >> Forgot to mention: seems like you should have added a location > >> argument to makeDefElem. > > > > I was hesitating to d

Re: [HACKERS] WAL consistency check facility

2016-08-22 Thread Michael Paquier
On Mon, Aug 22, 2016 at 9:44 PM, Kuntal Ghosh wrote: > Please let me know your thoughts on this. Since custom AMs have been introduced, I have kept that in a corner of my mind and thought about it a bit. And while the goal of this patch is clearly worth it, I don't think that the page masking int

Re: [HACKERS] UTF-8 docs?

2016-08-22 Thread Tatsuo Ishii
> On 8/22/16 1:16 AM, Tatsuo Ishii wrote: >> Just out of curiopusity, I wonder why we can't make the encoding of >> SGML docs to be UTF-8, rather than current ISO-8859-1. > > Encoding handling in DocBook SGML is weird, and making it work robustly > will either fail or might be more work than just

Re: [HACKERS] Re: [sqlsmith] FailedAssertion("!(XLogCtl->Insert.exclusiveBackup)", File: "xlog.c", Line: 10200)

2016-08-22 Thread Michael Paquier
On Mon, Aug 22, 2016 at 10:09 PM, Amit Kapila wrote: > Won't the similar problem exists for nonExclusiveBackups? Basically > in similar situation, the count for nonExclusiveBackups will be > decremented and if it hits pg_start_backup_callback(), the following > Assertion can fail. > pg_start_back

Re: [HACKERS] Re: [sqlsmith] FailedAssertion("!(XLogCtl->Insert.exclusiveBackup)", File: "xlog.c", Line: 10200)

2016-08-22 Thread Amit Kapila
On Mon, Aug 22, 2016 at 7:13 AM, Michael Paquier wrote: > On Sat, Aug 6, 2016 at 6:35 PM, Andreas Seltenreich > wrote: >> Michael Paquier writes: >> >>> Andreas, with the patch attached is the assertion still triggered? >>> [2. text/x-diff; base-backup-crash-v2.patch] >> >> I didn't observe the

Re: [HACKERS] LSN as a recovery target

2016-08-22 Thread Peter Eisentraut
On 8/22/16 8:28 AM, Michael Paquier wrote: > Thinking a bit wider than that, we may want to know such context for > normal GUC parameters as well, and that's not the case now. Perhaps > there is actually a reason why that's not done for GUCs, but it seems > that it would be useful there as well. G

Re: [HACKERS] Typo in comment to function LockHasWaitersRelation() [master branch]

2016-08-22 Thread Dmitry Ivanov
> Hi hackers, > > I've found a typo in a comment to function LockHasWaitersRelation() [lmgr.c > : > 271, master branch]: > >> This is a functiion to check Attached a patch. -- Dmitry Ivanov Postgres Professional: http://www.postgrespro.com Russian Postgres Companydiff --git a/src/backend/storag

[HACKERS] Typo in comment to function LockHasWaitersRelation() [master branch]

2016-08-22 Thread Dmitry Ivanov
Hi hackers, I've found a typo in a comment to function LockHasWaitersRelation() [lmgr.c : 271, master branch]: >> This is a functiion to check -- Dmitry Ivanov Postgres Professional: http://www.postgrespro.com Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@p

Re: [HACKERS] UTF-8 docs?

2016-08-22 Thread Peter Eisentraut
On 8/22/16 1:16 AM, Tatsuo Ishii wrote: > Just out of curiopusity, I wonder why we can't make the encoding of > SGML docs to be UTF-8, rather than current ISO-8859-1. Encoding handling in DocBook SGML is weird, and making it work robustly will either fail or might be more work than just completing

[HACKERS] WAL consistency check facility

2016-08-22 Thread Kuntal Ghosh
Hi, I've attached a patch to check if the current page is equal with the FPW after applying WAL on it. This is how the patch works: 1. When a WAL record is inserted, a FPW is done for that operation. But, a flag is kept to indicate whether that page needs to be restored. 2. During recovery, when

Re: [HACKERS] dsm_unpin_segment

2016-08-22 Thread Thomas Munro
On Tue, Aug 23, 2016 at 12:07 AM, Amit Kapila wrote: > + int control_slot = -1; > ... > + if (control_slot == -1) > + elog(ERROR, "cannot unpin unknown segment handle"); > > Isn't it better to use INVALID_CONTROL_SLOT for control_slot and use > datatype as uint32 (same is used for dsm_segment->con

Re: [HACKERS] LSN as a recovery target

2016-08-22 Thread Michael Paquier
On Mon, Aug 22, 2016 at 7:12 PM, Adrien Nayrat wrote: > As Julien said, there is nothing to notice that error comes from > recovery.conf. > My fear would be that an user encounters an error like this. Il will be > difficult to link to the recovery.conf. Thinking a bit wider than that, we may want

Re: [HACKERS] Binary I/O for isn extension

2016-08-22 Thread Fabien COELHO
Hello Shay, Attached is a new version of the patch, adding an upgrade script and the rest of it. Note that because, as Fabien noted, there's doesn't seem to be a way to add send/receive functions with ALTER TYPE, I did that by updating pg_type directly - hope that's OK. This patch does not ap

Re: [HACKERS] dsm_unpin_segment

2016-08-22 Thread Amit Kapila
On Mon, Aug 22, 2016 at 5:24 AM, Thomas Munro wrote: > On Sat, Aug 20, 2016 at 11:37 PM, Amit Kapila wrote: >> On Tue, Aug 9, 2016 at 10:07 AM, Thomas Munro >> wrote: >>> On Tue, Aug 9, 2016 at 12:53 PM, Tom Lane wrote: The larger picture here is that Robert is exhibiting a touching but >>

Re: [HACKERS] Patch: initdb: "'" for QUOTE_PATH (non-windows)

2016-08-22 Thread Michael Paquier
On Mon, Aug 22, 2016 at 8:45 PM, Tom Lane wrote: > Likely it would be better to refactor all of this so we get just one > reference to libpq and one reference to libpgport, but that'd require > a more thoroughgoing cleanup than you have here. (Now that I think > about it, adding -L/-l to LDFLAGS

Re: [HACKERS] Patch: initdb: "'" for QUOTE_PATH (non-windows)

2016-08-22 Thread Tom Lane
Michael Paquier writes: > On Sun, Aug 21, 2016 at 3:02 AM, Tom Lane wrote: >> I looked into this and soon found that fe_utils/string_utils.o has >> dependencies on libpq that are much wider than just pqexpbuffer :-(. > pqexpbuffer.c is an independent piece of facility, so we could move it > to s

  1   2   >